AI Engineer, Evaluation
Distyl AIEnterprise AI company
San Francisco, United States$150K - $250KMid
Lightspeed Venture Partners
Khosla Ventures
Coatue
DST Global
Dell Technologies Capital
Nat Friedman
Data & AI
About the role
TL;DR
Design and implement evaluation systems for AI systems deployed in customer environments.
- •AI Evaluation Engineers at Distyl focus on designing and implementing evaluation systems for AI systems deployed in customer environments.
- •Key Responsibilities Design and implement evaluation frameworks for AI systems.
- •Define how system quality is measured in each domain.
- •Build and maintain golden test cases and regression suites in Python.
- •Develop and maintain evaluation pipelines that integrate into system iteration loops.
- •Define, calibrate, and operate LLM-based graders.
- •Work closely with various teams to ensure evaluation frameworks guide system development and deployment.
- •Requirements 2+ years of software engineering experience Strong Python Engineering Skills Experience with Evaluation-Driven or Experiment-Driven Development Ability to Translate Human Judgment into Code Systems-Oriented Mindset AI-Native Working Style Travel between 10-50% of the time
Required skills
PythonPandasNumPyAirflowdbtTableauPower BILookerSnowflakeDatabricksFivetranDagsterPrefectApache KafkaApache Flink
Benefits & perks
equity, health insurance, flexible time off, retirement benefits, financial planning, wellness benefits, complimentary lunches, access to AI models, high-impact projects
Tech stack
PythonPandasNumPyAirflowdbtTableauPower BILookerSnowflakeDatabricksFivetranDagsterPrefectApache KafkaApache FlinkPostgreSQLMySQLMongoDBRedisElasticsearchDynamoDBCassandraSQLiteOracleSQL ServerNeo4jCockroachDBTimescaleDBInfluxDBSupabase