Data Scientist — Agent Evaluations & Quality
CleraIndustrial Electrification company
Palo Alto, United StatesSenior
Data & AI
About the role
TL;DR
Data Scientist to measure, understand, and improve AI agent quality.
- •We're an early-stage AI company building autonomous agents that handle real work
- •email, calendar, browser, business software, and more.
- •We're hiring a Data Scientist focused on Agent Evaluations & Quality to measure, understand, and continuously improve the quality of our agent capabilities.
- •Your mission is to translate ambiguous product behavior into measurable definitions of success, build representative evaluation datasets, design reliable graders and metrics, analyze failures, and create the feedback loops that guide engineering and product decisions.
- •This is applied data science at the intersection of evaluation design, statistics, experimentation, production Python, and deep understanding of how LLM agents behave in real products.
- •This is a full-time, on-site role based in Palo Alto, CA.
- •Visa sponsorship is not available.
- •Key Responsibilities Architect and maintain automated evaluation pipelines that measure agent quality across capabilities and product surfaces.
- •Translate capabilities into explicit success criteria
- •including pass, partial-pass, and failure definitions for complex multi-step tasks.
- •Build representative gold datasets and regression suites covering common workflows, ambiguous requests, long-tail behavior, edge cases, and adversarial scenarios.
- •Define and track metrics such as task success, partial completion, tool-selection accuracy, tool-use correctness, instruction adherence, factual consistency, user corrections, latency, cost, and reliability.
- •Analyze traces, tool calls, model outputs, user context, and production outcomes to identify root causes and build a useful failure taxonomy.
- •Requirements 5+ years of experience in data science, machine learning, or analytics roles building or delivering evaluation systems, metrics frameworks, or quality measurement solutions for production systems.
- •Demonstrated experience designing and implementing evaluation frameworks, metrics, and grading systems for ML/AI systems in production.
- •Production-quality Python and SQL proficiency with the ability to build automated data pipelines and analysis code at scale.
- •Experience designing evaluation methodologies: success criteria definition, dataset construction, metric selection, and distinguishing useful benchmarks from misleading ones.
- •Working knowledge of LLM behavior
- •including model-based graders, tool use, retrieval systems, multi-step execution, partial completion, and practical failure modes.
Required skills
PythonSQLLLMs
Domain expertise
aiproductivity
Tech stack
PythonSQLLLMs