Ensures production agents perform reliably and improve over time.
•As a Machine Learning Engineer on Agent Oversight, you will drive the end-to-end lifecycle that ensures our production agents perform reliably and improve over time.
•Key Responsibilities Build or contribute to observability into agent behavior in production Design evaluation methodologies and metrics for agentic applications Build, ship, and own ML systems that detect drift, anomalies, or misalignment in production agent behavior Design and run rigorous experiments to validate model and agent performance improvements before they ship Collaborate closely with product managers, customers, data annotators, Forward Deployed Engineers, and other engineering teams Requirements 5+ years of experience as an ML engineer or applied scientist Strong grounding in building or scaling evaluation, monitoring, or continuous-learning infrastructure for ML/agentic systems Hands-on experience with LLMs and agent architectures Comfortable partnering with software engineers to productionize research and experimental work Rigorous approach to experimentation