Senior Technical Program Manager, Human-in-the-Loop Operations
RobloxUser-Generated Content company
San Mateo, United States$226,400 - $277,350 USDSenior
Altimeter Capital
Dragoneer Investment Group
Andreessen Horowitz
Tencent
First Round Capital
Warner Music Group
Operations & StrategyNew
About the role
TL;DR
Manage Roblox's AI data labeling and model evaluation ecosystem, scaling operations and driving AI-assisted workflows.
- •Lead the end-to-end lifecycle of Roblox's AI data labeling and model evaluation ecosystem.
- •This high-leverage role is at the intersection of Research, Engineering, and Operations, scaling data operations to handle millions of items annually and driving the platform's evolution from manual to AI-assisted workflows.
- •Key Responsibilities Lead data programs end-to-end, from translating ML requirements to overseeing contractor execution and quality review.
- •Define and maintain ground truth by developing labeling guidelines, annotation schemas, and quality frameworks.
- •Manage a multi-vendor, distributed workforce of remote contractors, overseeing allocation, productivity, and budget.
- •Drive the shift to AI-assisted workflows by partnering with Engineering to implement LLM-based pre-labeling and evaluation.
- •Partner cross-functionally with ML Engineers, Data Scientists, and Product Managers to translate model needs into executable data programs.
- •Requirements 5+ years of experience in Technical Program Management, Data Operations, or Operations Management within an AI/ML or data-intensive environment.
- •Deep hands-on experience with data labeling, annotation, or model evaluation, including designing schemas and managing quality control.
- •Proven experience managing external vendor relationships and distributed contractor workforces.
- •Strong understanding of the ML lifecycle and the role of human-labeled data.
- •Proficiency in SQL and Python for data analysis and operational dashboards.
Required skills
PythonSQLLLMs
Domain expertise
gamingai
Tech stack
PythonSQLLLMsAnthropic APIGemini API