Computational Linguist, AI Evaluation
Twelve LabsVideo Understanding company
San Francisco, United StatesSenior
Data & AI
About the role
TL;DR
Define and build AI evaluation protocols for video-language models.
- •You will be a vital member of our ML Data Operations Team, leading the full spectrum of video-language data collection, labeling operations, and model quality measurement.
- •This role comes with high ownership and includes responsibilities such as defining dataset and evaluation requirements, designing and building data pipelines, and coordinating with vendor partners.
- •Key Responsibilities Define and build evaluation protocols for video-language model quality.
- •Build pipelines and analyses to surface model underperformance.
- •Design and execute video-language data collection and labeling projects.
- •Coordinate with vendor and outsourcing partners, ensuring high quality.
- •Work closely with Engineering and Research teams to align on data and evaluation needs.
- •Requirements Direct experience building or running model evaluation pipelines. 5+ years of experience in an AI-focused data operations organization.
- •Proven track record designing and executing large-scale data projects.
- •Ability to analyze complex data and distill findings into clear documentation.
- •Proficiency with Python and other industry automation tools.
Required skills
PythonNLPLLMsComputer VisionProject Management
Nice-to-have skills
Account ManagementEnterprise Sales
Domain expertise
aimedia
Benefits & perks
health insurance, dental insurance, vision insurance, flexible PTO, parental leave, wellness stipend, learning & development stipend, transportation stipend, daily lunch & dinner provided
Tech stack
PythonAWSS3EC2LinearNotionGitLinuxJiraConfluenceFigmaPostmanSwaggerOpenAPIREST APIgRPCWebSocketsRabbitMQCelerySidekiqSlackAsanaMonday.comAirtableExcelPowerPointGoogle AnalyticsGA4Looker StudioWorkday