Senior Manager, Forward Deployed Research
Snorkel AIAI Data company
New York, United States$185,000 - $321,600Manager
Addition
Greylock
Lightspeed Venture Partners
BlackRock
GV
Accenture Ventures
Data & AI
About the role
TL;DR
Lead forward-deployed research, benchmarking and tuning AI models with data.
- •Snorkel AI is hiring a Senior Manager on the Forward Deployed Research team to own how we show what our data does.
- •You own two related functions: benchmarking the latest frontier models against our data series to expose where they fall short, and tuning customer and open-source models on our data to demonstrate the lift it produces.
- •Key Responsibilities Own the system for measuring what our data does: define how we benchmark and tune models on our data, and the tooling and automation the function needs, partnering with Engineering to build it.
- •Recruit, hire, and develop a small team of engineers and researchers; a player-coach role, hands-on technical work plus team leadership.
- •Own the methodology and playbook for benchmarking and tuning: the model panels, the metrics we report, the tuning setups, and the quality bar, so results are consistent, repeatable, and defensible across accounts and data series.
- •Turn benchmark results into gap intelligence: clear analyses of where models fall short that serve as the evidence behind our data pitch and a primary input to what we build next.
- •Tune customer and open-source models on our data to demonstrate the lift it produces, and turn that into presales material and intelligence.
- •Requirements 8+ years in applied ML, model evaluation, or research-intensive engineering, with a track record of building technical systems.
- •Strong software engineering skills, with experience building data or evaluation pipelines, automation, and tooling that scale.
- •Deep understanding of model evaluation and benchmarking: designing evaluations, selecting model panels and metrics, and producing rigorous, defensible results.
- •Strong fluency in frontier AI concepts including LLMs, evaluation methodologies, post-training techniques (RLHF, DPO, RLAIF), and domain areas such as coding agents, reasoning, multimodal models, or RL environments.
- •Experience leading, hiring, and developing technical talent; comfortable as a player-coach who stays hands-on while growing a team.
Required skills
LLMsNLPPython
Nice-to-have skills
TensorFlowPyTorchscikit-learnKerasOpenAI APILangChainHugging FaceKubeflowSageMakerVertex AI
Domain expertise
aideveloper-tools
Tech stack
PythonJavaScriptSQLBashTensorFlowPyTorchscikit-learnKerasOpenAI APILangChainHugging FaceNLPLLMsMLflowKubeflowSageMakerVertex AIDockerKubernetesCI/CDGitLinuxJiraConfluencePostmanSwaggerOpenAPIREST APIgRPCWebSockets