Member of Technical Staff, Data Flywheel
Reflection AIDeveloper Tools company
New York, United StatesLead
Lightspeed Venture Partners
CRV
Endeavor Catalyst
Amex Ventures
Data & AINew
About the role
TL;DR
Builds data, feedback loops, and infrastructure to improve AI model performance.
- •The Data Flywheel team closes the gap between benchmark performance and useful performance in the real world.
- •You will identify and build the signals, data, and feedback loops that turn model usage into rigorous evaluations, targeted training data, and measurable improvements in future generations of models.
- •Key Responsibilities Design and build evaluations, graders, and feedback loops to measure priority real-world model behaviors.
- •Analyze model performance and failure modes, translating insights into targeted datasets and training interventions.
- •Develop human and synthetic data strategies for capabilities where existing data is insufficient.
- •Build infrastructure and pipelines for reliable data ingestion, inspection, versioning, and evaluation at scale.
- •Requirements Deep technical understanding of LLM training and evaluation, with hands-on experience in areas such as evaluation design, data curation, or reward design.
- •Strong software engineering skills and experience building automated data/evaluation pipelines or large-scale ML systems.
- •A track record of owning high-impact projects end to end, navigating ambiguity, and adapting quickly.
- •High agency and thrive in a fast-paced startup environment; bias for impact over process.
Required skills
LLMsNLPPythonSQLDockerKubernetesGit
Nice-to-have skills
Computer VisionTensorFlowPyTorchscikit-learnAWSGoogle CloudAzure
Domain expertise
aideveloper-tools
Benefits & perks
Stock options, Health & wellness, Meals, Paid parental leave, Unlimited paid time off, Sponsorship support, Team building
Tech stack
PythonSQLLLMsNLPComputer VisionTensorFlowPyTorchscikit-learnAWSGoogle CloudAzureDockerKubernetesGit
