Software Engineer - Benchmarking
Office HoursExpert Network company
Remote$160,000 - $210,000Senior
CRV
Susa Ventures
BoxGroup
Manresa Ventures
Greenoaks Capital
Flex Capital
Data & AI
About the role
TL;DR
Build and run the platform for AI model evaluations, ensuring consistent and reproducible results.
- •Office Hours is seeking a Software Engineer to build and manage the platform for AI model evaluations.
- •You will collaborate with researchers to prepare datasets, develop evaluation pipelines and environments, and publish results.
- •Key Responsibilities Prepare and maintain benchmark datasets, ensuring data quality and consistency.
- •Build and maintain evaluation pipelines for reproducible model testing.
- •Create containerized environments for evaluating model performance.
- •Develop the scoreboard and leaderboard for published results.
- •Build analysis tools to identify failure modes and track capability improvements.
- •Requirements 4+ years of professional experience building and maintaining complex systems, with strong Python skills.
- •Experience preparing, cleaning, and maintaining datasets with a focus on data rigor.
- •Comfort with Docker and building reproducible execution environments.
- •Ability to collaborate effectively with researchers and translate methodology into systems.
Required skills
PythonDockerHugging Face
Nice-to-have skills
OpenAI APIAnthropic APIGemini APIPyTorchReactNext.jsTailwind
Domain expertise
aideveloper-toolsproductivity
Benefits & perks
Competitive salary and equity, Medical, dental, and vision coverage, 401(k), Monthly wellness and fitness stipend, Paid time off policy, along with company holidays, Annual company off-sites, Parent-friendly policies, Remote flexibility, Paid family leave
Tech stack
PythonOpenAI APIAnthropic APIGemini APIHugging FacePyTorchDockerReactNext.jsTailwindSlackNotionLinear