Senior Software Engineer - Model Evaluation & AI Systems
DeepgramSpeech Recognition company
RemoteSenior
In-Q-Tel
Madrona
Tiger Global Management
AVP
Alkeon
Wing Venture Capital
Data & AI
About the role
TL;DR
Builds infrastructure to measure and validate AI model quality at scale.
- •Deepgram is seeking a Senior Software Engineer to join the team responsible for validating the quality of their speech, audio, and multilingual AI models before they reach customers.
- •This role involves defining evaluation methodology and building the infrastructure to measure model quality at scale.
- •Key Responsibilities Define and build evaluation methodologies for various AI models.
- •Design, build, and maintain automated evaluation pipelines.
- •Build scalable, reproducible evaluation infrastructure.
- •Translate research benchmarks into automated, enforceable pass/fail gates.
- •Build and operate continuous monitoring systems for quality regressions.
- •Requirements 5+ years of professional software or QA engineering experience.
- •Solid backend/scripting experience in Python, Rust, Go, or similar.
- •Experience designing and building automated test pipelines or evaluation frameworks.
- •Strong analytical skills and comfort reasoning about metrics.
- •Ability to take charge of ambiguous technical challenges and communicate effectively.
Required skills
PythonRustGoLLMsCI/CDDockerKubernetesGrafana
Nice-to-have skills
OpenAI APILangChainReact Native
Domain expertise
aideveloper-tools
Tech stack
PythonRustGoLLMsOpenAI APILangChainReact NativeGrafana
