Senior LLMOps Engineer
HeidiAI-powered Clinical company
Melbourne, AustraliaSenior
Headline
Point72 Private Investments
Blackbird
Latitude
Possible Ventures
Archangel
Data & AI
About the role
TL;DR
Build and own the operational layer for AI models, ensuring visibility, feedback loops, and unit economics.
- •Heidi is seeking a Senior LLMOps Engineer to build and own the operational layer for their AI models.
- •This role is crucial for ensuring full visibility into deployments, creating a feedback loop for model improvement, and managing model unit economics.
- •Key Responsibilities Build deployment health dashboards with monitoring and alerting.
- •Ensure traceability of incidents from sessions to model IDs and root causes.
- •Develop a feedback flywheel to turn clinician signals into better models.
- •Surface execution traces and generate summaries for stakeholders.
- •Filter high-value feedback into training data and monitor post-deployment performance.
- •Requirements 2-3 years of hands-on LLMOps experience building observability, tracing, evaluation, and feedback systems for production LLMs.
- •Experience from an AI company operating at or ahead of Heidi's maturity.
- •Proven ability to ship monitoring, alerting, distributed tracing, and scalable data models.
- •Experience building with LLMs, including using agents for feedback triage.
- •Comfort joining cost and revenue data into per-model unit economics.
Required skills
LLMsLangChainKubernetesDockerCI/CDPythonBashDatadogREST API
Nice-to-have skills
MLflowKubeflowGitHub ActionsJenkinsArgoCDHelmGitLab CICircleCIAnsibleChef
Domain expertise
healthtechai
Benefits & perks
$1,000 annual learning and development budget, $150/month health and wellness allowance, $500 home office budget, 26 weeks paid primary parental leave, 18 weeks paid secondary parental leave, fertility support up to $10,000, four weeks of work from anywhere per year, serious equity
Tech stack
PythonBashDatadogOpenAI APIAnthropic APIGemini APILangChainHugging FaceLLMsMLflowKubeflowDockerKubernetesCI/CDGitHub ActionsJenkinsArgoCDHelmGitLab CICircleCIAnsibleChefPuppetPrometheusGrafanaPagerDutyNginxHAProxyREST API