Machine Learning Engineer, Reliability
Fal.aiGenerative Media company
RemoteSenior
Andreessen Horowitz
Meritech Capital Partners
Salesforce Ventures
Kindred Ventures
Notable Capital
Shopify Ventures
Data & AI
About the role
TL;DR
Ensure reliability, security, and safety of generative media model APIs.
- •Fal AI is building the generative media ecosystem for AI products.
- •This role is a hybrid ML Engineering / Site Reliability Engineering position focused on ensuring the reliability, security, and safety of Fal AI's generative media model APIs.
- •Key Responsibilities Own availability, latency, and throughput SLOs for generative media model APIs.
- •Build monitoring and alerting for ML-specific failures and output quality degradation.
- •Harden model deployment workflows with canary releases and automated rollbacks.
- •Drive the security posture of the model fleet, including abuse detection and rate limiting.
- •Operationalize safety systems for generative media at inference time.
- •Requirements 3+ years of professional experience, with 1 year operating production ML or high-scale API systems.
- •Strong systems fundamentals in distributed systems, networking, and observability.
- •Working knowledge of modern generative models and their production failure modes.
- •Familiarity with ML system security and safety practices is a plus.
Required skills
PythonKubernetesDockerCI/CDGitLinuxPyTorchTensorFlowscikit-learnComputer VisionLLMsNLPAWSGoogle CloudAzure
Nice-to-have skills
Hugging FaceLangChainOpenAI APIAnthropic APIGemini APIMLflowKubeflowSageMakerVertex AIOWASP
Domain expertise
developer-toolsai
Tech stack
PythonKubernetesAWSGoogle CloudAzureDockerCI/CDGitHub ActionsJenkinsArgoCDHelmGitLab CICircleCIAnsibleChefPuppetPrometheusGrafanaDatadogNew RelicPagerDutyNginxHAProxyGitLinuxJiraConfluenceFigmaPostmanSwagger