Senior AI Reliability Engineer (Platform)Active

Flatiron Health
Apply now

The opportunity

We’re looking for a Senior AI Reliability Engineer (Platform) to help us accomplish our mission to improve and extend lives by learning from the experience of every person with cancer. Are you ready to be the next changemaker in cancer care?

What you'll do

  • Design, build, and continuously improve evaluation frameworks, benchmarks,: and automated testing pipelines for AI, LLM-powered, and agentic workflows.

  • Define and monitor quality, reliability, safety, performance, and cost: metrics for AI systems, including observability, drift detection, hallucination risk, retrieval quality, and end-to-end workflow behaviour.

  • Develop reliability engineering practices for AI-enabled systems, including: SLOs, SLIs, monitoring, alerting, incident response, runbooks, and root-cause analysis of AI failure modes.

  • Design orchestration, governance, and guardrails for multi-agent AI systems,: including agent coordination, permissions, auditability, human oversight, and secure deployment patterns.

  • Partner with platform, product, security, engineering, and data science teams: to evaluate AI solutions, establish reusable standards, and guide build-vs-buy, model selection, and AI adoption decisions.

  • Support experimentation with emerging AI technologies while helping the: organisation make pragmatic, scalable decisions in a rapidly evolving landscape, collaborating across global teams and participating in on-call rotations.

What they're looking for

  • + years of experience in platform engineering, SRE, machine learning, MLOps: or a related technical field, with strong Python skills and experience building production-quality systems.
  • Solid experience with the following AWS, Bedrock, AgentCore, ML Flow, Databricks, Cloudwatch, Gitlab
  • Experience designing experiments, evaluation frameworks, statistical: analyses, and quality metrics for ML or AI systems, with familiarity in LLMs, RAG, AI agents, prompt evaluation, and model behaviour.
  • Strong understanding of AI reliability and observability, including logging,: tracing, monitoring, drift detection, statistical analysis, uncertainty, alerting, and production system health.
  • Experience with modern cloud and ML infrastructure, including AWS,: containers, Kubernetes, CI/CD, data pipelines, workflow orchestration, versioning, and distributed compute platforms.
  • Knowledge of agentic and multi-agent systems, including orchestration, state: management, tool execution, governance, reliability, human-in-the-loop controls, and selecting the appropriate level of AI autonomy for a given problem.
  • Strong communication and collaboration skills, with the ability to explain AI: behaviour and tradeoffs to technical and non-technical stakeholders and thrive in a fast-moving, ambiguous environment with a pragmatic, enablement-focused mindset.
  • Fluent in English.