The opportunity
The Dashboards product is Datadog's unified single-pane-of-glass for metrics, logs, and traces—a comprehensive treasure trove of observability data. We are transforming Dashboards into an AI-native control surface and the central hub where every team moves seamlessly from…
What you'll do
Own the evaluation strategy for Dashboards, as well as sister teams within: our organization. Define the metrics — offline and online, quality and cost, single-turn and multi-turn — that the team and the broader organization optimize against.
Build the eval datasets, golden traces, and regression harnesses that catch: quality changes before they hit customers, and make those assets reusable by every team that is building dashboards and widgets through agents
Drive measurable improvements to retrieval relevance, tool-selection: accuracy, and context efficiency, partnering closely with the engineers on the team.
Provide technical leadership across the Dashboards team and the broader: organization through design reviews, working groups, and mentorship.
You have a BS/MS/PhD in a scientific field, or equivalent experience.
+ years of relevant engineering or applied science experience, including time as a technical lead.
What they're looking for
- Proven track record of leading ML or GenAI initiatives in a product-driven: environment, from research through production.
- Significant experience with evaluation, experimentation, or measurement of ML systems at scale.
- You bring a strong product mindset and are comfortable driving initiatives across cross-functional teams.
- You thrive in ambiguity and can make sound technical calls when the path isn’t yet defined.
