The opportunity
Klaviyo's Product Data Science team plays a critical role in shaping how we understand, measure, and improve product and business performance. We partner closely with Product, Engineering, Finance, and Go-to-Market teams to turn complex data into actionable insights that inform…
What you'll do
Own agent measurement end to end across Composer and Customer Agent,: developing comprehensive measurement frameworks that connect adoption, engagement, agent performance, customer value, and revenue impact.
Define what successful agent performance looks like by partnering with: Product and Engineering to establish meaningful measures of agent effectiveness, task completion, customer outcomes, failure rates, and the trade-offs between quality, cost, and business value.
Build scalable agent evaluation frameworks by combining conversation data,: agent traces, tool-call outcomes, human evaluation, and LLM-based classification to assess quality, identify failure patterns, and understand how changes to models and agent behavior influence product performance.
Measure causal impact and incremental value through experimentation and: advanced statistical methods, including quasi-experimental approaches such as difference-in-differences, matched comparisons, staged rollouts, and synthetic controls, to determine how AI agents influence customer behavior, retention, and revenue.
Connect agent performance to business strategy by partnering with Finance and: Product teams to evaluate usage-based pricing, packaging, credit economics, model costs, conversion, and the financial implications of product investments.
Establish trusted metrics and analytical standards by defining consistent: measurement methodologies, governed data models, and evaluation practices that enable teams to make informed decisions as AI products and underlying technologies evolve.
What they're looking for
- You have 10+ years of experience in data science, product analytics, applied: statistics, or a related quantitative discipline, including recent hands-on experience measuring AI, machine learning, or LLM-powered products in production.
- You bring deep expertise in product measurement, experimentation, statistical: modeling, and causal inference, with the ability to select and defend rigorous analytical approaches when traditional randomized experiments aren't feasible.
- You have experience evaluating AI systems in production, including designing: evaluation frameworks, analyzing agent interactions or conversations, leveraging human labeling or LLM-as-judge methodologies, and understanding how model, prompt, and tool changes influence product outcomes.
- You're highly proficient in SQL and Python or R, with experience developing: production-quality analytical solutions and familiarity with dbt and modern analytics engineering practices.
- You take an AI-first approach to your work, actively leveraging coding: agents, LLMs, and emerging tools to accelerate analytical workflows while understanding their limitations and ensuring the accuracy and reliability of their outputs.
- You've demonstrated technical leadership by setting direction on complex: analytical initiatives, mentoring other data scientists, reviewing methodologies, and influencing technical standards across teams.
- You combine statistical rigor with strong product and business intuition,: understanding how customer engagement, AI product performance, usage-based pricing, and unit economics translate into meaningful customer and business outcomes.
- You're comfortable tackling ambiguous, emerging problems where established: metrics or methodologies may not yet exist, and can develop practical frameworks that balance analytical rigor with business needs.