The opportunity
Elastic, the Search AI Company, enables everyone to find the answers they need in real time, using all their data, at scale — unleashing the potential of businesses and people. The Elastic Search AI Platform, used by more than 50% of the Fortune 500, brings together the…
What you'll do
Be a primary contributor to our AI strategy, helping validate and test AI: infrastructure, custom solutions and third-party SaaS offerings.
Test Strategy & Execution: Design and implement comprehensive test strategies for AI/ML systems, including accuracy, bias, robustness, and regression testing.
Rubric-Based Evaluation: Design and implement self-contained evaluation tasks, including prompts, supporting files, and detailed grading rubrics to assess AI performance on functional workflows.
Automation & CI/CD: Automate validation suites for agentic/multi-agent systems, integration testing, and CI/CD pipelines for ML models.
Data Validation: Validate that AI/ML models are consuming accurate, authorized, and properly structured data sources; ensuring data quality across training and inference.
Observation & Reporting: Meticulously observe and document AI agent behaviors, producing crisp, precise summaries and reports on model performance and hallucinations.
What they're looking for
- Output Grounding: Validate prompt engineering outputs from a data accuracy standpoint, ensuring responses are grounded in verified data sources.
- Refinement & Iteration: Iterate and refine evaluation tasks and rubrics based on feedback and team collaboration to ensure robust benchmarking methodologies.
- Security & Governance: Ensure all AI data sources and structures meet governance, regulatory, and compliance standards, while implementing best practices for security and data privacy.
- Collaborate with teams from different areas. These areas include IT: Engineering, IT Operations, Data & Integrations, PMO, CRM, Risk & Compliance, and business technology.
