Model Validator for Agentic AI

Role Overview

The position involves validating AI agent models by developing evaluation datasets to benchmark their reasoning processes and performance.

Key Responsibilities

• Create evaluation sets that measure agent performance based on reasoning pathways.
• Conduct adversarial testing by providing conflicting instructions to challenge the agent.
• Perform scholastic regression testing to identify variations in agent behavior over time.
• Verify accuracy of agent calls to external APIs and databases.
• Analyze the reasoning chains of the agent and pinpoint divergences from business requirements documentation.
• Apply judge large language models (LLMs) to assess output quality.
• Use Python and relevant frameworks such as DeepEval and Langsmith proficiently.
• Conduct semantic debugging by reviewing the agent's thought process traces.

Preferred Qualifications and Skills

• Background as Software Development Engineer in Test (SDET) for coding and testing expertise, including evaluation development.
• Strong knowledge of data testing and SQL querying for validation purposes.
• Domain experience relevant to specific roles is considered advantageous.