Freelance Agent Evaluation Engineer

Role Overview

Mindrift specializes in connecting experts with project-specific AI engagements for top-tier technology firms, focusing on assessing and enhancing AI systems. This is a project-based role, not a permanent position.

Key Responsibilities

• Develop authentic developer environments simulating a company’s ecosystem, including codebases, infrastructure, and related context like tickets, documentation, and communication histories.
• Create tasks originating from intermediate environment states by preparing detailed prompts, establishing success criteria, and ensuring tasks are solvable by AI agents.
• Design rigorous test suites to validate agent-generated solutions by accepting all valid approaches while rejecting incorrect ones, striking a balance between strictness and flexibility.
• Iteratively refine tasks and associated tests based on quality assurance outcomes, analyzing agent performance and adjusting evaluations for fairness and robustness.

What This Role Does Not Include

• This position is not related to data labeling activities.
• It does not involve prompt engineering.
• It does not require writing code from scratch as the AI agent predominantly produces code; your focus is guidance and evaluation.

Candidate Profile

• At least five years of professional software development experience.
• Strong expertise in Python (including FastAPI), JavaScript/TypeScript (notably React), as well as Docker, Postgres, Kafka, and Redis.
• Proficient in authoring functional and integration tests.
• English language proficiency at B2 level or above.

Challenges of the Role

Designing tasks that effectively challenge advanced AI coding models is complex. Understanding the subtle nuances where models fail and devising scenarios that distinctly separate correct from incorrect solutions requires deep insight. Crafting test cases that accept all legitimate solutions without false rejections adds further difficulty.

Work Process

The workflow involves applying, passing required qualifications, joining projects, completing assigned tasks by set deadlines, and receiving compensation upon acceptance.

Time Commitment and Compensation

Task completion typically requires around 20 hours each, although this varies by complexity. You determine your own working schedule, provided deadlines are met. Payment is up to $50 per hour equivalent, contingent on experience and productivity.

Application Instructions

Please submit your CV in English and specify your English proficiency level.