Senior Python Engineer for AI Coding Agent Assessment (Freelance, Part-Time, Onsite)

Overview

Mindrift is connecting experts with project-based AI initiatives for leading technology firms, focusing on testing, assessing, and enhancing AI systems. This is a project-based role rather than permanent employment.

Project Summary

The role centers on developing a data set to assess AI coding agents' capabilities in handling real developer tasks.

Key Responsibilities

• Create authentic developer environments mimicking a virtual company with codebases, infrastructure, contextual materials such as tickets, documentation, and conversations that simulate a believable development history.
• Design tasks starting from intermediate states in these environments; define prompts, clarify what qualifies as "solved," and ensure each task is feasible for an AI agent.
• Develop testing suites to validate agent solutions—ensuring all valid implementations pass while incorrect ones are rejected, maintaining balanced strictness.
• Refine tasks and tests based on quality assurance feedback by reviewing agent outputs, diagnosing issues, and adjusting evaluations for fairness and reliability.

Clarifications on Role Scope

• This position does not involve data labeling.
• It is not focused on prompt engineering.
• Coding from scratch is not required; the AI agent generates most of the code while you provide guidance and evaluation.

Required Qualifications

• A minimum of 8 years in software development.
• Expertise primarily in Python (FastAPI), plus JavaScript/TypeScript (React), Docker, PostgreSQL, Kafka, and Redis.
• Proven experience in writing functional and integration tests.
• English proficiency at or above B2 level.

Challenges

Developing tasks that truly challenge state-of-the-art coding models is complex. Understanding failure points and crafting scenarios that distinguish between good and poor solutions is imperative. Tests must accept all valid solutions without allowing incorrect ones, which requires fine-tuned test design.

Working Methodology

The process involves application, qualification, project onboarding, task completion, and compensation.

Time Commitment

Task completion is estimated at around 30 hours each, with flexibility to manage your own schedule within deadlines.

Compensation

Payment can reach up to $200 per hour equivalent, based on experience and speed.

Application Instructions

Please submit your CV in English and specify your level of English proficiency.