Python (Machine Learning) QA Lead

About the Role

This is a remote, hourly contractor position for a Python and machine learning quality assurance lead. You will oversee the quality, consistency, and performance of trainers across AI training projects focused on Python and ML. The work involves reviewing AI-produced Python code, machine learning workflows, model explanations, and reviewer outputs; checking them against project rules; and delivering clear written feedback to keep standards high.

You will assess work for coding accuracy, ML methodology, statistical soundness, reproducibility, model evaluation quality, data leakage risks, package selection, debugging correctness, readability, maintainability, formatting, instruction adherence, and alignment with project rubrics. The role calls for strong Python and machine learning knowledge, excellent English communication, careful attention to detail, and the ability to coordinate quality processes within distributed technical teams.

This is part of a fast-scaling AI data services organization that supplies training data to major AI companies and foundation-model labs. Your leadership will help make sure the training data is accurate, executable, statistically valid, reproducible, clearly explained, and consistent with client expectations.

The selection journey includes an AI interview, a subject-specific task, and a recruiter interview. There is currently no immediate project assigned for this position; however, if you are a strong fit, you may be contacted first when suitable opportunities open up. You will also gain access to future projects through the expert network.

Employment Details

Job type: Contract

Work mode: Remote

Compensation structure: Hourly

Requirements

• A bachelor’s, master’s, or doctoral degree in Computer Science, Machine Learning, Data Science, Statistics, Mathematics, Engineering, or another closely related quantitative discipline.
• Strong command of English for understanding instructions, collaborating with teams, and writing precise technical feedback.
• At least 3 years of professional experience in Python development, machine learning, data science, ML engineering, model evaluation, research engineering, technical review, or ML teaching.
• Solid knowledge of Python basics, including data structures, functions, classes, iterators, comprehensions, exception handling, virtual environments, package management, testing, and debugging.
• Strong understanding of machine learning concepts such as supervised and unsupervised learning, feature engineering, train/test splits, cross-validation, model selection, data leakage, regression, classification, clustering, metrics, bias-variance, regularization, and reproducibility.
• Ability to judge ML content using detailed rubrics and detect issues such as incorrect methodology, wrong metrics, data leakage, non-reproducible code, invalid assumptions, hallucinated APIs, weak conclusions, or incomplete explanations.
• Familiarity with tools and libraries such as NumPy, pandas, scikit-learn, PyTorch, TensorFlow/Keras, XGBoost/LightGBM, Jupyter, matplotlib, seaborn, MLflow, Hugging Face, SQL, GitHub, Docker, and CI/CD is preferred.
• Prior experience leading or supporting remote teams of trainers, annotators, reviewers, engineers, data scientists, ML researchers, coding mentors, or QAs is strongly preferred.
• Comfort with fast-paced remote collaboration using Discord, Google Sheets, Google Docs, trackers, dashboards, GitHub, and project management tools.
• Strong organizational ability to manage style guides, trackers, FAQs, onboarding resources, honeypots, calibration tasks, and quality documentation.
• Experience in AI training, data annotation, LLM evaluation, code QA, ML QA, or rubric-based technical review is a plus.

Responsibilities

• Monitor quality by sampling Python ML tasks, identifying defects, sharing feedback directly, and escalating repeated or serious issues.
• Review AI-generated Python code, ML pipelines, preprocessing steps, training flows, evaluation logic, debugging responses, and written explanations for accuracy and reproducibility.
• Keep contributors informed through Discord about guideline updates, workflow changes, and Python/ML review expectations.
• Answer questions related to Python syntax, package use, data leakage, validation methods, metrics, statistical assumptions, reproducibility, notebooks, and rubric interpretation.
• Manage trainer and QA activation by messaging inactive contributors, prompting follow-up, and noting availability concerns.
• Develop and update Python ML style guides, trackers, FAQs, examples, honeypots, calibration tasks, and onboarding materials.
• Conduct onboarding and training sessions for Python ML contributors.
• Identify and flag recommendations that are misleading, overly confident, statistically unsound, not reproducible, insecure, or unsuitable for production use.
• Improve processes by spotting recurring quality gaps and designing scalable QA workflows.

Selection Process

The hiring process includes an AI interview, a role-specific task, and a discussion with a recruiter.

Additional Information

There is no immediate project available for this role. Qualified candidates may be contacted first when relevant opportunities arise and may also be considered for future projects through the expert network.