Medical Expert AI Evaluation Specialist

Role overview

This is a client assignment through Weekday AI, supporting a major AI research effort centered on healthcare applications. The work is designed for experienced medical professionals who can use their clinical judgment to help train and assess advanced AI systems.

The position focuses on creating realistic medical cases, reviewing AI-generated clinical answers, and offering expert feedback that improves how the models handle diagnosis, treatment choices, and patient care reasoning.

This opportunity is especially relevant for physicians from a range of specialties who want to contribute to the development of healthcare technology.

Compensation and location

Pay is listed at $130 to $180 per hour. Eligible applicants must be located in Canada or the United States.

Key responsibilities

• Build specialty-specific clinical case scenarios that mirror real patient situations.
• Prepare prompts around diagnostic reasoning, treatment planning, risk assessment, differential diagnosis, and guideline-driven decision-making.
• Write reference-quality clinical answers that reflect attending-level judgment and align with current evidence-based practice.
• Review AI medical outputs with structured scoring methods and evaluation rubrics.
• Spot clinical strengths, gaps, missing details, and reasoning mistakes in model responses.
• Provide clear written feedback to help improve accuracy and clinical performance.
• Join onboarding and specialty calibration sessions.
• Coordinate with research and clinical teams to keep evaluations consistent and high quality.

Eligibility and required qualifications

Applicants must meet at least one of the following pathways:

• Attending physicians with a board certification and an active, unrestricted medical license.
• Resident physicians in the final year of an accredited residency program, or recent residency graduates who are board-eligible.
• Fellows who are board-certified or board-eligible in their primary specialty and hold an active, unrestricted medical license.

Physicians from many specialties may apply, including Family Medicine, Pediatrics, Anesthesiology, General Surgery, ENT (Otolaryngology), Physical Medicine & Rehabilitation (PM&R), Occupational Medicine, and other medical or surgical specialties.

Preferred background

• Solid command of evidence-based medicine and clinical guidelines.
• Strong written communication ability.
• Background in medical education, clinical research, peer review, or healthcare quality work.
• Interest in artificial intelligence and healthcare innovation.

Working style and contract terms

• The engagement is fully remote and entirely asynchronous.
• Expected workload is about 20 hours per week, with the possibility of more hours depending on project demand.
• Work hours are flexible and can be completed on your own schedule.
• Payment is issued weekly.
• Onboarding happens on a rolling basis, with continuing project opportunities.
• Assignment length is reviewed periodically based on project requirements and performance.
• This is an independent contractor engagement.
• The work does not require access to confidential or proprietary information from any employer, client, or institution.
• Visa sponsorship is not available.
• Reasonable accommodations are available upon request.
• All qualified applicants will be considered without discrimination based on legally protected characteristics.

Why this role may appeal to you

• Contribute to the future of AI-enabled healthcare.
• Use your clinical expertise to strengthen advanced medical reasoning systems.
• Work alongside AI researchers and healthcare professionals.
• Help shape high-impact tools aimed at improving patient-centered care.