Data Scientist - AI/ML

Company Overview

Gremlin specializes in reliability engineering through chaos experiments, empowering organizations to detect and mitigate system outages before they impact customers. As a leading provider of chaos engineering solutions used by many top enterprises, Gremlin assists software teams in monitoring, testing, enforcing reliability standards, and automating reliability practices across their organizations.

Role Summary

The Data Scientist, AI/ML at Gremlin will leverage extensive machine learning expertise to analyze vast datasets from chaos engineering experiments, driving automated failure analysis and remediation strategies. This position involves collaborating with engineering teams to develop scalable AI-driven systems that enhance system reliability and customer experience for diverse clients ranging from Fortune 500 firms to smaller companies.

Key Duties

• Examine proprietary datasets of millions of chaos tests to identify patterns indicating failures, root causes, and resilience signals within distributed systems.
• Pretrain and fine-tune machine learning models to detect, classify, and interpret incidents uncovered during experiments.
• Design intelligent solutions that offer automated remediation guidance and orchestration by learning from historical experiment outcomes and system behaviors.
• Develop scalable data infrastructure including pipelines and feature stores for processing and serving large volumes of experiment data for both training and inference.
• Work closely with platform engineers and site reliability engineers (SREs) to embed AI-based failure analysis and remediation directly into the core product platform.
• Utilize advanced ML methods such as causal inference, graph machine learning, time-series analysis, and reinforcement learning to enhance model accuracy and actionable insights.
• Translate insights from chaos experiments into AI-powered features that automatically assess blast radius, detect root causes, and speed recovery.
• Research and deploy novel ML approaches including causal AI and agentic systems for transforming raw experimental data into automated and dependable remediation protocols.

Required Qualifications

• Minimum of 5 years professional experience developing and deploying machine learning, preferably related to distributed systems, infrastructure, DevOps, or SRE settings, alongside broader software development experience.
• Proficient in causal inference, graph machine learning, time-series modeling, or reinforcement learning techniques.
• Experienced in constructing data pipelines and feature stores for offline training and live model inference.
• Familiar with and capable of working within agile development environments.
• Strong proponent of rigorous experimentation, model validation, and engineering best practices.
• Excellent communicator and collaborative partner with platform engineers and SREs capable of translating research into product features.
• Capable of decomposing ambiguous challenges into clear, actionable plans and milestones.

Desirable Skills

• Knowledge of chaos engineering, site reliability engineering, or distributed systems.
• Experience with agentic AI systems or deploying large-scale causal inference models in production.
• Familiarity with MLOps tools including model serving, monitoring, and feature store management.
• Comfort working in remote-first teams.
• Experience participating in on-call rotations and incident management processes.

Employment Terms

Sponsorship for employment is not provided. Applicants who do not meet every listed qualification but are strong in certain areas and show interest are encouraged to apply.

Compensation and Benefits

The salary range is anticipated to be between 220000 and 290000 USD annually, with final offers depending on candidate skills, experience, budget, and market data. Gremlin provides competitive packages including 401k matching, equity options, flexible time off, and paid holidays.

Company Culture and Values

Gremlin values people, collaboration, and results. The company respects and aims to nurture diversity, treating all individuals with dignity and empathy. Teamwork and shared goals are fundamental, especially in a remote working environment. Gremlin promotes productivity, accountability, speed over perfection, and continuous improvement. It supports growth and values varied perspectives as key to building a more reliable internet.