GCP Lead / Senior Data Engineer - Healthcare Data Pipelines

About the Role

We are seeking a skilled GCP Lead or Senior Data Engineer to design, develop, test, and maintain cloud-based healthcare data pipelines and backend systems on Google Cloud Platform. This position involves close collaboration with senior technical staff to efficiently manage healthcare file processing through Google Cloud Storage (GCS), implementation of event-driven workflows using Pub/Sub and Eventarc, and development of data processing solutions with Cloud Run and Dataflow. The role also includes loading data into BigQuery, relational and document databases, and integration with FHIR Store alongside downstream healthcare applications.

Key Responsibilities

• Create, deploy, and maintain scalable healthcare data pipelines using GCP technologies.
• Handle healthcare data ingestion from GCS, routing through Pub/Sub, Eventarc, Cloud Run, Dataflow, BigQuery, and FHIR Store.
• Design and implement schemas for relational and cloud databases tailored for operational, analytical, and document-based workloads.
• Develop APIs to monitor pipeline statuses, file processing, data validation, and system health of various components.
• Manage database models for pipeline metadata, auditing, orchestration configurations, and search indexing.
• Build both batch and streaming data pipelines that include error handling such as retries and dead-letter queues.
• Integrate complex healthcare data including structured, semi-structured, and unstructured formats.
• Support advanced features like semantic search, named entity recognition (NER), and embeddings for healthcare documents.
• Optimize databases in terms of performance, cost, and indexing strategies with attention to storage efficiency.
• Implement operational best practices including monitoring, logging, alerting, and security with PHI awareness.
• Provide mentorship to junior and mid-level engineers, leading engineering best practices.
• Troubleshoot and resolve production incidents using cloud native logging and monitoring tools.
• Participate actively in code reviews and help maintain high-quality engineering standards.
• Collaborate with senior engineers to build scalable, secure, and maintainable cloud data infrastructures.

Required Qualifications

• Minimum of 5 years of hands-on experience with data engineering and database management, including relational and document databases.
• Strong capabilities in SQL, data modeling, query optimization, and data validation techniques.
• Proficiency with Google Cloud Platform services, notably GCS, Pub/Sub, Cloud Run, Dataflow, and BigQuery, with 5+ years of cloud experience.
• Advanced programming skills in Python, with a solid grasp of object-oriented design, modular development, error handling, logging, unit testing, and package management.
• Experience with RDBMS such as PostgreSQL, MySQL, SQL Server, or Oracle.
• Expertise in building and supporting batch, event-driven, API-linked, and streaming data workflows.
• Competence in integrating external APIs securely, handling authentication, error management, and monitoring.
• Working knowledge of Docker for containerized app development and Git/GitHub workflows including branching, pull requests, and code reviews.
• Familiarity with CI/CD tools such as GitHub Actions or Cloud Build.
• Strong troubleshooting skills to detect and resolve issues across applications, pipelines, databases, and cloud services using logs and monitoring solutions.
• Ability to interpret architecture documents, technical specs, and development tickets to implement solutions accurately.

Preferred Attributes

• Cloud certifications, preferably Google Cloud or database-related credentials.
• Experience with Google Cloud Healthcare API and knowledge of FHIR R4, HL7, CCDA, and healthcare interoperability standards.
• Exposure to advanced search technologies including embeddings, semantic search, ElasticSearch, OpenSearch, or vector databases.
• Familiarity with AI/ML platforms such as Gemini, Vertex AI, OpenAI API, or LLM integrations.
• Experience with Infrastructure-as-Code tools like Terraform.
• Awareness of HIPAA, PHI regulations, and healthcare data security compliance.

Eligibility and Education

Applicants must hold a graduate degree in any discipline.