GCP Data Engineer - Apache Kafka & Event-Driven Systems (US Shift)
Kalyan, Maharashtra, India · Tempo pieno
Sii il primo a candidarti
- Esperienza
- 5+ anni
- Stipendio
- INR 1,200,000 – INR 2,200,000 / year
- Aperture
- 1
- Pubblicato
- 2 ore fa
- Modalità di lavoro
- In ufficio
- Istruzione
- B.Tech / B.E.
- Requisiti di ammissibilità
- Candidates with a B.Tech or B.E. degree in any specialization are eligible to apply.
- Riprendere
- È necessario candidarsi
Dove lavorerai
Descrizione del lavoro
About the Role
Our client is transforming their Legacy platform into a modern event-driven system leveraging Google Cloud and Apache Kafka. This greenfield project involves designing and constructing a platform handling event ingestion from Legacy ECC systems and external carriers, with a Kafka streaming backbone processing approximately 10 million daily events. The platform includes a processing engine for rules and a compliant seven-year event archive stored across BigQuery and Cloud Storage.
As a core engineer, you will have full ownership and collaborate closely with a senior Legacy systems expert for domain knowledge, while you provide cloud, Kafka, and engineering expertise.
Key Responsibilities
- Designing and managing a Kafka cluster on Google Cloud, including configuration of topics, partitions, security, and retention policies.
- Developing an HTTPS event ingestion bridge using Cloud Run and Python to receive events from Legacy and external systems.
- Implementing a Kafka consumer-based rules engine in Python that processes events, manages state with Firestore, and sends triggers back to the Legacy system.
- Building Kafka Connect pipelines to archive all events in BigQuery and Cloud Storage with long-term compliance requirements.
- Setting up monitoring, alerts, dead-letter queues, and reconciliation processes ensuring zero data loss across the pipeline.
Required Qualifications and Experience
- A minimum of 5 years in software or data engineering focusing on backend or data platform development.
- At least 2 years hands-on experience with Apache Kafka across topics, partitions, consumer groups, offsets, and delivery guarantees.
- 2+ years working with Google Cloud Platform, particularly Cloud Run or GKE, IAM, VPC, and Cloud Monitoring.
- Proficient in Python for production development including REST APIs, robust error handling, and testing.
- Strong SQL skills with BigQuery, including working with partitioned and clustered tables.
- Experience in building resilient integrations featuring idempotency, retries, dead-letter queues, and ensuring at-least-once message processing.
- Ability to manage systems end-to-end covering design, development, deployment, monitoring, and documentation.
Preferred Skills
- Experience with Google Cloud Managed Service for Apache Kafka or Kafka Connect sink connectors.
- Familiarity with NoSQL databases such as Firestore or Bigtable.
- Knowledge of infrastructure-as-code tools like Terraform.
- Understanding of regulated or compliant data retention practices including WORM and audit trails.
Work Location & Schedule
- Position requires full-time office presence in Kalyan, India.
- The shift is fixed night hours from 6:30 PM to 3:30 AM IST, Monday through Friday, aligned with US business timings.
- A night-shift allowance will be provided.
Educational Eligibility
Applicants should have a B.Tech or B.E. degree in any specialization.