Databricks and PySpark Engineer - 6+ Years Experience - Pune - Immediate Joiners

About the Role

HCL Technologies, a global leader in next-generation technology services, is seeking experienced Databricks Engineers skilled in PySpark for an opportunity based in Pune. The ideal candidates will play a pivotal role in engineering scalable and efficient data solutions by leveraging cutting-edge tools and cloud platforms.

Primary Responsibilities

• Architect, build, and maintain robust data pipelines and ETL workflows utilizing Databricks and PySpark frameworks.
• Design and execute complex ETL processes to ingest, transform, and load data efficiently from various sources, ensuring seamless integration across systems.
• Optimize and continuously monitor data systems to guarantee high performance, scalability, and reliability.
• Uphold data quality, security protocols, and regulatory compliance standards throughout processes.
• Create comprehensive documentation related to data architecture, ETL processes, data mappings, and any procedural modifications.
• Collaborate closely with data scientists, analysts, and business stakeholders to meet data requirements and deliver dependable data solutions.
• Diagnose and resolve data-related issues proactively, identifying and addressing system inefficiencies promptly.
• Adopt and enforce best practices in data transformation/ingestion methods, including change data capture (CDC), schema evolution, and advanced error management.
• Provide mentorship to team members, delegate tasks efficiently, and contribute specialized knowledge to support team success.
• Remain updated on emerging technologies, new features in Databricks, and industry standards to continuously improve data capabilities.

Required Qualifications and Skills

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Technology, or a closely related discipline.
• At least 5 years of solid professional experience in data engineering or similar fields.
• Demonstrated hands-on expertise with Databricks and PySpark in a data engineering capacity.
• Proficiency in programming languages such as SQL and Python is essential.
• Experience working with cloud platforms, with a preference for Microsoft Azure.
• Familiarity with data visualization or virtualization tools like Denodo or Tableau is beneficial.
• In-depth understanding of CDC ingestion techniques, including watermarks, change tables, and log-based ingestion methods.
• Strong abilities in error handling, observability, and optimizing performance in data processes.
• Excellent analytical thinking, problem-solving skills, and communication capabilities.
• Ability to work independently and as part of a team, with proven leadership and mentoring experience.
• Fluency in English; knowledge of German is advantageous.
• Relevant certifications in Microsoft Azure or Databricks are highly preferred.

Additional Requirements

• Track record of managing complex tasks while supporting and guiding peers, delegating responsibilities effectively, and fostering overall team achievements.
• A strong enthusiasm for ongoing learning and career development.

Location and Contact

This position is based in Pune, India. Interested candidates should prepare details regarding total experience, Databricks exposure, Python/PySpark experience, current and expected compensation, notice period, and location preferences.