Azure Databricks Lead - Scala PySpark

Role overview

This position is for an Azure Databricks Lead who will guide data platform modernization efforts in Hyderabad/Bengaluru. The role centers on building scalable data solutions, leading migration work from Hadoop to Azure, and helping teams use data effectively across multiple portfolios.

Core responsibilities

• Design, develop, and maintain robust ETL pipelines that move data from multiple sources into data lakes and data warehouses.
• Lead the transition from Hadoop-based systems to Azure and define how data will be used across different portfolios.
• Create and implement data models that support efficient storage, fast retrieval, and effective analytics for varied data volumes and types.
• Bring together data from databases, APIs, streaming tools, and external systems while maintaining consistency and accuracy.
• Clean, transform, and enrich raw data so it is reliable and ready for reporting, analysis, and downstream consumption.
• Build and maintain data warehouse or data lake solutions capable of managing large structured and unstructured datasets.
• Apply governance and security practices to safeguard sensitive information and support compliance requirements such as GDPR and CCPA.
• Track and improve data processing and query performance by identifying bottlenecks and optimizing workloads.
• Work closely with data analysts and data scientists to understand their needs and deliver well-organized, dependable datasets.

Technical skills and experience

• Strong programming and scripting ability in Python, SQL, or Scala for automation, ETL, and data handling.
• Hands-on experience with Azure cloud services, Azure Databricks, SQL, Spark, and either Scala or Python.
• Working knowledge of Jira, GitHub, and DevOps-related tools and practices.
• Exposure to data integration and ETL frameworks such as Kafka, Spark, Airflow, or Talend.
• Understanding of data modeling, dimensional modeling, and database schema design.
• Experience with big data ecosystems such as Hadoop, Hive, HBase, or Cassandra.
• Familiarity with cloud-based data storage, processing, and analytics platforms.
• Knowledge of data governance, data protection, and regulatory compliance practices.
• Strong analytical thinking and troubleshooting skills with attention to detail.
• Clear communication and collaboration skills for working with technical and non-technical stakeholders.

Eligibility

Any graduate may apply for this opportunity.

Additional information

The sharing note in the source ends with a warm regards message from Bhuvaneswari GS.