Azure Databricks Engineer - Scala & PySpark

About the Role

This position is focused on building and improving data engineering solutions on Azure Databricks. The work involves creating reliable data pipelines, shaping data for analytics, and supporting the shift from Hadoop-based systems to Azure-based platforms.

Responsibilities

• Design and develop scalable ETL pipelines that move data from multiple sources into data lakes or data warehouses.
• Support the migration journey from Hadoop to Azure and help define how data is used across different portfolios.
• Create and maintain data models that improve how data is stored, accessed, and analyzed.
• Bring together data from databases, APIs, streaming systems, and third-party tools while maintaining consistency and accuracy.
• Clean, transform, preprocess, and enrich raw data so it is ready for reporting and analysis.
• Build and manage data warehouse or data lake solutions for structured and unstructured data at scale.
• Apply data governance and security practices to protect sensitive information and support privacy and compliance needs such as GDPR or CCPA.
• Track performance and tune pipelines and queries to remove bottlenecks and improve processing speed.
• Work closely with data analysts and data scientists to deliver dependable datasets that support analysis and modeling.

Key Skills and Requirements

• Strong coding and scripting ability in Python, SQL, or Scala, with experience automating ETL and data pipeline work.
• Hands-on exposure to Azure cloud services, Azure Databricks, SQL, Spark, and Scala or Python.
• Working knowledge of Jira, GitHub, and DevOps-related tools.
• Understanding of data integration approaches and ETL frameworks such as Apache Kafka, Apache Spark, Apache Airflow, or Talend.
• Knowledge of data modeling, dimensional modeling, and database schema design.
• Experience with big data ecosystems such as Hadoop, Hive, HBase, or Cassandra.
• Comfort with cloud platforms and their services for storage, processing, and analytics.
• Awareness of data governance, security practices, and compliance requirements.
• Strong analytical thinking and problem-solving ability with attention to detail.
• Good communication and collaboration skills for working with technical and non-technical stakeholders.

Eligibility

Applicants must be any graduate.

Preferred Candidate Profile

The source did not provide additional details in this section.