Azure Data Engineer

About Deloitte Technology & Transformation: EAD Engineering Practice

Deloitte's Technology & Transformation: EAD Engineering practice empowers organizations to unlock valuable insights hidden within extensive datasets. Leveraging a global network, the team delivers expert strategic guidance and implementation services to integrate and manage data from diverse sources, transforming it into precise, actionable intelligence that drives informed decision-making and competitive benefits. This practice spans capabilities including business intelligence, visualization, data management, performance measurement, and emerging technologies such as big data, cloud computing, cognitive computing, and machine learning.

Role Overview

As a Senior Consultant within this Technology & Transformation division, you will foster collaborative relationships with both teams and clients, aiming to surpass their expectations. Your responsibilities range from designing and developing solutions using a variety of tools and principles, to configuring robotic process automation workflows that are maintainable and understandable. You will analyze current processes, manage change through structured controls, resolve operational issues promptly, maintain comprehensive documentation throughout development, testing, and production phases, and collaborate closely with process owners to map and optimize automation flows.

Key Responsibilities

• Create, develop, and deploy scalable data solutions employing Databricks technologies including PySpark and Spark SQL, Azure Data Factory, and various Azure data services.
• Implement robust ETL or ELT pipelines that ingest and transform data from multiple sources into data lakes and warehouses.
• Enhance performance and scalability of data processing pipelines through optimization and tuning.
• Develop and refine complex SQL queries to support data extraction, transformation, and analysis tasks.
• Leverage PySpark to process and analyze data at scale effectively.
• Apply strategies such as data partitioning, bucketing, and indexing to improve data retrieval efficiency.
• Integrate structured, semi-structured, and unstructured data from diverse sources, including API interfaces, streaming data, and batch processing streams.
• Promote strong data governance ensuring quality, consistency, and security of datasets.
• Monitor and troubleshoot data pipelines for accuracy and availability.
• Collaborate with data scientists, analysts, and stakeholders to understand requirements and deliver impactful data solutions.
• Partner with DevOps teams to support deployment and operations of data pipelines in production environments.
• Document workflows, pipeline architectures, and processes for knowledge sharing and future improvements.
• Keep documentation on data architecture and modeling continuously updated.

Qualifications

Applicants should possess a Bachelor of Technology (B.Tech) or Bachelor of Engineering (B.E.) degree in any engineering specialization.