Senior Python and PySpark Data Engineer with AWS Expertise

About the Role

We are looking for an experienced Senior Data Engineer specializing in Python and PySpark to develop, deploy, and optimize scalable data platforms and pipelines using AWS technologies. This position plays a pivotal role in constructing modern data lakehouse architectures enabling advanced analytics and decision-making.

Responsibilities

• Design, develop, and maintain analytical data models and key metrics utilizing modern scalable data platform technologies.
• Architect and implement data lakehouse solutions utilizing AWS services.
• Build, optimize, and maintain robust data pipelines using PySpark, Python, and AWS Glue.
• Ensure data systems are high performing, reliable, and scalable.
• Collaborate with business stakeholders to translate their requirements into effective data solutions.
• Work with architecture and operations teams to enforce best practices, standards, and governance.
• Monitor and troubleshoot data workflows ensuring data quality and operational efficiency.
• Adhere to Information Security Management policies and procedures.

Required Skills and Experience

• At least 5 years of professional software development experience, emphasizing analytical platform development.
• Minimum 3 years of practical experience in data platforms and big data ecosystems.
• Strong command of big data tools and technologies including AWS EMR, Apache Spark, PostgreSQL, DynamoDB, and Apache Iceberg.
• Proficient in Python and Spark programming for scalable data systems.
• Experience developing low-latency, scalable data solutions (e.g., DynamoDB, Aurora, PostgreSQL, Redshift).
• Proven expertise designing and implementing AWS-based data lakehouse architectures.
• Hands-on experience with AWS services such as Step Functions, Lambda, and EventBridge.
• Experience working within Agile development frameworks.

Preferred Additional Skills

• Knowledge of AWS Kinesis Firehose, SQS, Bedrock, and SageMaker.
• Experience with real-time data streaming, event-driven architectures, and familiarity with machine learning data pipelines.

Soft Skills

• Ability to quickly learn and adapt to new technologies.
• Strong analytical thinking and problem-solving capabilities.
• Ability to work independently and collaboratively as part of a team.
• Ability to multitask effectively in a fast-paced and dynamic environment.
• Excellent written and verbal communication skills.

Additional Information

• Work location: Remote.
• Work shifts typically follow general hours with overlap to US timings; working hours may extend until 10:30 PM to 11:00 PM IST, with flexible morning start times.
• Immediate onboarding expected.
• Qualification: Graduate degree required.