Software Engineer - ETL Development

Role Overview

Join a driven and successful team that values inclusivity and high performance. As a Software Engineer focused on ETL development within the Canadian Banking Engineering Client Systems team, you will engage in designing, building, and modernizing enterprise-grade data integration platforms. Your work will involve constructing scalable batch and distributed data pipelines leveraging technologies such as Apache Spark, as well as transforming legacy ETL workflows into modern, cloud-compatible architectures.

Key Responsibilities

• Design, develop, and maintain scalable ETL and data pipelines using Talend and Apache Spark frameworks.
• Lead migration efforts of legacy ETL workloads to updated technologies like Spark.
• Develop and optimize large batch and distributed data processing pipelines.
• Analyze and tune ETL performance concerning CPU, memory, I/O, and runtime to eliminate bottlenecks.
• Create reusable ETL frameworks, components, and orchestration methodologies for enterprise-wide adoption.
• Implement data ingestion, transformation, and quality assurance across structured and semi-structured data.
• Develop complex SQL transformations, stored procedures, and data models.
• Integrate data pipelines with enterprise systems using messaging platforms (Kafka, MQ), APIs, and batch orchestration frameworks.
• Ensure data integrity, lineage tracking, reconciliation, and auditing throughout pipelines.
• Collaborate with architecture and engineering teams to design future-state data platforms and migration strategies.
• Participate in end-to-end integration and migration testing across distributed systems.
• Participate in technical design discussions and provide valuable input to stakeholders.
• Engage with cross-functional teams, including data engineering, application support, and infrastructure.
• Mentor junior developers promoting best practices in ETL design, performance tuning, and data engineering.
• Ensure adherence to coding standards, version control, and continuous integration/deployment using Git-based systems.

Technical Skills and Experience

• Hands-on experience in Talend ETL development.
• Proficiency with Apache Spark (PySpark or Scala) for processing large datasets.
• Expertise in distributed data processing and big data frameworks.
• Strong Unix/Linux scripting skills for ETL orchestration and automation.
• Solid background in relational databases such as DB2, Oracle, or equivalent.
• Advanced SQL knowledge including complex transformations, performance tuning, and stored procedures.
• Familiarity with large datasets and data warehousing concepts.
• Experience in enterprise-level ETL/data pipeline development and data migration projects.
• Competence with batch processing, data ingestion, transformation pipelines, and messaging systems like Kafka or MQ.
• API integration experience and version control knowledge with Git, Bitbucket, or GitHub.
• Background in ETL modernization or platform migration, particularly transitioning from traditional ETL tools (Talend, Informatica, DataStage) to Spark or cloud environments.
• Experience with re-engineering, re-platforming, and optimizing data pipeline performance.

Preferred Qualifications

• Experience in financial services or regulated industries.
• Familiarity with cloud data platforms such as Azure, AWS, or GCP.
• Knowledge of data orchestration tools like Airflow.
• Exposure to real-time or streaming data processing technologies (Spark Streaming, Kafka Streams).
• Understanding of data governance, lineage, and metadata management.
• Experience with data quality frameworks and reconciliation methods.
• Familiarity with event-driven data architecture concepts.

Additional Information

Location: Toronto, Ontario, Canada.

This position is on-site with Scotiabank, a leading financial institution in the Americas. The bank fosters an inclusive environment and is committed to accommodating individuals with disabilities throughout the hiring process. Only candidates selected for interviews will be contacted.