Apache Spark Scala Engineer - Chennai (6 to 10 Years Experience)

About the Role and Company

HCL Technologies, a leading global IT services company renowned for innovation and customer centricity, is seeking experienced Data Engineers proficient in Apache Spark and Scala to join their Chennai team. The company boasts a strong culture of invention, social responsibility, and a vast global presence with over 197,000 employees. This opportunity centers around working with advanced big data technologies to build scalable, robust data solutions.

Primary Responsibilities

• Lead and oversee technical teams to design, develop, and deploy data pipelines and solutions using Apache Spark, Scala, and Python, ensuring robustness and scalability.
• Offer expert guidance to troubleshoot complex issues, optimize Spark applications, and enhance system performance.
• Collaborate effectively with business stakeholders to translate requirements into technical implementations aligned with business goals.
• Implement best coding, testing, and deployment practices throughout the full software development lifecycle in an Agile environment.
• Conduct code reviews and perform performance tuning to maximize efficiency and reliability of data processing jobs.
• Diagnose and resolve technical challenges related to Spark jobs, big data infrastructure, and data pipelines for seamless operational workflows.
• Keep updated with the latest industry trends and technological advancements in big data and related ecosystems to innovate and improve solutions.
• Mentor team members, fostering skill development, knowledge sharing, and adherence to best practices in big data engineering.
• Manage end-to-end data pipeline development including complex SQL queries, data modeling, and deployment in production ensuring high data quality and integrity.

Required Skills and Qualifications

• Extensive hands-on experience in developing, optimizing, and maintaining Apache Spark applications and big data pipelines.
• Strong proficiency in Scala programming with practical experience applying it to large scale data solutions.
• Skilled in Python programming for big data integration, including deployment with Spark.
• Expert in complex SQL query writing, data modeling, and effective management of big data storage and retrieval.
• Demonstrated experience in leading technical teams, managing projects, and driving collaborative development efforts.
• Familiarity with software development lifecycles, Agile processes, and best practices in coding, testing, and production deployment.
• Ability to perform performance tuning, troubleshooting, and optimization to resolve bottlenecks and elevate data pipeline efficiency.
• Strong problem-solving, communication, and interpersonal abilities.
• Holds a Bachelor's or Master's degree in Computer Science, Engineering, or a related discipline.
• Possession of relevant certifications in Apache Spark, Scala, or Python will be advantageous.

Eligibility

Application is open to any graduate who meets the experience and skill requirements.

Additional Information

Location: Chennai, India

Interested candidates are encouraged to share their total experience, experience in Apache, Spark, Scala/Python, current and expected salary, notice period, current and preferred location for consideration.