Lead Data Engineer

Our Purpose

Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential.

Title and Summary

Lead Data Engineer

Who is Mastercard?
Mastercard is a global technology company in the payments industry. Our mission is to connect and power an inclusive, digital economy that benefits everyone, everywhere by making transactions safe, simple, smart, and accessible. Using secure data and networks, partnerships and passion, our innovations and solutions help individuals, financial institutions, governments, and businesses realize their greatest potential.
Our decency quotient, or DQ, drives our culture and everything we do inside and outside of our company. With connections across more than 210 countries and territories, we are building a sustainable world that unlocks priceless possibilities for all.
Overview
We’re looking for a Lead Data Engineer to join our SME Data team, which powers high-impact data solutions across both on-prem and cloud platforms. As a Lead Data Engineer, you’ll architect, build and optimize robust data pipelines and platforms and lead a team of engineers working at the intersection of Big Data and Cloud innovation.
This role is ideal for someone passionate about solving large-scale data engineering challenges and who thrives in a hybrid environment spanning on-prem Hadoop clusters and modern cloud-native technologies.
Role
As a Lead Data Engineer, you will:
Design, build, and maintain scalable data pipelines and architectures for both on-prem (Hadoop ecosystem) and Databricks Cloud environments.
Lead and mentor a team of data engineers—providing technical guidance, conducting code reviews, and fostering team growth.
Ensure data quality, reliability, and governance across all stages of data processing.
Work closely with data scientists, analysts, and business stakeholders to deliver impactful data solutions.
Drive best practices in CI/CD using Jenkins and Git and manage deployment pipelines for seamless production releases.
Optimize workflows using Airflow for orchestration and ensure performance across HDFS, Hive, Impala, and Spark.
Utilize Databricks features including Delta Tables, SQL Warehouses, DLT pipelines and Unity Catalog for secure, performant data management.
Champion team collaboration and continuous improvement in both engineering practices and team development.
All About You
The ideal candidate for this position should have:
Expertise in Spark (PySpark/Scala), Hadoop, Hive, Impala, and Airflow.
Strong coding skills in Python or Scala and a solid understanding of distributed data processing.
Hands-on experience with Databricks on cloud platforms (preferably AWS), including S3, Delta Lake, DLT, SQL Warehouses, Unity Catalog, and Databricks Asset Bundles (DAB).
Strong experience with HDFS storage and handling large-scale structured and semi-structured datasets.
Proficiency in Git-based development workflows and CI/CD pipelines using Jenkins.
Leadership experience, with a track record of mentoring, guiding, and supporting engineering teams.
Deep understanding of distributed systems, data partitioning strategies, and performance optimization.
Strong problem-solving skills and a delivery mindset with the ability to work cross-functionally.
Nice to have:
Experience with data governance and security best practices in regulated environments.
Familiarity with modern DataOps practices and observability tools.
Corporate Security Responsibility
Every person working for, or on behalf of, Mastercard is responsible for information security. All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and therefore, it is expected that the successful candidate for this position must:
Abide by Mastercard’s security policies and practices;
Ensure the confidentiality and integrity of the information being accessed;
Report any suspected information security violation or breach, and
Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.

Corporate Security Responsibility

All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:

• Abide by Mastercard’s security policies and practices;

• Ensure the confidentiality and integrity of the information being accessed;

• Report any suspected information security violation or breach, and

• Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.