Hands-on senior data engineer to design and build scalable Lakehouse data solutions, develop ETL/ELT and streaming pipelines, tune Spark workloads, manage Delta Lake ingestion and governance, implement infrastructure with Terraform, enforce IAM and data security, monitor pipeline health, collaborate in Agile with data scientists and stakeholders, and explore AI tooling to enhance the platform.
We are working with a leading global Financial Services business to hire a Senior Data Engineer into their growing data engineering practice. This is a hands-on technical role sitting at the centre of a major enterprise data platform build, with scope to influence architecture, drive pipeline development, and contribute to AI initiatives across a complex, high-volume financial data environment.
Responsibilities:
- Design and develop scalable data solutions on a Lakehouse architecture platform, supporting enterprise-wide data processing and analytics
- Build, optimise, and maintain ETL/ELT pipelines and structured streaming workflows for both batch and real-time data ingestion
- Configure and tune clusters and Spark jobs to deliver consistent performance at scale
- Utilise Delta Live Tables and Unity Catalog to manage data ingestion, transformation, and access governance
- Apply IAM best practices and maintain compliance with data security standards across the platform
- Support infrastructure provisioning and resource management using Terraform
- Implement monitoring frameworks covering pipeline performance, data quality, and operational health
- Contribute to code reviews, technical documentation, and team knowledge-sharing
- Work within an Agile delivery model, collaborating closely with data scientists, analysts, and business stakeholders
- Explore emerging technologies and AI tooling to enhance development productivity and platform capability
Requirements
- 6+ years in data engineering, with at least 2 years hands-on experience with Databricks
- Strong Python and Spark programming skills
- Solid AWS experience across core services including S3, Glue, and Lambda
- Deep understanding of data modelling, SQL, and ETL/ELT design patterns
- Experience with Delta Lake, Lakehouse architecture, and Git-based version control
- Demonstrable use of AI tools within a professional development workflow
- Strong communication skills and the ability to work effectively across technical and non-technical teams
Desirable:
- Financial services or fund administration background
- Exposure to AI/ML implementation patterns and real-time data processing frameworks
- Multi-cloud experience beyond AWS
- API development or data governance framework experience
- Experience mentoring junior engineers
Benefits
Salary: to be discussed, depending on experience
Length: Permanent contract
Similar Jobs
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Build and maintain scalable ETL pipelines, integrate data from databases, APIs, streaming platforms, and external providers, and transform large datasets for analytics and machine learning. Establish reusable engineering frameworks, data governance, monitoring, alerting, and reliability practices. Collaborate with analytics and business teams on cloud data migration, predictive model enablement, and healthcare cost reduction. Required skills include SQL, Python, Scala or Java, cloud platforms, Snowflake or Databricks, DevOps, Git, and CI/CD.
Top Skills:
AirflowAWSAzureAzure Data FactoryCi/CdDatabricksDevOpsEltETLGCPGitHadoopJavaKafkaPythonScalaShell ScriptingSnowflakeSparkSQL
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Develop and maintain complex data pipelines and data sources using Snowflake, Airflow, Azure, SQL, and Python. Integrate on-premises and cloud data, build transformation routines, data models, stored procedures, masking solutions, monitoring tools, and automated data quality controls. Create CI/CD pipelines with GitHub Actions and collaborate with architects, analysts, and data stewards to deliver reliable analytics foundations. The role also supports scalable payment integrity data solutions and requires knowledge of data warehousing, integration, and security practices.
Top Skills:
AirflowAzureAzure Blob StorageAzure Data FactoryAzure FunctionsAzure Key VaultCi/CdDockerEltETLGitGithub ActionsGreat ExpectationsJavaKubernetesNon-Relational DatabasesPythonRelational DatabasesScalaSnowflakeSodaSQL
Artificial Intelligence • Information Technology • Consulting
Own the design, delivery, and operation of complex data pipelines, datasets, and platform components. Translate business needs into scalable data models and reliable data products, improve data quality and observability, resolve production issues, and establish reusable engineering tools and standards. Collaborate with product and business stakeholders, contribute to architecture decisions, mentor engineers, and participate in on-call support.
Top Skills:
AirflowDagsterDockerGreat ExpectationsKubernetesPrefectPythonSQLTerraform
What you need to know about the Dublin Tech Scene
From Bono and Oscar Wilde to today's tech leaders, Dublin has always attracted trailblazers, with more than 70,000 people working in the city's expanding digital sector. Continuing its legacy of drawing pioneers, the city is advancing rapidly. Ireland is now ranked as one of the top tech clusters in the region and the number one destination for digital companies, with the highest hiring intention of any region across all sectors.
