Build and optimize scalable batch and streaming data pipelines, data lakes, warehouses, and feature stores across AWS, GCP, and Azure. Enable end-to-end MLOps workflows including model training, deployment, monitoring, retraining, and versioning. Implement infrastructure automation, data governance, security, observability, and cost optimization while partnering with data scientists and ML engineers to productionize machine learning systems.
This is a remote position.
We are seeking a Senior Data Engineer to build scalable, cloud-native data platforms and enable end-to-end MLOps workflows. You will design ETL/ELT pipelines, manage data lakes/warehouses/feature stores, and ensure high-performance, secure, and cost-efficient pipelines for AI/ML and analytics. This role blends Data Engineering + MLOps to deliver production-ready, automated, and reliable ML workflows.
Responsibilities
- Data Pipelines: Design & optimize batch/streaming ETL/ELT pipelines at scale.
- Platforms: Build/manage data lakes, warehouses, feature stores for ML/BI workloads.
- MLOps: Enable model training, deployment, CI/CD, monitoring, retraining, versioning using SageMaker (AWS), Vertex AI (GCP), Azure ML.
- Streaming: Implement real-time pipelines with Kafka, Spark Streaming, AWS Kinesis, GCP Pub/Sub, Azure Event Hubs.
- Automation: Leverage Terraform, CloudFormation, ARM, Kubernetes for infra-as-code & scaling.
- Quality & Governance: Ensure data lineage, metadata, observability, security, compliance, cost efficiency.
- Collaboration: Work with Data Scientists & ML Engineers to productionize ML models across cloud environments.
Requirements
- 5+ years of hands on experience in Data Engineering, Big Data, or Cloud Data Platform roles, working on large scale production systems.
- Strong command of Python and SQL, using them to build and optimize ETL/ELT pipelines.
- Deep working knowledge of distributed data systems (e.g., Spark, Hive, Presto, Dask) for batch and real-time processing.
- Proven track record with cloud-native platforms across AWS, GCP, or Azure — e.g., BigQuery, Redshift, EMR, Databricks — for data storage and analytics.
- Experience designing and maintaining event driven and streaming architectures (Kafka, Pub/Sub, Flink).
- Solid background in data modeling (star schema, OLAP cubes, graph databases) to support BI and analytics.
- Practical exposure to data security, encryption, and compliance frameworks (e.g., GDPR, HIPAA).
Preferred Skills
- Direct experience enabling MLOps workflows building feature stores, managing versioned datasets, or integrating pipelines with ML platforms (SageMaker, Vertex AI, Azure ML).
- Familiarity with real-time analytics systems such as Clickhouse or Apache Pinot.
- Exposure to data observability tools (e.g., Monte Carlo, Databand) to monitor quality, lineage, and reliability.
- Demonstrated ability to build scalable, resilient, and secure data systems that support mission critical applications.
- Interest and experience in supporting AI/ML innovation with robust data infrastructure.
- Strong mindset for automation, scalability, DevOps/MLOps practices, and engineering excellence.
Benefits
- Competitive compensation as per industry standards
- Opportunity to work on enterprise‑scale AI/ML and analytics platforms
- High‑impact role driving cloud‑native and MLOps transformation
- Collaborative, engineering‑driven work culture
- Strong growth path into Lead Data Engineer, ML Platform Engineer, or MLOps Architect roles
Similar Jobs
Blockchain • Analytics
Design, build, and scale high-performance data pipelines and infrastructure (ingestion, transformation, storage, modeling, serving) using ClickHouse, Postgres, Python and dbt. Manage data quality, reliability, and observability at terabyte and 500M+ address scale, collaborate with researchers and product, mentor engineers, and leverage AI tools to accelerate work.
Top Skills:
Ai Tools And AgentsClaude CodeClickhouseCursorDbtMcpsPostgresPythonSQLStreaming Data ArchitecturesWeb3
Gaming
Own and evolve the core data platform: warehouse design, dbt transformations, and production AWS pipelines. Refactor and harden API-driven data flows, ensure observability and testing, partner with backend and product teams, and contribute to data-aware backend features to deliver reliable, product-focused data.
Top Skills:
AWSBigQueryDatabricksDbtPythonRedshiftSnowflakeSQL
Fintech • Software • Analytics • Financial Services
Lead design and ownership of a cloud-native data platform: build scalable batch ETL and ingestion pipelines, develop analytical models (Athena/Redshift/ClickHouse), integrate data workflows with microservices on ECS, ensure data quality/observability, performance tuning, implement IaC and CI/CD, and mentor engineers.
Top Skills:
Amazon AthenaAmazon EcsAmazon Kinesis FirehoseAmazon RedshiftAmazon S3Aws AppflowAws DmsAws GlueAws LambdaAws Step FunctionsCi/CdClickhouseCloudFormationEventbridgeJavaKinesisPythonSparkSQLSqsTerraform
What you need to know about the Dublin Tech Scene
From Bono and Oscar Wilde to today's tech leaders, Dublin has always attracted trailblazers, with more than 70,000 people working in the city's expanding digital sector. Continuing its legacy of drawing pioneers, the city is advancing rapidly. Ireland is now ranked as one of the top tech clusters in the region and the number one destination for digital companies, with the highest hiring intention of any region across all sectors.
