Lead Data Engineer

About the role

We are looking for an experienced Lead Data Engineer to join a high‑impact team supporting financial institutions in Dublin. The role will drive the design, build and operation of a Databricks‑based lakehouse on AWS, enabling data scientists, investigators and product teams to detect criminal behaviour with confidence.

Key responsibilities

• Own end‑to‑end design, development and optimisation of scalable Spark/PySpark pipelines on Databricks (batch and streaming).
• Define and enforce Lakehouse/Medallion architecture standards (Bronze/Silver/Gold) including governance, lineage, quality SLAs and cost controls.
• Architect and maintain secure AWS data infrastructure (S3, IAM, Glue, Lake Formation, KMS, Lambda, Step Functions, EKS/EC2).
• Lead data ingestion using Apache NiFi, APIs, SFTP/FTPS and onboard diverse internal and external datasets.
• Implement robust orchestration with Airflow, Databricks Workflows and Step Functions, ensuring observability and reliability.
• Champion data quality, reliability and observability through SLIs/SLOs, alerting and runbooks.
• Embed metadata and lineage (Unity Catalog, Glue, OpenLineage) for auditability and regulatory transparency.
• Drive CI/CD and Infrastructure‑as‑Code practices (Terraform/CloudFormation) for data assets across environments.
• Mentor engineers on Spark performance, Delta Lake optimisation and cost‑performance trade‑offs.
• Collaborate with data science, product, security and compliance teams to deliver production‑grade data solutions.

Required profile

• Expert‑level SQL with hands‑on experience in Databricks, Snowflake, Python and PySpark.
• Proven production experience building and optimising large‑scale Spark pipelines (Delta Lake, Photon, cluster tuning).
• Strong expertise in the AWS data ecosystem, including security, networking, encryption and cost optimisation.
• Hands‑on experience with orchestration tools such as Airflow, Databricks Workflows and Step Functions.
• Solid background in CI/CD, Git workflows and IaC (Terraform or CloudFormation).
• Deep understanding of data governance, lineage and compliance (PII/PCI, retention, access controls).
• Demonstrated ability to lead, mentor and influence technical and non‑technical stakeholders.
• Pragmatic, delivery‑focused mindset with experience in incident management and on‑call readiness.

Required skills

• SQL
• Python
• PySpark / Spark
• Databricks
• Snowflake
• Delta Lake, Photon
• AWS (S3, IAM, Glue, Lake Formation, KMS, Lambda, Step Functions, EKS, EC2)
• Apache NiFi
• Airflow
• Databricks Workflows
• Unity Catalog, OpenLineage
• Terraform / CloudFormation
• Git, CI/CD