Skip to main content
Softblues
Back to Projects

Cloud Lakehouse with Change-Data-Capture Ingestion

Senior Data Engineer & Architect2022-2023Dany D.
DD
Dany D.

Lead Data & ML Engineer

Data Engineer & Big Data

Key Expertise

Declarative Data EngineeringAdvanced Stream ProcessingReal-time CDC PipelinesMedallion Lakehouse DesignMLOpsAgentic AI Architecture

Experience

8+ years

Timezone

CET (UTC +1)

Skills

AI / ML

statsmodelsLangGraphMLflowLightGBMLangChain

Languages

Python

Databases

DatabricksAuto LoaderPostgreSQLUnity CatalogDelta Lake

Infrastructure

Azure DevOps PipelinesKafkaAWSTerraformAzurebanditGitLab CIKubernetescentralized loggingmypyDatadogruff

Frameworks

typed configuration frameworkPydanticpytestdeclarative streaming pipelinesPySparkDatabricks Asset BundlesScikit-learnDatabricks Workflows

Integrations & Protocols

Model Context Protocollog-based CDC connectorsPower BI
7-day risk-free trial
Response within 24 hours
View Full Profile

Overview

The project involved designing and delivering a cloud-native data platform for a financial-services institution moving off a fragmented legacy ETL stack. The platform is built around a medallion lakehouse on Databricks, declarative streaming transformations for the silver layer, and log-based change-data-capture from operational relational sources via a managed Kafka service. A config-driven pipeline layer decouples table onboarding from code changes, and a data-quality engine splits each stream into a clean sink and a quarantine sink for audit and remediation.

Achievements

Replaced hand-written, per-table pipeline code with a declarative JSON configuration model - onboarding a new dataset becomes a configuration exercise rather than an engineering project. Materially reduced downstream data-quality incidents through per-column mandatory and type-cast validation, with full lineage of failed records into a quarantine table. Introduced log-based CDC with exactly-once delivery semantics, eliminating the polling overhead and latency of the previous arrangement while preserving schema evolution through the schema registry.

Responsibilities

  • Architected the bronze/silver/gold layout on cloud object storage and the declarative streaming transformation pipeline, where each silver table is materialised from a configuration entry declaring schema, constraints, paths, and source format.
  • Built the data-quality engine (per-column mandatory checks, type-cast verification, row-level faulty-record flagging) and the dual-stream writer pattern that sinks valid and faulty rows into separate Delta destinations for downstream reconciliation.
  • Implemented the CDC ingestion path: managed Kafka as the transport, log-based source connectors against the relational systems, an object-storage sink connector for landing, and a schema registry for evolution and serialization governance.
  • Wrote the Terraform IaC covering resource group, redundant object storage with container layout, secret store with role-based access control and managed secrets for platform credentials, private virtual network with service endpoints, workspace, and orchestration layer.
  • Established the CI/CD pipeline, a local test harness with session-scoped Spark fixtures, and a dev path mirroring the production storage topology so engineers can iterate without hitting cloud resources.

Technologies Used

DatabricksPySparkDelta Lakedeclarative streaming pipelinesAuto LoaderKafkalog-based CDC connectorsKubernetesTerraformAzurecentralized loggingAzure DevOps Pipelines
DD

This project was delivered by

Dany D.

View Full Profile

Ready to Build Your AI Team?

Get matched with the right AI experts for your project. Book a free discovery call to discuss your requirements.

We respond within 24 hours.