Monthly Water Meter Closing System
An enterprise-scale data processing architecture handling 15M+ monthly meter readings, cutting processing latency from 168 hours to 2 and generating €13M in annual operational savings.
Records / month
15.2M
Processing latency
168h → 2h
Annual savings
€13.0M
Throughput: last 12 runs
+97% fasterAirflow DAG runs
Illustrative interface with fictitious sample data, shown in place of confidential client screens.
The problem
Monthly meter-closing ran as a batch process that took up to 168 hours to complete, with inconsistent reconciliation between systems and manual intervention needed whenever the numbers didn't line up.
What I built
A Delta Lake architecture on Databricks Runtime 10.4, with custom PySpark transformations running on optimised 8-node clusters (dynamic allocation, spot instances) that process 15M+ readings in about 2 hours instead of 168, a Zero-ETL pattern that removed a whole class of data inconsistency between upstream and downstream systems.
Around the pipeline, 42 Airflow DAGs (178 tasks) handle orchestration and retries with comprehensive fault tolerance, and a multi-stage data-quality framework (85+ rules, schema enforcement, statistical anomaly detection, automated reconciliation) keeps accuracy at 99.9%.
Impact
Processing latency dropped 97% (168h → 2h), manual interventions fell 98%, and the combination generated roughly €13M in annual operational savings.