Data Integration

Modern data integration goes beyond traditional ETL and Data Warehouses. I help organisations build scalable, real-time data integration platforms that can further support your data strategy. Whether it concerns classic Data Warehouses, Data Lake(houses), API or Online Data Stores or Machine Learning and other AI use cases.

Ready to modernise your data capabilities?

What I can do

Data Platform Architecture

I have extensive experience setting up Enterprise-Grade Data Integration Platforms. Fully serverless solutions that provide scalable, affordable, and above all, reliable data pipelines. Designed as a self-service engineering platform, they empower data teams to unlock data for their own applications. All of this is governed by Data Governance policies, where data is clearly classified, described, and assigned to data owners. Data Stewards monitor the quality, classification, and documentation of data. Through authorization flows, Data Owners can review data usage requests and view current data usage.

Technology

Data integration technology I work with:

• AWS Serverless Data Integration: S3 (Tables), Iceberg, Glue, Athena, Redshift, Data Pipelines, Step Functions
• Data Lake Design: S3, Azure Data Lake, Google Cloud Storage
• Data Warehouse: Snowflake, BigQuery, Redshift architecture
• Lakehouse Architecture: Delta Lake, Apache Iceberg implementations
• Data Mesh: Decentralized data architecture patterns, Federated queries

Real-Time Data Streaming

Not only classic Data Warehouse ETL batches, but also real-time data processing and analytics

• Event Streaming: Apache Kafka, AWS Kinesis, Google Pub/Sub
• Stream Processing: Apache Flink, Apache Storm, AWS Lambda
• Change Data Capture: Real-time database synchronization
• Event-Driven Architecture: Microservices data integration patterns

Data Quality & Governance

Data quality and governance policy implementation

• Data Validation: Automated data quality checks and monitoring
• Schema Evolution: Backward/forward compatibility management
• Data Lineage: End-to-end data flow tracking
• Data Classification: CIA Rating, GDPR, HIPAA, Data Masking, Data Classification automation

Data Integration Patterns

I have 22 years of experience with system and data integration. Do you have a use case where you just cannot manage to bring data together for your PowerBI reports, I can certainly help you further!

• Batch Processing: Scheduled ETL, Incremental Loading, Bulk Data Transfers
• Stream Processing: Event Sourcing, Change Data Capture, Data Stream processing, Real-Time Analytics
• Data APIs: MQTT, IoT, OData, GraphQL
• Data Formats: Parquet, JSON, XML, CSV, Avro, ORC