Aegon - Solution Architect Data Lake

Aegon - Solution Architect Data Lake

Project Overview

Following the data cache solution we started designing and building a Data Lake platform as a permanent solution for better data provision for the online portals. With this platform we also wanted to better support new advanced analytics initiatives such as Machine Learning and real time analytics.

Instead of coupling the legacy administration systems everywhere via integration solutions (which those systems were not designed for) we take the data out of those systems and bring it together in a Data Lake. From here it can be made available much more efficiently for all kinds of data use cases, whether it concerns traditional Data Warehousing and BI questions, or real time processing and self‑service portals.

At the moment a change was made in the administration system it was picked up directly and made (near) real time available via the data lake for the online environment. BI processes no longer had to wait for long and cumbersome bulk export processes but could fetch the latest state directly from the Data Lake.

For this we used technology such as change data capture and streaming solutions from AWS. The entire platform was developed on cloud technology. AWS and Apigee (GCP).

API Management Platform - Requirements & Selection

During this process I performed a requirements analysis and product selection for a new API Management Platform for Aegon. The new API platform had to fit better with the intended data lake and would have to replace the legacy IBM ESB implementation of that time.

Together with the respective vendors I executed proofs of concept with IBM API Management, Apigee and MuleSoft ESB. Ultimately Apigee was chosen because it best met the requirements. Apigee was rolled out during this project and the first API products were developed on it to expose data from the Data Lake to the online self‑service portals.

Technology Stack

Python, Apigee, Attunity Replicate, Aurora, AWS EC2, Lambda, S3, DynamoDB, Kinesis, SNS, SQS, Bamboo