The newbie cloud architect diary

I had the opportunity to present GitHub Actions (see my previous post) during our team call, with report on our evaluation activities and findings. This actually rose good questions and feedback.

At the same time I've been concerned, potentially complaining, about not having a clear role and acceptance when entering projects. On one hand, I still have a lot to learn before becoming fully operational. On the other hand, I had not real engagement since I started, and cannot wait for it.

There is one program which I can definitely contribute too, based on my available skills and experience: Building our internal Data lake. For some reason, this topic is pretty must resting in the background of other initiatives although it has actually been initiated 2 years ago. So that I fell empowered to make it great again ;-)

A Data lake is a single distributed data store used for Ingesting, Refining and Analyzing any kind of enterprise data at scale. Unlike a Data Warehouse, it does not natively support Data definition language (DDL). But it supports Schema-on-read and unstructured data. A Data Lake is a critical asset that requires Security and Governance.

Because of the high volume of information ingested into a Data lake, it is also a common source for traditional Machine Learning (ex. detection of financial fraud, energy network anomaly or customer churn, dynamic pricing or forecast) and modern Deep Learning (ex. voice and face recognition, analysis of medical tumor or manufacturing quality, augmented reality, GANs) use-cases.

There is obviously a common business need and strategy behind building a Data lake, which is overall to bring the organization through digital transformation, get more market or customer insight (i.e. 360-degree feedback) and eventually take smarter decisions both at executive and operational level.

According to Confluent experts, we already reached the second main Generation of Data Lakes:

Credits: Kafka as your Data Lake, is it feasible? by Guido Schmutz

Like food, information gets ingested and digested, both process steps being decoupled from each other. A typical pattern for Data ingestion is the Hub-and-spoke architecture (as opposed to point-to-point), where a central server is used to run E(T)L jobs that synchronously connect to source systems in order to request and poll data which is then optionally transformed and loaded somewhere else.

Unfortunately, batch extraction is a resource-intensive operation that might slow down, freeze or even crash a source system. In order to prevent any negative impact for the business on productive applications, integration jobs are usually scheduled to run when the business is over (ex. at nights or weekends) or on vendor supported mirror (ex. distributed database replica, disaster and recovery system or snapshot). 

Even more importantly, it is definitely worth checking whether your source system can handle events which could be pushed "as they come" (i.e. as a Stream) through "outbound connectivity" (ex.  Webhook) to a broker system, and then collected asynchronously by the ingestion process. This method is commonly used in the financial, retail and industrial sector for real-time analysis.

Like any other architecture component, a data loader brings some kind of interface (input/output), logic (internal orchestration) and integration (external communication). Therefore, in order to enforce control on your ingestion process, a couple of basic validations should be considered:

Now you want a Data lake, not a data swamp! A typical mistake is collecting as much data as possible and transform it as soon as possible. Not only this approach does not scale, but it prevents any future change right from the begining. A good initial Data lake design and user guideline are absolutely required in order to get some value out. We’ll put 5 principles together.

We plan to allow some access the Trusted (ex. Data Analyst via Snowflake) and the Refined (ex. Business users via Tableau) zones. Later on, we may also enable some application integration (ex. via service endpoint).

#2 Design the structure of your raw zone (so that anybody can understand what's in it, and select a dataset)

On one hand, be aware that this zone has a high impact on your sizing, since it typically contains a high redundancy, high retention time and low compression. On the other hand, the structure will help for data partitioning and selection.

Unlike for a Data Warehouse, the process of refinement is much more than just data normalization. It should reconciliate separately stored entities, enrich records stepwise with all information and prescriptions relevant for the end-consumer.

Remember you are building a growing system, able to process a huge amount of data from many sources, and to potentially serve use-cases for your entire business. Forget about contracting an external consultant who is likely to disappear after coding a big monolith script running somewhere, somehow... This might be ok for a "quick win" but not for the long-term. Instead, you should try to standardize as much as possible your data promotion pipeline from source to refinement. Here is a list of high-level requirements: 

Build your own integration platform or select a vendor product if you can afford it, it typically ships with a solid toolkit for designing and executing Data integration flows based on graphical pipes-and-filters supporting ETL patterns, a job scheduler for planning and orchestrating pipelines, an operational console for user administration and monitoring. We use Talend for development and Kubernetes for operations.  Plus, you may have the option to extend to Data Quality (ex. a portal to define standardization rules, match your records, remove duplicates and reconciliate errors) or Advanced Analytics (ex. self-service BI). In case your data source/target is in the Cloud, then it definitely makes sense to run your integration and analysis processes in the Cloud as well.