The newbie cloud architect diary

I finally have the warm feeling that I well integrated the department which I joined nearly half a year ago. Why such a long time for such a simple goal? Well, if I had to retrospect on this development, I would split it into the following under-estimated phases:

I was definitely not as easy as I expected given my background and motivation. I was glad to have the chance to touch-point on various topics from DevOps to API and Big Data. It was much less pleasant to partly have to wait for months until I got access to the systems and services I needed. Obviously there were some social restrictions related to COVID-19 and time-zones. To finish with, I not only needed to get used to our workflows, but confident with our goals, which also asked for some normal adaptation time.

Let's now follow-up with a series of articles around the Data Lake. In part 1, I explained what it is and how to architect it. Now let's have a look at how to govern it.

 

Electronical Information Governance aka. Data Governance is the general practice of linking records of a storage system (ex. a Data Lake) with transparent processes and responsibilities. The term Governance is profoundly anchored in Information Technology (IT). It is derived from the Greek verb "kubernaein" which means "to steer", and reveals our natural effort to subordinate all kinds of things and assets that might represent a risk in case they are not observed and controlled. In a corporate context for example, IT Governance defines and maintains policies that allow for People, Process and Technology to drive better Business outcomes. 

First Data Governance concerns emerged in the early 1990s from the new practice of building Data Warehouses (DW), which one involved the replication of significant amounts of Data from Host systems aka. IBM Mainframe. Data Governance then moved into the background of Web Service Governance in the early 2000s, itself not only decoupling Business Applications from underlying Systems, but also standardising interfaces and policies as required by some newly establishing industrial conformity around Governance, risk management and compliance (GRC). 

With the rise of Big data and Cloud computing technologies in the early 2010s, three major risks were eventually assessed:

Since only the first point is mandatory, hard to implement by organisations and difficult to control by federal instances, the budget and commitment assigned to Data Governance initiatives is usually kept relatively low. Indeed, like for other strategical and support processes of a core business, there is almost no limit in possible spending, no direct return and no clear estimation of the cost of not taking action. 

First and foremost, some authority is required to bring Data Governance to the agenda of top-priorities in an organisation, typically a Chief Data Officer (CDO). 

Then, creating a high-level documentation of (if possible, all) information processes is a mandatory, yet unpleasant and ephemeral step to more data understanding, guidance and automation.

Finally, a culture that is transparent, supportive and collaborative is recommended by the book Non-invasive Data Governance by Robert S. Seiner (2014).

A native part of Data Governance is Metadata management. In a nutshell, Metadata is Any Data about Data, and is often referred to as the "navigation system" of a Data lake. 

According to Ralph Kimball there are 3 main kinds of metadata:

Except from the CDO, there are typically 4 personas interested by Metadata:

In this section, we will introduce a few Open-Source projects that are supporting Metadata management.

To finish with, there are a number of recent papers showing the importance, complexity and actually of Metadata handling: