The newbie cloud architect diary

Data Governance is such a wide topic in theory, that most people are confused about which practical approach to choose regarding their Data Lake. The weekly mood I finally have the warm feeling that I well integrated the department which I joined nearly half a year ago. Why such a long time for such a simple goal? Well, if I had to retrospect on this development, I would split it into the following under-estimated phases: Observation Fail-fast  Exploration Deep-dive Immersion Go Mainstream I was definitely not as easy as I expected given my background and motivation. I was glad to have the chance to touch-point on various topics from DevOps to API and Big Data. It was much less pleasant to partly have to wait for months until I got access to the systems and services I needed. Obviously there were some social restrictions related to COVID-19 and time-zones. To finish with, I not only needed to get used to our workflows, but confident with our goals, which also asked for some normal ...

As trend monitoring and networking increasingly happens online, IT professionals need methodology for attending Meetups, Webinars or Conferences. The weekly mood Two weeks ago I spontaneously registered to the  ApacheCon  for free, the yearly rendezvous of the  Apache Software Foundation  (ASF) with Open-Source enthusiasts. ApacheCon is not a conference like others. Indeed, its content is not addressed to potential Software Buyers, therefore owned and influenced by  Marketers . Instead, it is addressed to enthusiastic Contributors and Users of a Developer community, owned and produced by  Members  of a  Project Management Commitee  (PMC). They are not  selling their  product but sharing their concept and vision .  The ASF probably hosts the largest base of software projects , including  httpd  which is the most used web server according to W3Tech . The ASF is also characterized by open-source terms ( Apache ...

Tableau Desktop is  a powerful BI tool for the Data Warehouse. Tableau Prep is a promising but less mature product for  cleansing  new Datasets. The weekly mood This the end of my first quarter working as a full-time Architect. During this period, I had the opportunity to deep-dive into different topics such as DevOps, API and Data Management. It feels like a year from a content perspective, but actually not more than a month from a collaborative standpoint. Indeed, it has been and it is still a challenge for me to connect with other people within a distributed organisation and a world of pandemic. And as I am currently gaining visibility through a Data Lake project, I remind that as for my previous position (I've been working for years as a Solution Engineer), this Architect role might be sometimes very unclear, difficult to endorse and align with other people's expectation. Architecture is a double-edge. It brings people together around  Software design principles ...

A star schema is the simplest style of data mart schema and is the approach most widely used to develop data warehouses and dimensional data marts. The weekly mood My role as a cloud architect is slowing transiting from supporter to enabler . As a supporter, you might make some background noise, but you are not really part of the game, worse you are not part of the staff. That's perfect to observe and learn. Now as an enabler, you might indirectly influence or  contribute to  activities. Enablers are for example advisers and trainers. They aren't working on the field or taking decisions on the border, but they are a useful part of the staff, and feel rewarded by its achievements. In a previous post , we introduced the need to infer and flatten the source schema of some  MongoDB  collection, for the purpose of data analytics. In this article, we'll take intermediary tables aka. Staging area for granted. We will focus on defining the target schema , or the "ma...

Snowflake is a Cloud Data Warehouse that is comprehensive to use and probably belongs to the most successful cloud platforms over the last few years. The weekly mood This week slightly challenged my sense of responsibility and communication . It started with some (expected) sign of non-satisfaction from my manager, and ended with some (unexpected) sign of Leadership from me. In fact there are enough areas where I usually put enough of pressure on myself, like for example "delivering what I promised". However, I still definitely need a manager to motivate me for other areas like for example "making things happen". As an outcome, I am currently enjoying a much better team work around our Data Lake project and API Gateway evaluation. It's a shame that this would probably not happen without an external trigger, but that's also the essence of Lifelong learning . This experience reflects the common mistake that people make when focusing on results rather than on o...