The newbie cloud architect diary

A star schema is the simplest style of data mart schema and is the approach most widely used to develop data warehouses and dimensional data marts. The weekly mood My role as a cloud architect is slowing transiting from supporter to enabler . As a supporter, you might make some background noise, but you are not really part of the game, worse you are not part of the staff. That's perfect to observe and learn. Now as an enabler, you might indirectly influence or  contribute to  activities. Enablers are for example advisers and trainers. They aren't working on the field or taking decisions on the border, but they are a useful part of the staff, and feel rewarded by its achievements. In a previous post , we introduced the need to infer and flatten the source schema of some  MongoDB  collection, for the purpose of data analytics. In this article, we'll take intermediary tables aka. Staging area for granted. We will focus on defining the target schema , or the "ma...

MongoDB is a popular document store for backing Web applications, but a challenge for the Data Engineer to ingest its content into a Data Warehouse. The weekly mood We worked in team around the functional specification of our API Gateway as well as our Data Lake project, which I enjoyed very much.  As part of the Data Lake, we currently need to source and analyse information from a specific Web application that uses MongoDB as operational database. Aggregations should be ran inside our corporate Cloud  Data Warehouse  (DW) based on  Snowflake , BI-reports in Tableau .  While this post sets the focus on MongoDB, I do plan a series of articles around Data Engineering and Architecture, including our finding based end-to-end transformation pipeline from source to destination. What is MongoDB MongoDB  is an open-source NoSQL database that stores its data in JSON-documents. The project started in 2007 at a company called "10gen" which was renamed Mongo...

Curve fitting is probably one of the key enabler for data analysis. It helps understanding historical datasets, recognize patterns and take action. The weekly mood I took a couple of days off to visit my parents. I had not seen them for the last 7 months, not only because of the 8 hours drive required to reach their place, but also because of the sanity restrictions introduced with COVID-19. Given some increasing expectations from my job, I found that moving my blogging activity from posts that used to come out almost on a daily basis would be more realistic on a weekly basis.   Following to the  Garmin's outage  that happened during the week-end, I shortly came to think again about my Strava segment effort analysis topic. Although I was quite satisfied about the status of my previous post , I had to admit that comparing segment efforts "only" through average power by average grade is probably not enough for creating a performance indicator. Performance range My...

Python project working around Strava API rate limitation in order to extract and analyse full segment effort power data from multiple ride activities. The daily mood It's been such a hard week of work and I needed kind of a distractive weekend. First I helped my son build the Lego ISS  (consisting of 864 pieces). Then we had neighbourhood discussions about the recent announcement of a huge construction project which will start next year, take place just in front of our apartment doors, get incredibly noisy and finally reduce our green area into a urban "wasteland".  Beside that I was able to join an outdoor training of my runner club for the first time since three month of pandemic. A good transition to the little Python programming initiative around riding power analytics that I started last week. In my  previous post , we were able to collect Strava segment power data from a single ride, and visualise it. Still there is potential for improvement and I am now looking fo...

Python project that extracts and analyzes a data sample from Strava API. The records are effort power values calculated for given segments of a ride. The daily mood Today is a bank holiday. I didn't work on duty, but I did "kind of" work for fun.  As a roadbike enthusiast I am riding once a week in the summer. I usually record my  activities using a Garmin watch, then track and share them with friends on  Strava  which is the most popular social-media platform (mainly) dedicated to endurance sports. Strava offers a free account and a commercial subscription with additional features. I used to pay for premium for a couple of months but I personally didn't find it worth. The fascination of cycling Cycling is addictive. First, it catches you with outdoor experience, in which you are escaping duty routine, tanking natural oxygen, discovering new places. Second, it treats your body well through carefull and efficient  biomechanics . It is (almost) fre...