The newbie cloud architect diary

Curve fitting is probably one of the key enabler for data analysis. It helps understanding historical datasets, recognize patterns and take action. The weekly mood I took a couple of days off to visit my parents. I had not seen them for the last 7 months, not only because of the 8 hours drive required to reach their place, but also because of the sanity restrictions introduced with COVID-19. Given some increasing expectations from my job, I found that moving my blogging activity from posts that used to come out almost on a daily basis would be more realistic on a weekly basis.   Following to the  Garmin's outage  that happened during the week-end, I shortly came to think again about my Strava segment effort analysis topic. Although I was quite satisfied about the status of my previous post , I had to admit that comparing segment efforts "only" through average power by average grade is probably not enough for creating a performance indicator. Performance range My...

Amazon SageMaker is a Machine-Learning-as a-Service (MLaaS) framework that focuses on ML model development and automation, for example model serving. The daily mood I realize that I am looking much at Cloud Native technology but not using much of Cloud provider solutions, which is probably as much important if you do not want to re-invent the wheel at application level.  As already discussed in my  previous post  about "ML model deployment", we've been using both AWS Databricks and Amazon SageMaker as part of our internal Data lake project. In my last post , I looked at MLflow which is actually the relevant part of Databricks for deployment. Today I am looking at Amazon SageMaker which is indeed integrated by MLFlow for model deployment via the SageMaker SDK, but also offers a slightly different tooling and approach for development and operations. Why AWS for ML AWS currently offers one of the largest set of managed capabilities for Machine Learning . The offer consists ...

Because a Machine Learning model is a living asset, it is best-practice to automate its development  lifecycle to ensure repeatability. The daily mood Today was my weekly office day and I had the opportunity to meet a colleague from the SRE team, that I had actually not seen for a long time anymore. He told me about the SRE organisation, pains and achievements, which I found very insightfull. We also joined 3 other colleagues for quite a long lunch break at the terrasse of a restaurant. None of us had much social contact during the last months of lock-down, and it looks like we all enjoyed meeting again. Beside that, I am now looking at some ML tooling for enforcing best-practices and operationalization, that we are using in a project (see my previous post on ML model deployment) and which I was not yet much familiar with. First candidate MLflow is actually pretty straight forward, at least in theory. What is MLflow MLflow is a ML lifecycle management platform written in Python. ...

Machine Learning (ML) Deployment is one of the dark sides of both Data Science and Data Engineering. Mmanaged services like Databricks might help. The daily mood As already mentioned in a previous post , I have the privilege to shadow a starting Data lake project . The team already prepared data and built a first Machine Learning (ML) model for a specific use-case.  They are currently in the process of deploying the  scoring application and find this quite challenging. We are going to discuss why it is difficult, and of course how technology and automation may help. Big Data & ML adoption In 2009, the Knowledge Discovery in Data Minining (KDD) conference became a competition (KDD Cup) which reached the IT world with a  disrupting report  of lessons learnt in large scale ML projects. The industry just started to realize the rize of Big Data (ex. IoT) and the potential of ML for Business. In the following years,  Business Intelligence  (BI) organizati...