The newbie cloud architect diary

Because a Machine Learning model is a living asset, it is best-practice to automate its development  lifecycle to ensure repeatability. The daily mood Today was my weekly office day and I had the opportunity to meet a colleague from the SRE team, that I had actually not seen for a long time anymore. He told me about the SRE organisation, pains and achievements, which I found very insightfull. We also joined 3 other colleagues for quite a long lunch break at the terrasse of a restaurant. None of us had much social contact during the last months of lock-down, and it looks like we all enjoyed meeting again. Beside that, I am now looking at some ML tooling for enforcing best-practices and operationalization, that we are using in a project (see my previous post on ML model deployment) and which I was not yet much familiar with. First candidate MLflow is actually pretty straight forward, at least in theory. What is MLflow MLflow is a ML lifecycle management platform written in Python. ...

Machine Learning (ML) Deployment is one of the dark sides of both Data Science and Data Engineering. Mmanaged services like Databricks might help. The daily mood As already mentioned in a previous post , I have the privilege to shadow a starting Data lake project . The team already prepared data and built a first Machine Learning (ML) model for a specific use-case.  They are currently in the process of deploying the  scoring application and find this quite challenging. We are going to discuss why it is difficult, and of course how technology and automation may help. Big Data & ML adoption In 2009, the Knowledge Discovery in Data Minining (KDD) conference became a competition (KDD Cup) which reached the IT world with a  disrupting report  of lessons learnt in large scale ML projects. The industry just started to realize the rize of Big Data (ex. IoT) and the potential of ML for Business. In the following years,  Business Intelligence  (BI) organizati...