Machine Learning (ML) Deployment is one of the dark sides of both Data Science and Data Engineering. Mmanaged services like Databricks might help. The daily mood As already mentioned in a previous post , I have the privilege to shadow a starting Data lake project . The team already prepared data and built a first Machine Learning (ML) model for a specific use-case. They are currently in the process of deploying the scoring application and find this quite challenging. We are going to discuss why it is difficult, and of course how technology and automation may help. Big Data & ML adoption In 2009, the Knowledge Discovery in Data Minining (KDD) conference became a competition (KDD Cup) which reached the IT world with a disrupting report of lessons learnt in large scale ML projects. The industry just started to realize the rize of Big Data (ex. IoT) and the potential of ML for Business. In the following years, Business Intelligence (BI) organizati...
Learnings & thoughts collected on-the-job