Skip to main content

Posts

Showing posts with the label distributed processing

044: Data Warehousing with Snowflake

Snowflake is a Cloud Data Warehouse that is comprehensive to use and probably belongs to the most successful cloud platforms over the last few years. The weekly mood This week slightly challenged my sense of responsibility and communication . It started with some (expected) sign of non-satisfaction from my manager, and ended with some (unexpected) sign of Leadership from me. In fact there are enough areas where I usually put enough of pressure on myself, like for example "delivering what I promised". However, I still definitely need a manager to motivate me for other areas like for example "making things happen". As an outcome, I am currently enjoying a much better team work around our Data Lake project and API Gateway evaluation. It's a shame that this would probably not happen without an external trigger, but that's also the essence of Lifelong learning . This experience reflects the common mistake that people make when focusing on results rather than on o...

031: GPX file analysis with Spark3

Digital mobility comes with GPS navigation and tracking. We'll use Apache Spark for processing and analysing GPX files generated by a running watch. The daily mood Beside riding, I also like running in my free-time. Well, because of my actual life-style and health, a category of time investment that sits somewhere in between  leisure  and duty . As part of my running activities, I recently started with interval training on a 400m tartan track. I run 8x 1000m fast + 600m slow , for a total of about 1 hour workout plus 15 min warm-up and 15 min cool down. This is rather a traditional training method as compared to modern  High-intensity-interval-training  (HIIT), but I actually needed a convenient plan for improving my racing pace, without being too much concerned of potential pain and injury. I am equiped with a second-hand  Garmin Forerunner  watch, probably one of the most popular devices for capturing passable GPS/HR measures at an affordable price range...

004: Container orchestration with Kubernetes

Kubernetes is a Container Orchestrator able to run in distributed mode. It has become the default platform for operating scalable applications. The daily mood My primary focus is to become familiar with, and improve our current Continuous Delivery & Deployment (CD) process, a software development and operation engineering discipline that requires high level of automation in order to improve the pace of innovation and quality of production. Our organisation already uses  Kubernetes  intensively. In terms of CD, we are slowly transiting from a traditional operational model to the  GitOps  philosophy. In my previous post  I have setup a local Kuberntes cluster and some tooling. Now we are going to talk about Kubernetes concepts and create an example application from scratch.  What is Kubernetes Kubernetes  (K8s) is a distributed system written in Go and used for container orchestration, i.e. for configuration, deployment and operation of contain...