The newbie cloud architect diary

Flux Helm Operator supports the automated deployment of Helm charts to Kubernetes while maintaining compliance with the GitOps operational model. The daily mood I watched a bit the  DockerCon 2020 . Although most talks are addressed to a C-Level or beginner audience, I found some interesting pointers around application packaging, integration and delivery. Among others, I will definitely have a look at KubeStack  which seems to be able to solve some problems around GitOps.  My today's goal is to have a look at Flux for automating Helm chart deployments. Source: Revelry.co / Bitnami Requirements Same as in this  previous post about Flux without Helm. Installation steps Delete previous objects kubectl delete namespace flux kubectl delete clusterrole flux Add new Helm repository helm repo add fluxcd https://charts.fluxcd.io Apply HelmRelease CRD to the cluster kubectl apply -f https://raw.githubusercontent.com/fluxcd/helm-operator/master/deploy/crds.yaml Create Flux na...

Skaffold is a client-only tool that watches changes on your local file system for automatically deploying to a local or remote Kubernetes cluster. The daily mood I heard several times that Flux may not offer the best developer experience. On one hand it might not fullfill all our requirements as per our Helm custom plugin. On the other hand despite it is fairly easy to use, it might ask developers and DevOps engineers for too much delegation to the operator running "behind the scenes", without offering a clear feedback loop. Today we are looking at Skaffold, a more developer-friendly tool. What is the particularity of Skaffold Skaffold  follows a declarative approach but is not a pure GitOps implementation since Git is not the single-version-of-the-truth but a local copy of it. The project was started 2y ago, is backed by Google and actually used behind their cloud-based IDE Cloud Code  (IntelliJ/VSC as-a-Service). Setup Download Skaffold command-line and add it to your P...

Flux is a GitOps Operator for Kubernetes which supports both Git sync, automated deployment and rollback, so that deliveries can go faster and safer. The daily mood I had the opportunity to attend to a meeting with developers about the plan to migrate one of our applications. They might need the support of our team. I really enjoyed the topics discussed even though I felt a bit overloaded with developer jargon, specific acronyms and historical details.  Back to my ramp-up, we already introduced what is Flux in a previous post, a typical GitOps implementation for K8s. In this test use-case, the developer basically commits something to his branch and the cluster just takes care of the rest i.e. updates the application on the cluster. Basic requirements K8s client and cluster (ex. our local minikube or microk8s) Git client and repo (ex. empty project in GitHub) K8s application (ex. our nginx application from previous post ) Installation steps Setup Flux command-line tool ( ...

Software Development Lifecycle (SDLC) management is an old topic that is hot again for DevOps teams in charge of operating applications in Kubernetes. The daily mood According to the famous book Accelerate  by Nicole Forsgren, Jez Humble and Gene Kim, we are often looking at automation as an opportunity to improve both development velocity and operational reliability. We know what works and what doesn't work, yet we often don't know well enough the methods and discipline to make it lean and consistent. I need to build my own mindset about our path towards GitOps. I just looked at some  Dzone contributions from vamp.io and split.io around the concept of Continuous Release Cycle. * Deployment approaches Conitnuous Deployment : New versions are automatically rolled-out as they are available. Ideally with no risk but this is extremely hard to achieve, therefore it doesn't get any credit. Continuous Delivery : New versions are submitted to manual deployment approval as...

This article discusses different proxy types and introduces Envoy, a high performance distributed proxy designed for Microservices architectures. The daily mood In the past I intensively worked on our Platform-as-a-Service (PaaS) as a user. Although I was interested in it, I didn't have access to architecture details, developer repositories, operational processes and environments. Now that I have all this, it feels a bit stange that I am not yet able to bring them to life, in a way that I could immediately run and use our solutions as needed... Not that easy! Indeed, an application that basically looks like a monolith web server and database on the frontend side, actually appears to be a big and resource-intensive collection of  Microservices . Those need to be configured, secured and provisioned individually so that the application can be operated in Kubernetes.  This finding "positively" explains how we can scale accross a large number of tenants. It also "negati...

This article discusses different approaches and solutions around GitOps application delivery for Kubernetes and our path towards the adoption of Flux. The daily mood I requested access to our LucidCharts. Wit this I expect to be able to review any existing chart of our delivery process, and finally be able to create new ones. In the mean time I am looking for available industry solutions and practices around Continous Delivery in Kubernetes. Hereby I found the Manning book GitOps and Kubernetes of a great source of understanding and inspiration. GitOps implementation Basically GitOp supports both  Pull - and  Push -based approach, provided that configuration changes happens either in Git via InfraCode, or on the cluster via new image tags, rather than through the action of a human operator. GitOps can be implemented using existing technology or new emerging tools, thus with some slight differences in the approach, structure and infrastructure support. Poc Our team already ran...

This article dresses a naive picture and history of Continuous Delivery (CD), while introducing GitOps as the best operational model for Kubernetes. The daily mood My manager introduced me the different tracks we are working on. Fortunately I am free to organise my time and directions for ramping-up. For now I have enough backlog with Kubernetes, Helm and Continuous Delivery. Still I hope that he can support my integration within the team, its initiatives and accountability. CD: A retrospective Software delivery has always been a tedious and time-consuming process. In the old times developers were sending an application archive per e-mail to the administrator of an application server, along with database creation statements and application parameters. Developers were potentially using distributed version control (ex. SVN, Git), build and dependency management tools (ex. Maven, Gradle). But they had no idea of how the production environment is configured. Admins were potentially using ...

Report on the presentation of a PoC result around the orchestration of data workflows/jobs. 2 different tools were evaluated: Cadende and Airflow. The daily mood It's already been 10 days of part-time ramp-up. I have a first opportunity to attend a peer's presentations on a different topic.  This time is about the evaluation of 2 different solutions for workflow orchestration outside Kubernetes : Airflow and Cadence . This DataOps initiative has the goal to improve our architecture and governance around scheduling of Data flows. What is DataOps DataOps is the application to Data analytics of Continuous Delivery (CD) and DevOps . In short, it is about the automation of all stages of the Data flow including ETL, Data science, BI and Analytics. In a world of Microservices, the service orchestrator becomes a major enabler of DataOps. What is Airflow Airflow  started at  Airbnb  in 2014 as a solution to manage increasing workflow complexity. It is written in Pytho...

A Docker registry is an open-source server that is able to store Docker images organised in repositories, and distribute them via an HTTP REST API. The daily mood We had our second team call (we mostly talked about internal/confidential topics). I doubt a bit about my directions, and I feel the need to prevent it to turn into frustration or failure. I wonder if I should involve more support from my team, or apply more structure inside my posts, in oder to ensure productive Goals, Steps, and Results on a daily basis. Though my previous achievements were not too bad after all, so why change and make it worse? Let's rather enjoy what has been done so far and keep going. Following to my  previous blog  on Deployment stacks, I realised that I need a local Docker repository in order to be able to accelerate my deployment activities or switch-off VPN tunnel to our internal network. Docker registry We need a registry hosting a repository. I added an insecure registry entry ...

Traefik is an open-source HTTP reverse proxy and load balancer that is easy to use.  Let Traefik point at your orchestrator and you are ready to go. The daily mood A fellow peer explained me our approach to Software deployment . Instead of getting lost in tons of infromation sources, it is such a high value to get told about the history, as well as to get idiot questions answered. In this case, that opportunity definitely raised my degree of understanding, comfort and motivation. Project history Following to our adoption of Helm for immutable Kubernetes packaging, our SRE team leveraged known tools for configuration management ( Ansible ) and pipeline automation ( Jenkins ). In a few years our platform grew to a hundred of services developped by 12 teams, staging accross 4 environments and different Cloud imfrastructure providers. At some point, we not only needed more DevOps automation, but also process standardization and parametrization (e.g. via templating). Ansible ...

Site Reliability Engineering (SRE) is a particular approach to DevOps for keeping systems highly-available (HA) while continuously releasing features. The daily mood I enjoy my work a lot. However, I just start to realize how important it is to stay consistent and reflective in my ramp-up. I do have directions but nobody feeds me with a concrete planning and tasks, so that every new working day starts with a self-review, questioning and planning. In general, it is just so much important to read, learn and practice a lot now that i have time allocated to this, in order to be able to contribute later when I'll have more responsibility and less time.  One of my focus areas is the science of operationalization. I am making good progress in reading and understanding  Google SRE Book : 11/32 chapters covered so far, mainly introduction and principles. Introduction The  Site Reliability Engineer (SRE) role truely emerged at Google around 2003 while traditionnal sysadm...

Helm is a template-based spec format for configuring, packaging and deploying  applications that consist in multiple resource objects to Kubernetes. The daily mood My last posts were about setting up a Kubernetes environment. We'll now learn the tooling. In general i am not yet much familiar with common developer productivity tools like  Z-Shell ,  Gists  and  Lints , but i assume this will come with the time. Now we are going to look into the packaging of Kubernetes applications. What is Helm Helm is a commonly used package manager for Kubernetes. It is not the only one option for managing your Kubernetes applications but the one which happens to reach critical mass at enterprise grade, just like Docker did for Containerization and Kubernetes for Orchestration. One of the reason for its great popularity is the Helm Hub , the official Helm public repository which is hosting tons of standard packages. At its core, Helm generates Kubernetes manifests out of...