The newbie cloud architect diary

This week slightly challenged my sense of responsibility and communication. It started with some (expected) sign of non-satisfaction from my manager, and ended with some (unexpected) sign of Leadership from me. In fact there are enough areas where I usually put enough of pressure on myself, like for example "delivering what I promised". However, I still definitely need a manager to motivate me for other areas like for example "making things happen". As an outcome, I am currently enjoying a much better team work around our Data Lake project and API Gateway evaluation. It's a shame that this would probably not happen without an external trigger, but that's also the essence of Lifelong learning.

This experience reflects the common mistake that people make when focusing on results rather than on outcomes. Let's assume that consciously, everybody wants a better world. However unconsciously, everybody strives for looking good and productive (results), not necessarily for making the world easy and sustainable (outcome). We may work on a better world based on a current status that needs to be "under control". Changes as a result often happen by "force" or by "accident", and so the world might become even more hectic and complicated. Consciously again, we will repair broken things and live with originated complexity. For this reason, it is generally worth following a Lean approach.

Anyway I expect more of such challenges coming, and so I'll definitely try to put more attention on the outcome in the future, than on the result. A fairly good transition to one of the key responsibilities of the Architect: Support Business Decision making not only with the right culture (ex. Consensus) but also with the appropriate Infrastructure, Processes and Tools, including their ability to collect, analyse and reflect the current status of Business operations. "If you can't measure it, you can't improve it" [Peter Drucker]. Such analytical information are for example Key Performance Indicators (KPI) that help better understand the market demand as well as the effective capacity to operate, drive changes and innovation. See also Gartner's 5 Steps to become a Digital Leader. 

Bill Inmon is recognized as the father of the Data Warehouse (DW) thanks to his book Building The Data Warehouse which was first released in 1992, and revealed the following accepted definition of what it is: A subject oriented, non-volatile, integrated, time-variant collection of data in support of management's decisions. The book also invites to design all enterprise assets using an Entity relationship (ER) model, a top-down approach also known as the Enterprise Data Warehouse.

In 1996, Ralph Kimball made the alternative proposal of a bottom-up approach also known as Dimensional Modeling in his popular book the Data Warehouse toolkit. Not the that I've already been writing about that kind of modeling in this previous article. Kimball's approach finally reached the critical majority with the rise of self-service Business Intelligence (BI) tools since 5 years.

DW concepts didn't change much since then, apart from Data Vault (DV) Modeling proposal in 2000, as well as the reduced cost and increased elasticity of Online analytical processing (OLAP) engines since 2010s.

Under the hood, the compute architecture initially called Elastic Parallel Processing (EPP) is a hybrid of SMP and MPP, in that the Platform is able to manage multiple MPP clusters of different sizes. After the term EPP didn't really establish outside Snowflake ecosystem, the custom architecture is now referred to as a Multi-cluster DW.

The storage architecture is equally amazing. In spite of I/O latency introduced by the third party Object Store, Snowflake takes advantage of durability and multipart features for splitting 500 MB sized chunks of data called micro-partitions. Within each partition, each table is then indexed and compressed at column level. All related metadata is hold by the service layer. 

Eventually, this pretty unique combination of compute and storage architectures theoretically allows for an unlimited number of virtual clusters and virtual nodes to share the same virtual disk while running data intensive operations, without any drop of performance from the physical infrastructure layer.

Snowflake supports different levels of Storage:

Materialized views (MV) are pre-computed SELECT queries which data is stored in the Result Cache, allowing shorted latency and higher consistency. The result never changes unless the view is replaced.

External tables were recently released as a preview feature, so that the platform opens to non-internally managed object stores. Users can better address their storage compliance requirements while running Data-Lake-style queries. Generally supported file formats (CSV, JSON, Avro, ORC, Parquet, XML) and compression (Gzip, Bzip2, Zstandard) are possible. Of course access is read-only and slower, but we'll discuss about the use-case in a bit.

In case you don't have any yet, you need to create an account. There is a 30 days trial plan available for free, including sample data and covering all infrastructure costs.

Then you can access Snowflake via the Web-Interface, the snowsql CLI, the snowpipe REST API, or the driver SDK which is already supported by a number of third-party applications.

From what I read, the boundaries between the DW and the DL are currently melting and there are good chances to see Snowflake entering the Data Science market in the future.