MongoDB is a popular document store for backing Web applications, but a challenge for the Data Engineer to ingest its content into a Data Warehouse. The weekly mood We worked in team around the functional specification of our API Gateway as well as our Data Lake project, which I enjoyed very much. As part of the Data Lake, we currently need to source and analyse information from a specific Web application that uses MongoDB as operational database. Aggregations should be ran inside our corporate Cloud Data Warehouse (DW) based on Snowflake , BI-reports in Tableau . While this post sets the focus on MongoDB, I do plan a series of articles around Data Engineering and Architecture, including our finding based end-to-end transformation pipeline from source to destination. What is MongoDB MongoDB is an open-source NoSQL database that stores its data in JSON-documents. The project started in 2007 at a company called "10gen" which was renamed Mongo...
Learnings & thoughts collected on-the-job