Captain’s Log Star Date 2024.03.15
Study Progress This Week
- Data Pipelines Pocket Reference – Moving and Processing Data for Analytics
- Chapter 4 – Data Ingestion: Extracting Data
Findings/Notes
I’ve kind of slowed down my technology research recently, I’ve been really busy with my other demands right now. The most important thing I studied this past week was understanding Kafka and Snowflake and how those technologies are used in modern data architectures. I know that Kafka is used for building real-time data pipelines and Snowflake is a cloud-based data warehousing platform. I am aware it’s a common coupling to have Kafka with Snowflake to ingest data and provide a means to analyze a data warehouse. I will most likely look at Kafka as it is open source, but given that Snowflake is commercial I’m not sure what I can get done with it for free. I would still like to learn how to use Apache Spark also. I am just seeing so much of it mentioned in the online communities I follow that I think they need some attention.
Other than that, I am transitioning out of a time intensive volunteer role so I can spend some more of my free time exploring technical topics and I am looking forward to that for sure. I am behind on my training plan and would like to make up for it.
![DALL·E 2024-03-24 21.23.52 - Imagine a data pipeline with a strong cyberpunk aesthetic. This pipeline is set in a neon-lit urban landscape, where the blend of old and new technology](http://www.joshuarust.com/wp-content/uploads/2024/03/DALL·E-2024-03-24-21.23.52-Imagine-a-data-pipeline-with-a-strong-cyberpunk-aesthetic.-This-pipeline-is-set-in-a-neon-lit-urban-landscape-where-the-blend-of-old-and-new-technolo.webp)