Captain’s Log Star Date 2024.05.10

Reading

“Data Pipelines – Pocket Reference, Moving and Process Data for Analytics” – James Densmore – Chapter 6 Transforming Data

“The Enterprise Big Data Lake, Delivering the Promise of Big Data and Data Science” – Alex Gorelik – Chapter 4 Starting a Data Lake

“The Making of a Manager: What to Do When Everyone Looks to You” – Julie Zhou – Chapter 4 The Art Of Feedback

Courses

A Cloud Guru – “DevOps Essentials” Chapter 2 DevOps Culture

Udemy – “Complete Agile Scrum Master Certification Training”

Notes

I learned about data models a little more. I always built mine in Excel, but I’m learning this can be stored anywhere, and is often stored in a Data Warehouse. Good data models should take into account the right measures and attributes that offer business value and hold the right granularity. To pipe it in though, there are different processes and logic available for pipe in data for full rebuilds, appending tables, or capturing change data. I also learned about the Urchin Tracking Module (UTM) parameters in URLs and how that’s used to track marketing information on unique click throughs.

As far as starting a data lake from scratch, there is a lot to learn here. The book I am reading suggest Hadoop, or some Hadoop like product. It’s cheaper to utilize than a standard RDBMS. There’s also a lot to take into consideration involving data governance as well and making sure it is available to users. There is a mention of Spark and Kafka again and that may be a topic to review down the road.

I am also learning more about DevOps and the technologies that are being used. I hear so much about CI/CD and the technologies surrounding it. What is interesting is how DevOps addresses so many issues between Development and Operations and tries to resolve their conflicting goals. I get the attempt to end the “Us Versus Them” and “throwing it over the wall” mentality. DevOps makes the team happier and ultimately customers happier.

I’m also liking Julie Zhou’s book about management. There’s a lot to learn here about managing people and yourself.