Reading Notes presented by BookClub.dev: Recent Episodes
Dan Cook
There is no end to learning that you need to do when you work in the software industry, and it can feel overwhelming. But we will take on that challenge one book at a time. Every Monday host Dan Cook shares what he learned from that week's reading.
This is the final episode of Reading Notes.
THANK YOU for joining me on this journey.
This Thursday at 7 pm eastern we wrap up Code That Fits in Your Head by Mark Seemann and look at what we've taken away from this book.
You can sign up for free at BookClub.dev.
As always happy reading
This week we look at chapter 16 the last chapter in Code That Fits in Your Head by Mark Seemann.
Join us for the discussion of this chapter Thursday at 7 pm eastern. You can sign up for free at BookClub.dev
This week we look at chapter 15 The Usual Suspects in Code That Fits in Your Head by Mark Seemann.
Resources
- Guide Refactorings With Behavioral Code Analysis (YouTube)
- The Pragmatic Programmer (Amazon)
- Code Complete (Amazon)
- Clean Code (Amazon)
- Software Design X-Rays (Amazon)
- Your Code as a Crime Scene (Amazon)
Join the companion discussion at 7 pm eastern this Thursday. You can sign up for free at BookClub.dev.
Happy Reading
This week we look at chapter 14 Rhythm in Code That Fits in Your Head by Mark Seemann.
Because we are building software for the long term we get the advantage of seeing different rhythms and patterns develop in how we work as individuals, as teams, and even in the code.
I hope you'll join us as we discuss the ideas from this chapter in our free companion discussion. You can sign up for free at BookClub.dev.
Until next time happy reading.
This week we look at chapter 13, Separation of Concerns in Code That Fits in Your Head: Heuristics for Software Engineering, by Mark Seemann.
This chapter is mainly about the composition and decomposition of code. Mark recommends that we aim towards creating a functional core with an imperative shell and shows us some of the advantages of this approach.
We'll discuss the ideas from this chapter on Thursday at 7 pm Eastern. You can sign up for free to join this conversation at BookClub.dev.
I hope to see you there and until next time happy reading.
This week we look at chapter 12 Troubleshooting of Code That Fits in Your Head by Mark Seemann.
Mark shares a bunch of different techniques for troubleshooting. The most intriguing one is using StackOverflow as a rubber duckie.
Resources
- The Pragmatic Programmer (Amazon)
- Rubber Duckie (Amazon)
- Code That Fits in Your Head (Amazon)
Join the free companion discussion Thursday at 7 pm eastern BookClub.dev
This week we are looking at chapter 11 of Editing Unit Tests of Code That Fits in Your Head: Heuristics for Software Engineering by Mark Seemann.
This week Mark mentions three books in the chapter
- Working Effectively with Legacy Code (Amazon)
- Refactoring (Amazon)
- xUnit Test Patterns (Amazon)
We'll be talking about these books as well as other ideas from this chapter on Thursday in the free companion discussion at 7 pm Eastern. You can sign up for free at BookClub.dev.
Happy reading!
This week we look at chapter 10 Augmenting Code in Code That Fits in Your Head by Mark Seemann.
Resources:
- Strangler Fig Pattern
- Semantic Versioning
Join the free companion discussion this Thursday at 7 pm Eastern. You can sign up at BookClub.dev.
This week we look at chapter 9 Teamwork in Code That Fits in Your Head by Mark Seemann.
We'll be discussing the ideas in this chapter this Thursday at 7 pm Eastern. You can sign up for free at BookClub.dev
This week we look at API Design or chapter 8 of Code That Fits in Your Head by Mark Seemann.
We talk about two concepts this week
- poka-yoke or the idea of mistake-proofing
- Writing for our readers
Mark includes a Hierarchy of Communication
- Guide the reader by giving APIs distinct types.
- Guide the reader by giving methods helpful names.
- Guide the reader by writing good comments.
- Guide the reader by providing illustrative examples as automated tests.
- Guide the reader by writing helpful commit messages in Git.
- Guide the reader by writing good documentation.
We'll be discussing these ideas and more this Thursday at 7 pm eastern in the free companion discussion.
You can sign up for free at BookClub.dev.
I hope to see you there and until next time happy reading.
This week we look at Decomposition or chapter 7 of Code That Fits in Your Head by Mark Seemann.
Resources
- -2000 Lines of Code
- Cyclomatic Complexity
- Accelerate (Amazon)
We'll be discussing the ideas of this chapter this Thursday at 7 pm eastern. You can sign up for the free discussion at BookClub.dev.
I hope to see you there and until next time, happy reading.
This week we look at chapter 6 of Code That Fits in Your Head by Mark Seemann.
This chapter builds on many ideas we've seen in the previous few chapters and shows them in action. The most important of these is Uncle Bob's Transformation Priority Premise.
You can join the free weekly companion discussion this Thursday at 7 pm Easter by signing up at bookclub.dev.
I hope to see you there and until next time happy reading.
This week we look at chapter 5 Encapsulation of Code That Fits in Your Head by Mark Seemann.
This chapter is packed full of great stuff and you get to see it in action.
Resources
- The Pragmatic Programmer (Amazon)
- The Transformation Priority Premise
- The Robustness Principle
- RFC 761
You can join the free weekly companion discussion this Thursday at 7 pm Easter by signing up at bookclub.dev.
I hope to see you there and until next time happy reading.
This week we continue our journey through Code That Fits in Your Head by Mark Seemann.
We are reading chapter 4 Verticle Slices which finally gets into the code!
This chapter also is heavily inspired by Growing Object-Oriented Systems Guided by Tests (Amazon), and I highly recommend the book.
If you want to join the companion discussion for this week you can sign up at bookclub.dev. The discussion will start at 7 pm eastern Thursday, January 27th.
This week we look at chapter 3 of Code That Fits in Your Head, Tackling Complexity.
You can join the free weekly companion discussion this Thursday at 7 pm Easter by signing up at bookclub.dev.
I hope to see you there, and until next time happy reading.
This week we look at chapter 2 Checklists in Code That Fits in Your Head.
This chapter is heavily influenced by The Checklist Manifesto by Atul Gawande. (Amazon)
Mark also gives us a checklist for how to start a new project:
- Use Git
- Automate the build
- Turn on all error messages
We'll discuss this chapter Thursday, January 13th, 2022 at 7 pm eastern. You can sign up for this free discussion at bookclub.dev
We start our journey through Code That Fits in Your Head with chapter 1, Art or Science.
This chapter looks for the best analogy for what it is we do when we create software.
Join us this Thursday at 7 pm Eastern to discuss this chapter. You can sign up for free at bookclub.dev.
Code That Fits in Your Head: Heuristics for Software Engineering (Amazon)
Join the free companion discussions starting Thursday, January 6th at 7 PM Eastern (bookclub.dev)
.NET Rocks! Episode 1759: Code that Fits in Your Head with Mark Seemann (.NET Rocks)
.NET Rocks! Episode 1745: CUPID with Dan North (.NET Rocks)
Heuristics definition (Wikipedia)
Zero-Friction TDD (ploeh.dk)
We come to the end of Building Evolutionary Architectures: Support Constant Change, it's time to look back and decide if the book is was worth the read.
One of the authors drops a hint that there might be a second edition to clarify some parts of the book in this podcast.
If you found this book interesting you might also enjoy
- Clean Code
- Working Effectively with Legacy Code
- Extreme Programming Explained
This week, we look at the last chapter in Building Evolutionary Architectures: Support Constant Change, chapter 8 Putting Evolutionary Architecture into Practice.
This chapter is largely about creating the right incentives for teams to build evolutionary architectures. In fact, they probably sound pretty familiar.
- Use cross-functional teams
- Focus on products, not projects
- Align teams and work to deliver business value
Next week I'll be back with my reflections on the book as a whole.
Then in January the final season of Reading Notes will begin as we discuss Code That Fits in Your Head: Heuristics for Software Engineering by Mark Seemann. You can pick up a copy of the book on Amazon
Until next time happy reading!
This week we look at the penultimate chapter of Building Evolutionary Architectures: Support Constant Change, chapter 7, Evolutionary Architecture Pitfalls and Antipatterns.
This chapter looks at 5 antipatterns and 5 pitfalls spread across three categories:
- Technical Architectures
- Incremental Change
- Business Concerns
Technical Architecture
The antipatterns for this category are:
- Vendor King
- Last 10% Trap
- Code Reuse Abuse
The pitfalls are:
- Leaky Abstractions
- Resume-Driven Development
Incremental Change
The antipattern for this category is:
- Inappropriate Governance
The pitfall is:
- Lack of Speed to Release
Business Concerns
The antipattern for this category is:
- Reporting
The pitfalls are:
- Product Customization
- Planning Horizons
This week we continue our look at Building Evolutionary Architectures: Support Constant Change and we discuss chapter 6, Building Evolvable Architectures.
In this chapter, the authors lay out the basic mechanics and some guidelines for building evolvable architectures.
The basic mechanics of building an evolutionary architecture are:
- Identify dimensions affected by evolution
- Define fitness functions for each dimension
- Use deployment pipelines to automate fitness functions
The guidelines for building an evolutionary architecture are:
- Remove Needless Complexity
- Make Decisions Reversible
- Prefer Evolvable over Predictable
- Build Anticorruption Layers
- Use Service Templates
- Build Sacrificial Architecture
- Mitigate External Change
- Updating Libraries versus Frameworks
- Prefer Continuous Delivery to Snapshots
- Version Services Internally
Join our Thursday night discussions to continue the conversation about this chapter. You can sign up at bookclub.dev. The discussion starts at 7 pm eastern and will go for about an hour.
Happy reading
This week we look at chapter 5 Evolutionary Data in Building Evolutionary Architectures.
This chapter has a guest contributor Pramod Sadalage, one of the coauthors of Refactoring Databases: Evolutionary Database Design (Amazon). This is a great reference book and goes a lot deeper into the ideas of this chapter.
Often our databases don't keep pace with our code. There is a cost to this, and here we look at ways to work faster and safer in the database.
Join the free weekly discussion for this chapter by signing up at bookclub.dev. The conversation starts at 7 pm eastern.
Happy reading
This week we are reading chapter 4, Architectural Coupling in Building Evolutionary Architecture: Support Constant Change.
This chapter introduces the concept of Architectural Quanta which the authors define as, "an independently deployable component with high functional cohesion, which includes all the structural elements required for the system to function properly."
This chapter also digs into several architecture patterns and evaluates how well they support the three primary aspects of evolutionary architecture fitness functions, incremental change, and appropriate coupling.
If you'd like to continue discussing these ideas join the free Thursday night discussion at 7 pm eastern by signing up at bookclub.dev
This week we look at chapter 3 of Building Evolutionary Architectures, Engineering Incremental Change.
Resources
- Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Amazon)
- Release It!: Design and Deploy Production-Ready Software (Amazon)
- Lean Enterprise: How High-Performance Organizations Innovate at Scale (Amazon)
- The Netflix Simian Army (blog post) (repo)
- GitHub Scientist (blog post) (repo)
- Fitness Function Katas
Keep the conversation going by joining our free Thursday night discussions at 7 pm Eastern. Sign up by going to bookclub.dev.
This week we look at chapter 2 Fitness Functions in Building Evolutionary Architectures: Support Constant Change (Amazon)
We look at the various aspects of fitness functions including
- atomic vs. holistic
- triggered vs. continual
- static vs. dynamic
- automated vs. manual
- intentional vs. emergent
- temporal
- domain-specific
This Thursday the conversation will continue at 7 pm eastern. You can sign up to join for free by going to bookclub.dev.
Also this week we'll be working on some fitness function katas.
This week we look at chapter 1 Software Architecture of Building Evolutionary Architecture.
Who needs an architect?
How Do Committees Invent? (Original paper on Conway's Law)
Building Evolutionary Architecture (Amazon)
Join the free Thursday night discussion of this chapter by signing up at bookclub.dev. The discussion will start at 7 pm eastern.
Join an exciting 9 week series of discussions about Building Evolutionary Architectures: Support Constant Change (Amazon).
Each week we'll discuss a chapter from the book and how we can apply it to our daily work.
You can sign up for the discussions at bookclub.dev.
I hope to see you there!
All good things must come to an end, and this is our final discussion of Extreme Programming Explained 2nd edition by Kent Beck.
What will you do differently after reading this book?
I hope you'll join our discussion this Thursday at 7 pm eastern as we discuss this. You can join by going to bookclub.dev/live
This is our final week reading from Extreme Programming Explained 2nd edition by Kent Beck.
We look at taking action to live by our values as well as the need to find balance.
Resources
- A Pattern Language (amazon)
- Design Patterns (amazon)
- How Buildings Learn (amazon)
- Office Space (amazon)
- Building Evolutionary Architecture (amazon)
- XP Yahoo Group
Join the discussion live at 7 pm eastern this Thursday bookclub.dev/live
Our journey through Extreme Programming Explained: Embrace Change is nearly at its end.
This week we look at Beck's pragmatism and how that has helped other methodologies to adopt parts of XP.
We also look at what drives a company to outsource development teams.
I hope you'll join us this Thursday at 7 pm eastern as we discuss these topics live. You can watch the stream by going to bookclub.dev/live.
This week we talk about applying XP.
Beck identifies 2 keys that are required for any team to make rapid change, pain and a shared vision.
We'll be talking more about these ideas this Thursday at 7 pm Eastern. You can join the conversation by going to bookclub.dev/live
Resources
- Westrum's model of organizational culture
This week we look at how to Scale XP.
We are reading chapter 15 and 16 of Extreme Programming Explained: Embrace Change 2nd edition by Kent Beck.
I reference a great interview between Gene Kim and Michael Nygard. There are three of them the first is Architecture as the Organizing Logic for Components and the Means for their Construction.
Beck gives his three steps for breaking down a large task into a small task. It's great because it scales to solve any challenge.
- Turn the problem into smaller problems
- Apply simple solutions
- Apply complex solutions if any problem is left
We also see the importance of having executive sponsorship when starting any initiative.
I hope you'll join us as we discuss scaling practices from one to many, this Thursday at 7pm eastern. You can join the discussion by going to bookclub.dev/live.
This week we look at three chapters from Extreme Programming Explained 2nd Edition by Kent Beck.
Chapter 12 Planning: Managing Scope
Chapter 13 Testing: Early, Often, and Automated
Chapter 14 Designing: The value of time
The common thread I see running through these chapters is that these ideas have gone mainstream.
So what is the new extreme version? Or are these the fundamentals that we now need to master?
Join the discussion this Thursday at 7 pm eastern by going to bookclub.dev/live.
I hope to see you there!
This week we continue our journey through Extreme Programming Explained 2nd edition by Kent Beck.
I'm trying something a little different though. This week I cover chapter 11 The Theory of Constraints, chapter 18 Taylorism and Software, and chapter 19 Toyota Production System.
All three of these are ideas that come from the world of manufacturing and have been applied to creating and deploying software.
Taylorism or Scientific Management has fallen out of style, but is still influential in many areas, even if it's no longer called that.
The Toyota Production System (TPS) is the inspiration for Lean and focuses on reducing waste.
The Theory of Constraints (TOC) works off the premise that there is one step in a process that determines the throughput for the entire process.
Beck gives a very relatable example to illustrate this... Laundry.
Washing 45 minutes
Drying 90 minutes
Folding 15 minutes
Try some thought experiments with this system.
How long does it take to process two loads of laundry?
What if washing only took 15 minutes?
What if folding only took 5 minutes?
What if drying only took 60 minutes?
What if you did four loads of laundry?
Resources:
- Team of Teams (Amazon)
- The Goal (Amazon)
- The Phoenix Project (Amazon)
- The Critical Chain (Amazon)
Happy reading
This week we look at Chapter 10, The Whole XP Team from Extreme Programming Explained 2nd Edition by Kent Beck (Amazon).
I discuss:
- Conway's Law and the Inverse Conway Maneuver
- The roles on an XP Team
- The two metrics Beck uses to measure the health of an XP Team
We'll dive into all this and more Thursday, July 15th at 7 pm eastern on Twitch. You can join by going to bookclub.dev/live.
Also if you are interested in more deep dives into concepts like Conway's Law and the Inverse Conway Maneuver drop me a line hello@bookclub.dev.
Conway's Law: Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.
This comes from Mel Conway's 1967 paper "How Do Committees Invent?"
Roles on an XP Team
- Testers
- Interaction Designers
- Architects
- Project Managers
- Product Managers
- Executives
- Technical Writers
- Users
- Programmers
- Human Resources
Two Metrics for XP Team health
- The defect rate
- Length of time from idea to it generating revenue
Resources
- Extreme Programming Explained 2nd Edition (Amazon)
- How Do Committees Invent?
- Inverse Conway Maneuver
- The Unicorn Project (Amazon)
- The Phoenix Project (Amazon)
This week we look at chapter 9 of Extreme Programming Explained 2nd edition by Kent Beck and reflect on the exercise from chapter 8, Getting Started, that we did live last Thursday on Twitch.
After doing the exercise, I recommend the following format. Spend 5 minutes brainstorming on each of the three questions. After that, go back and refine what you came up with; this should take 10-5 minutes for a total of 20 minutes per practice.
The three questions to answer are
Why do you want to do the practice?
What are the things that lead towards doing the practice well?
What are the signs that you are moving in the wrong direction?
In this chapter, Beck described 11 corollary practices for XP. These are more advanced than the original 14 primary practices.
- Real Cusomter Involvment
- Incremental Deployment
- Team Continuity
- Shrinking Teams
- Root-Cause Analysis
- Shared Code
- Code and Tests
- Single Codebase
- Daily Deployment
- Negotiated Scope Contract
- Pay-Per-Use
We will discuss these practices live on Twitch this Thursday at 7 pm eastern You can join the discussion by going to bookclub.dev/live.
This week we look at chapter 8, Getting Started of Extreme Programing Explained 2nd Edition by Kent Beck (amazon)
The beauty of Extreme Programming (XP) is that you can start where ever you are right now. Therefore, it's not necessary to adopt all the practices at once. In fact, it's even recommended not to try and do them all at once.
The question is, where to start? For that, Beck gives us a mapping exercise to help us describe each practice of XP. From these maps, we can find where we are and how to move towards where we want to go.
Thursday, July 1st, I'll be going through this mapping exercise live on Twitch at 7 pm eastern. But I won't be doing it alone! You can join in by going to bookclub.dev/live
This week we look at the primary practices of Extreme Programming (XP) from chapter 6 Practices and chapter 7 primary Practices of our current book, Extreme Programming Explained 2nd edition by Kent Beck.
The 13 primary practices are
- Sit together
- Whole team
- Informative workspace
- Energized Work
- Pair Programming
- Stories
- Weekly Cycle
- Quarterly Cycle
- Slack
- Ten-Minute Builds
- Continuous Integration
- Test-First Programming
- Incremental Design
Join the live discussion this Thursday at 7 pm eastern as we discuss these practices and how we can use them in our daily work. You can join the conversation by going to bookclub.dev/live
This week we look beyond the 14 principles of Extreme Programming (XP) and how they follow the iterative paradigm of learning instead of the entity paradigm of learning.
To continue the conversation about these paradigms and the 14 principles of XP check out the live stream Thursday at 7 pm eastern at bookclub.dev/live.
I discovered those two paradigms of learning from Josh Waitzkin's book The Art of Learning (Amazon)
The 14 Principles of XP are
- Humanity
- Economics
- Mutual Benefit
- Self-Similarity
- Improvement
- Diversity
- Reflection
- Flow
- Opportunity
- Redundancy
- Failure
- Quality
- Baby Steps
- Accepted responsibility
Kent Beck lays out five values as the basis for Extreme Programming (XP)
- Communication
- Simplicity
- Feedback
- Courage
- Respect
These are not the only values that you may need for your needs, but they make up the values for XP.
Join the discussion of these values Thursday night at 7 pm Eastern live
This week we dive into the first 3 chapters of Extreme Programming Explained by Kent Beck (amazon).
While XP is not the software development methodology that everyone is talking about these days, its practices have become pervasive in our industry. Through reading this book I've started to think that we aim for these practices not for their inherent value but because of the values that lead to them.
I reference the Westrum topology for organizational culture. There are a ton of great resources about these different types of cultures and delivering software.
- DevOps Culture
- Westrum’s Organizational Model in Technology Organizations
- Accelerate
I hope you'll check out the live discussion for these chapters on Twitch Thursday, June, 3rd at 7 pm eastern.
Next week we start our discussion of Extreme Programming Explained by Kent Beck (amazon)
While there are other methodologies that are more popular for creating software, they are all heavily influenced by XP.
Kent Beck is also just an inspiring author. Here are three ideas from the preface to the book that have stuck with me.
- No matter the circumstances you can always improve.
- You can always start improving with yourself.
- You can always start improving today.
You can join in the discussion about Extreme Programming Explained on Twitch starting Thursday, June 3rd at 7 pm eastern by going to bookclub.dev/live
This is the end of season one of Reading Notes. Next season we'll be looking at Extreme Programming Explained by Kent Beck.
Six takeaways from Site Reliability Engineering: How Google Runs Production Systems
- The 4 Golden Signals of Monitoring
- Error Budgets and SLOs
- Eliminate Toil
- Explain why
- Create your own culture
- Assume the best of intentions
This week we look at Chapter 33 - Lessons Learned from Other Industries and Chapter 34 - Conclusion
Join the free companion discussion at 7 pm Eastern by signing up at bookclub.dev/thursday.
We look at 4 themes of SRE across several industries
- Preparedness and Disaster Testing
- Postmortem Culture
- Automation and Reduced Operational Overhead
- Structured and Rational Decision Making
This week we look at Chapter 31 - Communication and Collaboration in SRE and Chapter 32 - The Evolving SRE Engagement Model.
Join the free companion discussion by going to bookclub.dev/thursday.
Production Meeting Agenda
- Upcoming production changes
- Metrics
- Outages
- Paging events
- Nonpaging events
- Issues that probably should've paged, but didn't
- An issue that is not pageable but requires attention
- An issue that is not pageable and does not require attention
- Prior action items
This week we look at Chapter 29 - Dealing with Interrupts and Chapter 30 - Embedding an SRE to Recover from Operational Overload.
We are almost at the end of Site Reliability Engineering: How Google Runs Production Systems and that means it's time to pick a new book. Please send your suggestions to hello@bookclub.dev.
Join the companion discussion Thursday night at 7 PM Eastern for free by signing up at bookclub.dev/thursdays.
Resources
- Flow
- Explain your reasoning
We are almost at the end of Site Reliability Engineering: How Google Runs Production Systems and that means it's time to pick a new book. Please send your suggestions to hello@bookclub.dev.
This week we are discussing Chapter 27 - Reliable Product Launches at Scale and Chapter 28 - Accelerating SREs to On-Call and Beyond.
Join the Thursday night discussions from 7-9 pm Eastern by signing up at bookclub.dev/thursdays.
Resources
- Appendix E
- The Checklist Manifesto
- Wheel of Misfortune Online
- Wheel of Misfortune Template
We look at Chapter 25 - Data Processing Pipelines and Chapter 26 - Data Integrity: What You Read Is What You Wrote.
Join the companion discussion Thursdays from 7-9 PM Eastern by signing up at bookclub.dev/thursdays.
System prevalence
Prevayler Java implementation of system prevalence.
Types of failure that lead to data loss
Root cause
- user action
- operator error
- application bug
- infrastructure defect
- hardware fault
- site disaster
Scope
- wide
- narrow
Rate
- big bang
- slow and steady
Sign up for the Thursday night companion discussions at bookclub.dev/thursdays. Discussions go from 7-9 pm Eastern.
This week we look at chapter 23 Managing Critical State: Distributed Consensus for Reliability and chapter 24 Distributed Periodic Scheduling with Cron from Site Reliability Engineering: How Google Runs Production Systems.
Things to monitor with a distributed consensus system
- The number of members running in each consensus group, and the status of each process (healthy or not healthy)
- Persistently lagging replicas
- Whether or not a leader exists
- Number of leader changes
- Consensus transaction number - in a healthy system, this number will be incremented over time
- Number of proposals seen and number of proposals agreed upon
- Throughput and latency
- Latency distributions for proposal acceptance
- Distributions of network latencies observed between parts of the system in different locations
- The amount of time acceptors spend on durable logging
- Overall bytes accepted per second in the system
Projects mentioned
- Apache ZooKeeper™
- Consul
- etcd
Articles from this week's chapters
- Harvest, Yield, and Scalable Tolerant Systems
- Brewer’s Conjecture and the Feasibility of
Consistent, Available, Partition-Tolerant Web
Services - CAP Twelve Years Later: How the "Rules" Have Changed
- Existential Consistency:
Measuring and Understanding Consistency at Facebook - The trouble with timestamps
- F1: A Distributed SQL Database That Scales
- Impossibility of Distributed Consensus With One Faulty Process
- The Part-Time Parliament
- In Search of an Understandable Consensus Algorithm
(Extended Version) - Zab: High-performance broadcast for primary-backup systems
- Mencius: Building Efficient Replicated State Machines for WANs
- ZooKeeper Recipes and Solutions
- ZooKeeper: Wait-free coordination for Internet-scale systems
- The Chubby lock service for loosely-coupled distributed systems
- Stumbling over consensus research: Misunderstandings and issues
- Paxos for System Builders: An Overview
- Implementing Fault-Tolerant Services Using the State Machine
Approach: A Tutorial - Spanner: Google’s Globally-Distributed Database
- The Google File System
- Bigtable: A Distributed Storage System for Structured Data
- MapReduce: Simplified Data Processing on Large Clusters
- Unreliable Failure Detectors for Reliable Distributed Systems
- Paxos Replicated State Machines as the Basis of a High-Performance Data Store
- Photon: Fault-tolerant and Scalable Joining of Continuous Data Streams
- High-Availability at Massive Scale: Building Google’s Data Infrastructure for Ads
- Paxos Quorum Leases: Fast Reads Without Sacrificing Writes
- Fast Paxos
- Classic Paxos vs. Fast Paxos: Caveat Emptor
- Egalitarian Paxos
- Tuning Paxos for High-Throughput with Batching and Pipelining
- Paxos made live: an engineering perspective
- Practical Byzantine Fault Tolerance
- WHAT TAKES US DOWN?
- Large-scale cluster management at Google with Borg
This week we look at chapter 21 Handling Overload and chapter 22 Addressing Cascading Failures of Site Reliability Engineering: How Google Runs Production Systems.
Sign up for the free companion discussion at bookclub.dev/thursdays. The discussion is from 7-9 pm eastern on Thursdays.
Overload is the leading cause of cascading failures.
Overload is what happens when a server gets more requests than it can handle. There are basic methods for relieving server overload:
- make the requests easier to process
- add more resources
- reduce the amount of traffic
This week we look at chapter 19 Load Balancing at the Frontend and chapter 20 Load Balancing in the Datacenter.
Join the companion discussion Thursday at 7 pm Eastern by signing up at bookclub.dev/thursdays.
Resources
- The Pragmatic Programmer: Your Journey To Mastery, 20th Anniversary Edition
- The Actor Model
- The Twelve-Factor App
Join the free companion discussions Thursday night at 7 pm Eastern.
Sign up at BookClub.Dev/Thursdays.
Share the show BookClub.dev/Reading-Notes.
This week we look at chapter 17 Testing for Reliability and chapter 18 Software Engineering in SRE
Resources
- Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations
- The Netflix Simian Army
- The Pragmatic Programmer: Your Journey To Mastery, 20th Anniversary Edition
This week we look at Postmortem Culture: Learning from Failure and Tracking Outages.
Join the free companion discussion for this episode on Thursday at 7 pm Eastern. Sign up at bookclub.dev/thursdays.
Links
Appendix D: Example Postmortem
Dilbert: Must Escalate
Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations
Westrum organizational culture
Disaster Recovery Testing
Wheel of Misfortune Template
Join the free companion discussions Thursday nights at 7 pm Eastern. Sign up at bookclub.dev/thursdays.
This week we look at chapter 13 Emergency Response and chapter 14 Managing Incidents.
Incident Command System Resources
Best Practices for Incident Management
Prioritize.
Stop the bleeding, restore service, and preserve the evidence for root-causing.
Prepare.
Develop and document your incident management procedures in advance, in consultation with incident participants.
Trust.
Give full autonomy within the assigned role to all incident participants.
Introspect.
Pay attention to your emotional state while responding to an incident. If you start to feel panicky or overwhelmed, solicit more support.
Consider alternatives.
Periodically consider your options and re-evaluate whether it still makes sense to continue what you're doing or whether you should be taking another tack in incident response.
Practice.
Use the process routinely so it becomes second nature.
Change it around.
Were you incident commander last time? Take on a different role this time. Encourage every team member to acquire familiarity with each role.
This week we look at chapter 11 Being On-Call and chapter 12 Effective Troubleshooting.
You can join the companion discussions Thursday at 7 pm Eastern for free by signing up at bookclub.dev/thursdays.
If you've found these episodes useful, please tell me about it by emailing hello@bookclub.dev.
Resources from the show:
- Practice of Cloud System Administration, The: DevOps and SRE Practices for Web Services
- Figure 12-1
- Best Guess Who Strategy
This week we look at chapter 9 Simplicity and chapter 10 Practical Alerting from Time-Series Data from Site Reliability Engineering: How Google Runs Production Systems.
If you've found these episodes useful, please send me an email at hello@bookclub.dev and tell me about it.
The companion discussions to this podcast happen Thursdays at 7 pm Eastern. You can join by signing up at bookclub.dev/thursdays.
In the episode, I mention some books I've enjoyed.
- The Goal
- The Phoenix Project
- The Unicorn Project
- Clean Architecture
Prometheus is mentioned as a tool for monitoring and alerting along with AppInsights.
Next week we'll look at chapter 11 Being On-Call and chapter 12 Effective Troubleshooting.
This week we look at The Evolution of Automation at Google and Release Engineering.
The companion discussion to this week's episode starts at 7 pm Eastern. You can join by going to bookclub.dev/thursdays.
Notes
Automation is meta-software, software that acts on other software
The Value of Automation
- Consistency
- A Platform
- Faster Repairs
- Faster Action
- Time Savings
The levels of automation
- No automation
- Externally maintained system-specific automation
- Externally maintained generic automation
- Internally maintained system-specific automation
- Systems that don't need any automation
Reliability Is The Fundamental Feature
XKCD: 1205
Philosophy of Release Engineering
- Self-Service Model
- High Velocity
- Hermetic Builds
- Enforcement of Policies and Procedures
Bazel
Next week we'll read chapter 9 Simplicity and chapter 10 Practical Alerting from Time-Series Data
Join the companion discussion on Thursday at 7 pm Eastern bookclub.dev/thursdays
Eliminating Toil "If a human operator needs to touch your system during normal operations, you have a bug. The definition of normal changes as your systems grow."
-Carla Geisser, Google SRE
4 types of work
- Software engineering
- Systems engineering
- Toil
- Overhead
What makes it toil?
- Manual
- Repetitive
- Automatable
- Tactical
- No enduring value
- Scales linearly with service growth
None of these individually are enough to make it toil, but the more boxes it checks the more likely it is toil
Toil isn't always bad, and it is not possible to completely eliminate
Figuring out somethings type of work largely revolves around how much value it creates and on what time scale
Toil tends to grow and expand if left unchecked
Tracking types of work
Google has quarterly surveys to ensure they are meeting or beating their <= 50% toil time
"If we all commit to eliminate a bit of toil each week with some good engineering, we’ll steadily clean up our services, and we can shift our collective efforts to engineering for scale, architecting the next generation of services, and building cross-SRE toolchains. Let’s invent more, and toil less."
Monitoring Distributed Systems Terms around monitoring
Not consistent, but a basic idea
- Monitoring
- White-box monitoring
- Black box monitoring
- Dashboard
- Alert
- Root cause
- Node and machine
- Push
What can you get from monitoring?
Trends, help debugging, alerts, baselines to compare from, data for the business to analyze, and things to analyze in the event of a security breach
This is a large scale endeavor. Every 10-12 person has at least 1 "monitoring person"
Even with a dedicated person, the monitoring needs to be simple enough for everyone on the team to understand, especially if it's something that triggers a page.
White-box and black-box monitoring
Black-box can tell you when there is an issue, but not when there is going to be an issue
White-box can see inside the system and see when there are imminent problems on the horizon
Not only is white-box predictive, but for certain things like an application thinking a DB is slow it's the only way to distinguish between an issue with the network and an issue with the database
Symptoms vs causes
Monitoring should address 2 questions, what's broken and why
Table 6-1 shows some symptoms and causes
A symptom is "I'm serving HTTP 500s and 404s" the cause is "Database servers are refusing connections"
Paging should be based on symptoms while data around causes should be used for debugging
From the perspective of someone monitoring an application "Database servers are refusing connections" should not generate the page, "I'm serving HTTP 500s and 404s" should.
At the same time if you are monitoring the DB "Database servers are refusing connections" is a symptom for you and not a cause
4 golden signals
- Latency
- Traffic
- Errors
- Saturation
Scale and accuracy
Find the right resolution for your needs
You can't see more detail than what you collect. Monitoring at the finest detail is costly and creates a lot of noise
If your SLA is 99.9 than checking something more than once a minute is probably unnecessary
What you monitor will change
You will find new things to monitor, but also some things will prove to not be worth monitoring anymore.
If you are collecting some data but not using it in any alerts or dashboards is it worth keeping?
Pagers and Pages
- Every time the pager goes off, I should be able to react with a sense of urgency. I can only react with a sense of urgency a few times a day before I become fatigued.
- Every page should be actionable.
- Every page response should require intelligence. If a page merely merits a robotic response, it shouldn’t be a page.
- Pages should be about a novel problem or an event that hasn’t been seen before.
Here are some questions to ask about your alerts to make sure you aren't paging for the wrong things
- Does this rule detect an otherwise undetected condition that is urgent, actionable, and actively or imminently user-visible?
- Will I ever be able to ignore this alert, knowing it’s benign? When and why will I be able to ignore this alert, and how can I avoid this scenario?
- Does this alert definitely indicate that users are being negatively affected? Are there detectable cases in which users aren’t being negatively impacted, such as drained traffic or test deployments, that should be filtered out?
- Can I take action in response to this alert? Is that action urgent, or could it wait until morning? Could the action be safely automated? Will that action be a long-term fix, or just a short-term workaround?
- Are other people getting paged for this issue, therefore rendering at least one of the pages unnecessary?
Short and long term balance
Responding to a page is toil, and it takes away resources from more valuable work. Finding the root cause and resolving it is often the best thing to do for the long term. If it can't be resolved, fully automating the response is the next best option
The book shares two case studies
BigTable over alerting, the solution was to lower the SLO to create space to solve underlying issues
Gmail had an issue where the team was concerned if they automated away a rote task the real underlying issue would not be addressed. This showed a lack of faith in the team's ability to clean up their technical debt which is a deeper issue that needs to be addressed, probably by escalating.
The Thursday night discussions start at 7 pm Eastern.
Sign up at bookclub.dev/thursdays
This week I share my notes from chapters 3 & 4 of Site Reliability Engineering: How Google Runs Production Systems
Topics covered
- Error budgets
- Calculating availability
- SLAs
- SLOs
- SLIs
This week we read the forward, preface, and the first two chapters of Site Reliability Engineering. We discuss the origins and basic tenants of SRE, look at how Google manages risk, and think about how we can incorporate SRE into our work. You can join our free discussions Thursdays at 7 pm Eastern by signing up at https://www.bookclub.dev/thursdays.
Resources
- The Wheel of Time Series (Amazon)
- Awareness: The Perils and Opportunities of Reality (Amazon)
- SRE Book companion site
- Principles of Network and System Administration (Amazon)
- Practical Reliability Engineering (Amazon)
- Facts and Fallacies of Software Engineering (Amazon)
- The Factors That Impact Availability, Visualized
- The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines, Second Edition
- A Study of Non-Blocking Switching Networks
- Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network
- B4: Experience with a Globally-Deployed Software Defined WAN
- BwE: Flexible, Hierarchical Bandwidth Allocation for WAN Distributed Computing
- Large-scale cluster management at Google with Borg
- MapReduce: Simplified Data Processing on Large Clusters
- The Google File System
- Bigtable: A Distributed Storage System for Structured Data
- Spanner: Google’s Globally-Distributed Database
- The Chubby Lock Service for Loosely-Coupled Distributed Systems
- Searching for Build Debt: Experiences Managing Technical Debt at Google
- The Motivation for a Monolithic Codebase: Why Google stores billions of lines of code in a single repository
- Borg, Omega, and Kubernetes
We are reading Site Reliability Engineering: How Google Runs Production Systems.
Next week we will discuss the first two sections, Introduction and The Production Environment at Google, from the Viewpoint of an SRE.
You can join our discussion Thursdays at 7 pm Eastern for free by signing up at bookclub.dev/thursdays.
Google offers a free digital version