If you've spent any time working with software developers in the past 20 years you've probably heard the terms "waterfall", "agile", and "scrum" with varying levels of derision. I've been writing software for most of those 20 years and I have some... let's say strong... feelings about those terms. Since this is my blog and I get to write whatever I want, I figured I'd take a crack at explaining these things and why they are derided, fairly and unfairly.
Waterfall
Waterfall is kind of a catch-all term most commonly used as a synonym for "the worst way to develop software" these days. In my humble opinion, that is absolutely spot-on. Waterfall sucks. An oft-cited problem with the waterfall development model is that it takes too long to deliver anything. Waterfall got its name from the way the phases of the project look in project planning software: the first phase has to complete before the next phase can begin.
I'll give you an example: I once worked as a consultant for a major health insurance company. They had a major initiative, the details of which I can't even remember anymore, but it was a Big DealTM. To accomplish this major initiative, the Powers That Be spent a significant amount of time setting forth everything they wanted the initiative to encompass. I have no idea how long they spent on this venture, but when they were finished they turned over an 18 page document of "business requirements". One of my fellow consultants then spent several weeks (I think six, but I could be wrong) converting that business requirements document into a technical requirements document. Then we started working on the actual code. We spent six-ish months implementing every single scenario we could think of that might come up during testing in order to fulfill those technical requirements and we finally delivered what we had to QA. They destroyed it. They had built test cases around the original business requirements, we had written our code based on the technical requirements, and apparently some things had been lost in translation. The translator had since left the consulting firm. Everything was a mess. We spent another two weeks fixing the code, then two more weeks having it all tested again and it was finally ready. Roughly 8 months after it was requested. It was not what the Powers That Be had envisioned and the changes were scrapped.
If Waterfall is so bad, why did we use it? Well, because it worked. Software used to be delivered on much longer timelines. If you're old enough to think about computers and software in the '90s, we weren't cranking out new features on any kind of regular basis because there was no reason to. Even Windows didn't really have patches, and if they did they had to be shipped out on floppy disks. The Internet changed everything. With the Internet - particularly high speed Internet service - we could deliver updates to software quickly and easily to all of our customers, and our customers knew that and (rightfully) demanded bug fixes. Waterfall was still the dominant software development methodology, but that wasn't going to work for the future.
Agile
Enter Agile. I've made a few jumps from Waterfall to Agile, but ultimately those jumps are small stepping stones on the larger path from Waterfall to Agile (the generically named iterative development is one such stepping stone, which was basically waterfall, but with shorter cycles). What is Agile? This is where things start to get confusing for people, I think. Agile is "a mindset". Super helpful, right? What the hell does it mean to be "a mindset"? In this case it simply means we're going to think about delivering software differently than we used to. The Agile Mindset was delivered unto us in the form of The Agile Manifesto.
A group of software folks got together and asked themselves how they could write software better. They came up with some bullet points they could refer back to and they published those bullet points as The Agile Manifesto. The Agile Manifesto makes some suggestions about what we can do to deliver better software that our customers actually want, but doesn't tell us anything about how to do it. It's just a list of suggestions on how we might do things, based on collectively decades (at that time) of software development experience. That's all Agile is. That's it. Agile doesn't prescribe a set of activities or actions or how we define requirements or anything. It's just a general set of suggestions (a mindset) that we can follow to write and deliver better software. Full stop.
Scrum
We've finally arrived at the much-maligned Scrum. In my experience people either love it or hate it, and they usually fall on the hate side. You can find a ton of hate online for Scrum (check out r/programmerhumor on Reddit if you want a taste) or just talk to pretty much any developer who has ever been subjected to daily standups that go on for an hour and planning meetings where nothing much gets planned. They'll all say some variation of the same thing: Scrum just added meetings to my schedule and didn't actually help me deliver software any better. Unfortunately for those people - and the reputation of Scrum - that's just abusing the name of Scrum. When done properly, Scrum works. Let me start by briefly explaining what Scrum is and what it is not. Then I'll go into areas that I see go wrong the most often.
Scrum is honestly pretty simple. You can get the gist of it by reading the Scrum Guide (it takes 20 - 30 minutes), but I'll go over the basics here since you're already reading my blog. Scrum just puts a framework around the Agile methodology, dictating specific events and what must happen during those events. Scrum defines three roles that participate on a Scrum Team and what each of those roles is supposed to do.
Roles
Scrum Master
Scrum Master is probably the most hated on position in software engineering. I've heard developers say Scrum Masters are useless, they have to justify their job by making the Developers' lives hard, and they don't even bring anything valuable to the team. Scrum Masters are important, but if they're doing their job properly it looks like they're not doing much. The whole purpose of a Scrum Master is to - as I call it - shepherd the process (I'm sure I picked that up somewhere so don't go thinking too highly of my creativity). The Scrum Master introduces and supports Scrum as a concept and a framework to the Scrum Team and the organization as a whole. In the event Developers are prevented from moving forward for some reason the Scrum Master seeks to fix whatever is blocking them. It's a pretty straightforward job that I do while also being a contributing software engineer on the Scrum Team.
Product Owner
The Product Owner owns the product. S/he is responsible for working with users to figure out what to have the product do. The primary job of the Product Owner is to define what they want the product to do so the Development Team can build it. They do that by putting items on the Product Backlog (called - shockingly - Product Backlog Items) and then clarifying those requirements for the Development Team.
Developer
"Developer" comprises everyone on the Scrum Team who isn't the Product Owner or Scrum Master. Whether their skills are front-end, back-end, full-stack, testing, or anything else makes no difference. Everyone who participates on the work product to move it toward the Product Goal is a Developer. That's it.
Events
The events of Scrum are: The Sprint, Sprint Planning, Daily Scrum, Sprint Review, and Sprint Retrospective.
The Sprint
The Sprint is a bit of a unique "event" in that it is a container for all of the other events. The Sprint has a pre-defined time period of up to one month. The most common length of a Sprint in my experience is two weeks. During those two weeks, the Scrum Team works toward delivering an Increment - a set of deliverables that move the Scrum Team toward delivery of the Product Goal - while engaging in the rest of the Scrum Events.
Sprint Planning
At the beginning of the Sprint, the Scrum Team meets to review the Product Backlog - the list of all work that defines what the Product will be when it is complete - and determine which parts of the Product Backlog the Scrum Team believes they can complete during the Sprint. Product Backlog Items (PBIs) are measured by the team for their complexity: how tricky is the PBI going to be to implement, how much risk is involved in implementing it, and how much is unknown about what needs to be done. It is important (and difficult) to remember that complexity may not correlate to actual time spent on a PBI once the work is taken on. Complexity is commonly measured using story points in the Fibonacci sequence or "t-shirt sizes".
After the complexity is set for a PBI, I find it best to have the Developers break each PBI down into tasks and estimate the hours of each task. This is the best guess the Developers can make for each item based on their experience. They might end up working many more or many fewer hours on each task; these are just estimates. Once the Scrum Team agrees they have enough work to fill the duration of the Sprint, they stop bringing work in. However, the Sprint Planning meeting may continue so the Scrum Team can ask questions about future work and even perhaps assess complexity of future PBIs (without diving into the tasks of those stories). Doing this can help the Scrum Team provide an estimate for when a given PBI might get worked on.
Daily Scrum
The purpose of Daily Scrum is for the Developers to tell each other how things are going. It is an opportunity to explain to each other where they're stuck, what they learned, helpful tips about something they've been working on, or anything else relevant to the Sprint. Since Scrum Masters aren't managers and managers shouldn't be in the Daily Scrum there's no reason to provide information based on what management expects to hear. If you're doing that, you're doing Scrum wrong.
Sprint Review
In my opinion, Sprint Review and Retrospective are the most important events in Scrum (not that any of them are optional). Sprint Review is the time when the Scrum Team goes over what they accomplished during the Sprint. It is common for key stakeholders to join this meeting and provide feedback on what was delivered in the Increment as well as guidance for what they'd like to see in upcoming Increments. However, it is vital to not turn the Sprint Review into a demo. The feedback is the most important part so hearing what the Product Owner and stakeholders have to say is key to the success of the meeting. I like to tell people the Sprint Review is the opportunity to review and improve the Product so use this time to work to that end.
Sprint Retrospective
Finally we come to the Sprint Retrospective. This is an opportunity for the Scrum Team to meet alone (no management; no stakeholders; just Developers, the Scrum Master, and the Product Owner) to discuss how the Sprint went. There are lots of ways to design a good retrospective, but I like to use a site called Retromat to build a retrospective plan. As the Sprint Review is where the Scrum Team reviews and improves the Product, the Retrospective is where the Scrum Team reviews and adopts changes to the people and processes. For that reason, Retrospective has to be confidential and protected. Nothing should ever come out of Retrospective except Action Items (which are one to two concrete actions the Scrum Team agrees to take to improve). The team is free to discuss anything related to the Sprint as it relates to the team. Did someone constantly show up late to Daily Scrum? Say something. Did you feel like you did all the work in the Sprint? Say something.
The ultimate goal of the Sprint Retrospective is to be a better team, but that can't happen if people hold back their true feelings on how each sprint goes. Be open. Be honest. Be fair.
Misunderstandings
Everyone Does Scrum Differently
I think the easiest place to start is with the idea that Scrum is different at each company. That's not usually true. Scrum is a framework and while we are free to adapt within that framework we cannot operate outside the framework and still be doing Scrum. If you are doing Scrum then you are adhering to the Scrum framework. If you're not adhering to the Scrum framework then you're not doing Scrum. And that's OK! Sometimes Scrum isn't right for your organization so you can't adhere to the Scrum framework. There are lots of other methodologies out there that work very well. If you're aggravated by Scrum then look closely at what you were doing and compare that to the Scrum framework. My bet is you weren't actually doing Scrum.
Scrum Master as Team Lead/Manager
In my experience, this is commonly the result of the Scrum Masters themselves not understanding their role as a facilitator and instead falling into a team manager. I've seen Scrum Masters assign tasks to Developers. I've seen Scrum Masters ask Developers during Daily Scrum for a detailed update of the work they completed the previous day. You've probably seen worse. The Scrum Master is absolutely not above anyone and absolutely should not operate like they are.
Daily Scrum as a Report to Management
I've already touched on this, but it's worth repeating: Daily Scrum is the time for Developers to communicate with each other, not with management. Management shouldn't even be in the Daily Scrum and the Scrum Master isn't in charge of anything. Daily Scrum is just for Developers to communicate how well they're moving toward the Sprint Goal.
Daily Scrum as Only a Status Update
I'm beating a dead horse here, but it's a common gripe. Communicate. Communicate. Communicate! That's what Daily Scrum is for. If you find your Daily Scrum meetings entail a bunch of people droning on about what they did yesterday, what they plan to do today and how they have no impediments, you're probably not getting the most out of your Daily Scrum. Tell your teammates where you got stuck or what you're stuck on now, even if it doesn't seem to you like it's an impediment and even if you don't want anyone's help resolving the problem right now. It's an opportunity for someone else to say "Oh, I'm dealing with that exact same thing. What have you tried?" COMMUNICATE!
Work Can Change on the Fly
Once Sprint Planning has been completed, the work in the Sprint is fixed unless the Scrum Team agrees to change it. That explicitly does not mean the Product Owner can demand the Developers work on something new. There can be a negotiation process if priorities have changed or something urgent has come up since Sprint Planning, but it's always up to the whole team whether they change what they're working on. If you find you're constantly being asked to change work once the Sprint starts, try shortening your sprints.
Some Ceremonies are Optional
I can't stress this enough: all five Scrum Events are required. If you're not doing one of them, you're not doing Scrum. You don't cancel the Sprint Review because you don't think you completed enough. You don't cancel the Sprint Retrospective because last time no one participated. Every single event serves a purpose and every single event is absolutely required.
Conclusion
So that's it. That's my take on software development in the middle of my career. Scrum isn't perfect, but when you do it properly, it's damn good. And anything is better than waterfall.
