Describe Your Ideal Development Process

All too often discussions about how to improve a team’s/company’s software development process are met with “It won’t work here”, so I find it often helps to discuss the ideal process and then look at the next step(s) that could be taken to move closer to it. Yes, you might not get there in its entirety, but that’s why it’s an ideal process and we’re just trying to move closer to it.

This video by Dave Farley is a presentation of what an ideal process might be, so I’m going to step through it and expand on some of his points. Feel free to disagree with what that ideal looks like, but for now, suspend any thoughts of “it won’t work here” and just focus on what you would like it to be.

The Goal of Software Development

Solve Problems for Others, with Software
It’s hard to disagree with this premise:
- if you’re not solving a problem, then why are you building something? Now it could be that you’re building something truly revolutionary and your target audience doesn’t know they have a problem yet, but someone has identified an opportunity in the market. Maybe we need a broader understanding of what “a problem” is but if we’re not building a solution then we should probably stop.
- Dave mentioned that you’re being paid to do it, so it must be “for others” unless you have some convoluted cross-billing system for your personal finances! (Even if I’m building something for myself I tend to use a simplified version of this process but let’s focus on professional software development organisations for this post.)
- I think “with software” is a reasonable assumption given we’re talking about a software development process but I will just add that one of my favourite solutions to a customer problem was when the team laminated some instructions rather than build a new help tool.

Understand the Problem
Dave mentions requirements. I find teams often misunderstand what this word means. A “requirement” is just a description of the problem… and the description could be in the form of diagrams and conversations. It does not have to be specified in great detail, written down and signed off – in fact, that’s exactly what we don’t want!
Solve the Problem
Write some code. I don’t think this needs an explanation.
Confirm the Problem is Solved
I’ve encountered disagreement when discussing the need for writing tests but (remember this is an ideal process we’re discussing) everyone wants tests to exist. Rapid, reliable, repeatable tests. There was unanimous agreement that flaky tests are worse than not having them.

I agree with Dave’s summary that the Ideal Dev Process results in Great Quality with Minimal Work, but this is where I’ve seen people get confused by “minimal work”. Often they take it to mean that they need to know everything before they start so that they only do exactly what’s needed, but (the video will get to this in a minute) we cannot know everything upfront – sometimes the customer doesn’t know exactly what they want (or need), but even when they do it’s hard to convey that precisely. But let’s assume the dev team perfectly understand the goal; there’s more than one way to reach that goal so we may need to try some alternatives before we pick one. Oh, and of course, everyone writes perfect bug-free code the first time. So let’s agree that “minimal work” means “based on what we know at the time we’re doing it” which leads nicely into Dave’s discussion about continuous learning during the process.

For those reasons (and more) we should be discovering new information throughout the development process. We could be learning more about what the problem is, how to solve it, or how we work together to solve it. Any new information should result in the team deciding how to incorporate it; that may result in changing work which was previously considered finished, but what’s the alternative? If your customer initially said they wanted the screen to be blue but now they want it to be green, would ignoring that result in delivering Great Quality? I won’t get into the practicalities here but on my backlog of blog topics, I have the sunk cost fallacy and what to do if your customer keeps changing their mind.

The ability to incorporate new information requires the willingness and ability to change our minds and change our software.

How do we make it “Easy to Spot the Need for Change”? Frequent feedback from the right people. We should be asking our customers (and end-users) for feedback on whether we’re solving the problem; sometimes that leads them to realise the problem they thought they had isn’t the one we need to address first. We should also be asking the people involved in creating the solution (the dev team but also the people we are working with who are outside the team) for feedback on how we’re collaborating. In both cases that means taking small steps.

We also need to make it “Easy & Safe to Make Changes” – we need safety nets to make it easier to spot when we’ve made a mistake and to fix it. Those safety nets can consist of tools, tests, processes and procedures; together they help developers move with more confidence.

Dave goes on to explain why setting up a deployment pipeline is essential, and I completely agree with him – if you build a system by hand then you run the risk of making mistakes, leading to inconsistencies from build to build. This is as bad as having flaky tests – you cannot rely on the results, so no matter how good your code is you cannot be confident in what you deliver to your customer. Automating the integration process also reduces the effort it takes, which means you can build more often, which in turn means you can give your users a new release more often and thus get feedback more frequently.

Part of the safety net needed to bolster developers’ confidence when making changes is another feedback loop: does your new code work with the rest of the system? There needs to be a way to let the team know if a change has broken the system, and then the team need to make it their priority to fix it. If the system is broken, why would you continue making changes before you fix it? You won’t know if your new changes work until you test them and you can’t test without a working system, so all you’re doing is postponing that knowledge, which in turn means you may need to backtrack further to fix your code – that doesn’t meet our aim of Minimal Work.

So the principles of our ideal development process are:

minimal work (minimize unnecessary work)
great quality
ability to change our minds
make changes in small steps
regularly verify progress (frequently seek feedback)
work incrementally (incorporate feedback)
creates working, usable system
keeps the system working

The video goes into more detail about Continuous Integration and Continuous Deployment, but I think that if we can agree on these principles of an ideal development process then CI/CD is a given.

I read some of the comments under the video on YouTube and a major thread was around estimation. Given that the focus of Dave’s presentation was on the development process, I think estimation is out of scope (you can develop great quality software without estimation) but it probably deserves similar treatment, i.e. thinking through what an ideal product/project management process might look like. Estimation is a possible solution depending on the problem, but often it’s assumed that it’s needed instead of asking “Why, what will the estimates be used for? What problem are you facing for which you think estimation is the solution?” Often it’s considered necessary because someone wants to know “when will we be done?” but that should be a question to the customer as part of that frequent feedback loop – “are we getting closer to what you want?”. If we don’t know where the finish line is, how can we hope to say when we’ll reach it? Instead, focus on making small incremental steps and also asking whether taking another step down that path is the most valuable thing to be doing – sometimes we’ve gone far enough that we should park this work and move on to something else.

—
I linked to it a couple of times in my post, but in case it’s not clear the video I’m referencing is “The Most Powerful Software Development Process Is The Easiest” by Dave Farley of Continuous Delivery.

Happy Birthday Agile

It’s twenty years since the Agile Manifesto was created in Snowbird, Utah. There’s obviously been lots of people posting about whether it’s still relevant, whether it should be revised, and so on, but I think it’s still important and applicable.

It starts with: “We are uncovering better ways of developing software by doing it and helping others do it.”
It doesn’t claim to be definitive; “We are uncovering” means we still have room to learn and improve how we do this. There are many “flavours” of agile, including Heart of Agile and Modern Agile, and some organisations develop their own take on agile (but “hybrid” agile-waterfall monstrosities don’t count in my book).

The next few lines (i.e. the rest of the manifesto) seem to be problematic for some people because they don’t fully understand “we value the items on the left more” – the manifesto’s authors even spell it out by saying “while there is value in the items on the right” and yet some people still think “Agile means we don’t do documentation”.

The first of these, “we have come to value: Individuals and interactions over processes and tools” is one I’ve see many organisations fail to grasp. It’s a long-running joke in the community that “installing Jira” equals “adopting agile”. I’ll refrain from ranting about Jira because the point is that any tool is not the way to become agile – it’s about people. I remember another coach saying “If a company doesn’t understand that it’s all about people then they don’t deserve agile”! 🙂

The next line, “we have come to value: Working software over comprehensive documentation“, is the one I see people misunderstand most – I wonder if that’s because so many people hate writing documentation? If you think about the two extremes, neither is ideal: documentation without software is clearly not a useful product, but also software with no documentation is rarely desirable (a simple tool with no user interaction might just about be ok, but even then anyone trying to fix/enhance the code is likely to benefit from some docs). That’s why we value working software more than documentation but not instead of.

I think the third line is easier to appreciate if you’ve ever worked on a waterfall project where we’re expected to blindly follow the documented specifications and hit deadlines, even if it means delivering something we know to be wrong. By valuing “Customer collaboration over contract negotiation” it means we would rather work with our customer, e.g. show them demos (or better yet give them something they can use) and get feedback early and often. If you don’t work with your customer and respect them, then how can you expect to keep them as a customer? (Ignoring monopolies/cartels, obviously; add banks to the list of rants I’m avoiding in this post!)

The final line, “Responding to change over following a plan” is closely linked with the previous one – in an ever-changing environment like software development there is always going to be change, whether that’s feedback from the customer “course-correcting” the direction of the product, or how the team works together (e.g. coming out of a retrospective), or the tools/technology that we’re working with. To ignore those changes is to say that we can’t learn anything during a project; I can’t think of a single instance where that’s been true, even working on tightly constrained legislation/standards-based projects – there’s always something discovered along the way. However, just like documentation, that doesn’t mean we work without plans – we need to understand where we’re heading and how we think we can get there… but we also need to revise the plan when things change. We need a roadmap (i.e. a destination and options for getting there) rather than a single route set in stone … but that feels like a future post 🙂

Individuals and interactions

If you have ever tried to solve a complex problem, you probably discovered the benefits of working with someone else – whether that person had more experience than you or could just ask questions that challenged your thinking. Compare that to a time when you were handed a checklist and just ticked things off – which produced the better solution?
Continue reading “Individuals and interactions” →