Describe Your Ideal Development Process

All too often discussions about how to improve a team’s/company’s software development process are met with “It won’t work here”, so I find it often helps to discuss the ideal process and then look at the next step(s) that could be taken to move closer to it. Yes, you might not get there in its entirety, but that’s why it’s an ideal process and we’re just trying to move closer to it.

This video by Dave Farley is a presentation of what an ideal process might be, so I’m going to step through it and expand on some of his points. Feel free to disagree with what that ideal looks like, but for now, suspend any thoughts of “it won’t work here” and just focus on what you would like it to be.

The Goal of Software Development

  • Solve Problems for Others, with Software
    It’s hard to disagree with this premise:
    Know your customer

    • if you’re not solving a problem, then why are you building something? Now it could be that you’re building something truly revolutionary and your target audience doesn’t know they have a problem yet, but someone has identified an opportunity in the market. Maybe we need a broader understanding of what “a problem” is but if we’re not building a solution then we should probably stop.
    • Dave mentioned that you’re being paid to do it, so it must be “for others” unless you have some convoluted cross-billing system for your personal finances! (Even if I’m building something for myself I tend to use a simplified version of this process but let’s focus on professional software development organisations for this post.)
    • I think “with software” is a reasonable assumption given we’re talking about a software development process but I will just add that one of my favourite solutions to a customer problem was when the team laminated some instructions rather than build a new help tool.
  1. Understand the Problem
    You keep using that wordDave mentions requirements. I find teams often misunderstand what this word means. A “requirement” is just a description of the problem… and the description could be in the form of diagrams and conversations. It does not have to be specified in great detail, written down and signed off – in fact, that’s exactly what we don’t want!
  2. Solve the Problem
    Write some code. I don’t think this needs an explanation.
  3. Confirm the Problem is Solved
    I’ve encountered disagreement when discussing the need for writing tests but (remember this is an ideal process we’re discussing) everyone wants tests to exist. Rapid, reliable, repeatable tests. There was unanimous agreement that flaky tests are worse than not having them.

I agree with Dave’s summary that the Ideal Dev Process results in Great Quality with Minimal Work, but this is where I’ve seen people get confused by “minimal work”. Often they take it to mean that they need to know everything before they start so that they only do exactly what’s needed, but (the video will get to this in a minute) we cannot know everything upfront – sometimes the customer doesn’t know exactly what they want (or need), but even when they do it’s hard to convey that precisely. But let’s assume the dev team perfectly understand the goal; there’s more than one way to reach that goal so we may need to try some alternatives before we pick one. Oh, and of course, everyone writes perfect bug-free code the first time. So let’s agree that “minimal work” means “based on what we know at the time we’re doing it” which leads nicely into Dave’s discussion about continuous learning during the process.

For those reasons (and more) we should be discovering new information throughout the development process. We could be learning more about what the problem is, how to solve it, or how we work together to solve it. Any new information should result in the team deciding how to incorporate it; that may result in changing work which was previously considered finished, but what’s the alternative? If your customer initially said they wanted the screen to be blue but now they want it to be green, would ignoring that result in delivering Great Quality? I won’t get into the practicalities here but on my backlog of blog topics, I have the sunk cost fallacy and what to do if your customer keeps changing their mind.

The ability to incorporate new information requires the willingness and ability to change our minds and change our software.

How do we make it “Easy to Spot the Need for Change”? Frequent feedback from the right people. We should be asking our customers (and end-users) for feedback on whether we’re solving the problem; sometimes that leads them to realise the problem they thought they had isn’t the one we need to address first. We should also be asking the people involved in creating the solution (the dev team but also the people we are working with who are outside the team) for feedback on how we’re collaborating. In both cases that means taking small steps.

Safety netWe also need to make it “Easy & Safe to Make Changes” – we need safety nets to make it easier to spot when we’ve made a mistake and to fix it. Those safety nets can consist of tools, tests, processes and procedures; together they help developers move with more confidence.

Dave goes on to explain why setting up a deployment pipeline is essential, and I completely agree with him – if you build a system by hand then you run the risk of making mistakes, leading to inconsistencies from build to build. This is as bad as having flaky tests – you cannot rely on the results, so no matter how good your code is you cannot be confident in what you deliver to your customer. Automating the integration process also reduces the effort it takes, which means you can build more often, which in turn means you can give your users a new release more often and thus get feedback more frequently.

Part of the safety net needed to bolster developers’ confidence when making changes is another feedback loop: does your new code work with the rest of the system? There needs to be a way to let the team know if a change has broken the system, and then the team need to make it their priority to fix it. If the system is broken, why would you continue making changes before you fix it? You won’t know if your new changes work until you test them and you can’t test without a working system, so all you’re doing is postponing that knowledge, which in turn means you may need to backtrack further to fix your code – that doesn’t meet our aim of Minimal Work.

So the principles of our ideal development process are:

  • minimal work (minimize unnecessary work)
  • great quality
  • ability to change our minds
  • make changes in small steps
  • regularly verify progress (frequently seek feedback)
  • work incrementally (incorporate feedback)
  • creates working, usable system
  • keeps the system working

The video goes into more detail about Continuous Integration and Continuous Deployment, but I think that if we can agree on these principles of an ideal development process then CI/CD is a given.

Finish lineI read some of the comments under the video on YouTube and a major thread was around estimation. Given that the focus of Dave’s presentation was on the development process, I think estimation is out of scope (you can develop great quality software without estimation) but it probably deserves similar treatment, i.e. thinking through what an ideal product/project management process might look like. Estimation is a possible solution depending on the problem, but often it’s assumed that it’s needed instead of asking “Why, what will the estimates be used for? What problem are you facing for which you think estimation is the solution?” Often it’s considered necessary because someone wants to know “when will we be done?” but that should be a question to the customer as part of that frequent feedback loop – “are we getting closer to what you want?”. If we don’t know where the finish line is, how can we hope to say when we’ll reach it? Instead, focus on making small incremental steps and also asking whether taking another step down that path is the most valuable thing to be doing – sometimes we’ve gone far enough that we should park this work and move on to something else.


I linked to it a couple of times in my post, but in case it’s not clear the video I’m referencing is “The Most Powerful Software Development Process Is The Easiest” by Dave Farley of Continuous Delivery.

Long time, no see

It’s time to blow the cobwebs off this blog and get back to a somewhat-regular posting schedule. I have a backlog of ideas and some past topics that might benefit from an update, but if you have an idea then please drop me a note – I’ve not tried co-creating a blog post on this site although I have for internal blogs at past employers, so let me know if that’s of interest too.

Incidentally, I was cross-posting these posts to Medium and then to LinkedIn and Facebook; I wonder if this is still the best approach.



Start With The End In Mind

We have completed another lap around the sun, and that’s often a time to think about our plans for the new year. In my previous post, I talked about New Year’s Resolutions and thinking in smaller steps rather than a goal that might take 12 months to complete. But if you only think in small steps and focus on where you want to be in a few days or weeks, then it’s often hard to know if you’re going in the right direction – there can be such a big gap between where you are and where you want to be.

If you were planning on travelling from Toronto to Montreal then you need to plan further ahead than just “which street do I take now?” You might think that was a bad example as the 401 covers the vast majority of the distance, but that assumes you’re optimising for time and that your method of travel is by road, i.e. you have already ruled out many alternatives. Take a step back and consider your goals: if minimising travel time is the primary aim, then flying is significantly faster. If you want the freedom to move around and have a meal without it impacting your travel time, then the train could be your best option.

But let’s go back to driving between the two cities, and assume we have picked the route we would like to take. We set off but before long we discover there’s a problem ahead: an accident is causing a long delay. If we had only been inspecting our progress over small steps, then we might not see the impact of the accident until we’re sitting in the traffic jam and it’s too late to do anything else. We would see a similar outcome if we set the route and followed it regardless of what we discovered along the way. One of the reasons we chose to drive is to have flexibility – we can’t easily change our route if we chose to fly or take the train.

Choosing flexibility (aka agility) may not be the cheapest, fastest or most comfortable path, but it gives us options when new information (whether that’s delays or shortcuts) is available … so we should ensure that we take advantage of that discovery, otherwise we probably shouldn’t have picked this approach!

The plans we create should provide clarity on where we want to be in the future (and why!) but we also should revise them when we see a better alternative. Sometimes it’s a detour that still gets us to our destination but in a better (faster / cheaper / safer) way, but sometimes the discovery is that our goal needs to change – maybe the reason we were heading to Montreal can also be achieved in Ottawa, saving about an hour of driving. If we don’t respond to that new information, then why didn’t we just get on the train, put our feet up, and just enjoy being passengers?

Incidentally, this is why we call some plans “roadmaps” – their purpose is to show potential alternative routes and to allow for changing course while still heading towards the destination. If our plan has only one path from A to B, then it’s not a roadmap. If the plan says “build X” rather than “solve problem Y”, then it’s not a roadmap.

I’m not saying you always have to drive (i.e. use an agile approach) – there are times when the train makes sense, for example when I travel to Montreal for a concert I know the destination isn’t going to change and I would rather get there with minimal stress. In the same way, if we know exactly what we need to build and don’t expect any changes along the way, then we could create a plan, write detailed specifications, and do all the design upfront … but, in our industry, how often can we go from A to B with no learning, no discoveries, no need to adapt?

So: start with the end in mind (understand where you want to go and why); consider what your constraints are (e.g. time) and how you expect to react to new information (do you want to accommodate discoveries or follow the plan); then choose your vehicle and route accordingly.

*Yes, I know the second habit that Stephen Covey describes in “The 7 Habits of Highly Effective People” is “Begin…” but he has a registered trademark for that phrase so I’m avoiding it, and anyway I’m not using the phrase in the way he does.

New Year’s Resolutions

I’m sure most people will be happy to see the end of 2021 but also trying not to build up too much expectation for 2022. There won’t be a sudden end to COVID – it’s going to take a continued effort from everyone. It won’t go away just because we’re all tired of dealing with it. Stay strong; hang in there; support each other; and keep safe, please.

The New Year is a time when many people set Resolutions – things they want to improve, e.g. eating better, getting more exercise. It’s a handy reminder to review how happy we are with various things, but is doing it once a year really enough? It often leads to huge goals because it’s going to be so long until the new review… and, of course, huge goals aren’t usually achievable. That’s why most resolutions are abandoned within a few weeks. It probably helps to have a vision of how you would like things to be in a few months (even a year), but set smaller goals – what are the steps that you think will take you towards that vision?

Hopefully that sounds familiar: regular retrospection, identifying where you want to improve, deciding on small steps to go in that direction; take a step, then go back to retrospection, rinse and repeat.

Engineers and product owners should be well practised in this – retrospectives to look at how we work, and experiments to probe what to build. So how do we convince HR that annual appraisals should be replaced with frequent feedback, and Finance that regular (e.g. monthly) planning trumps attempting to predict six or even twelve months ahead? That’s a challenge I’d like to add to my backlog for next year. Suggestions are welcome!

p.s.
2014: Didn’t jog
2015: Didn’t jog
2016: Didn’t jog
2017: Didn’t jog
2018: Didn’t jog
2019: Didn’t jog
2020: Didn’t jog
2021: Still haven’t jogged
This is a running joke! 🙂