Feb 10, 2006

Organizational change and Scrum

Over the last 6 months I've been trying to bring about organizational change in order to improve the product development process, change how product development interacts with the rest of the business and to improve the productivity of my department.

As you would expect there were two central ideas to this strategy. Firstly I needed to change engineering practices and secondly I needed to change the management of those practices. It was time to kiss goodbye the traditional waterfall/gantt chart based project management time to move to agile methods.

I started by migrating the team to agile development methods (based on the fundamentals of extreme programming) whilst still retaining the documentation needed to support our ISO accreditation. This has been quite successful. The team understands the need to improve, and the actual methods of producing software are fairly easy to alter since it is an internal process (ie internal to my department). The only areas I still need them to improve relate to refactoring and code reviews. This will come with time and I am quite happy with things being at the stage they are at this point in time.

I've also been changing the management of the development process to Scrum. I tried to do this gradually for a number of reasons, the main one being that I was new to the organisation and didn't want to upset the apple cart or be seen as the newbie trying to change something when they didn't understand the corporate culture.

Well, suffice to say, my attempts at the gradual adoption of scrum have been a complete disaster. Why? Because I only implemented portions of Scrum under the thinking that I need to give people time to get used to the new methods before implementing more change, and that too much change too quickly would upset the organisation. For example, I instituted the daily meetings to foster open communications between the team and I set up the prioritisation of tasks (the backlog) so that the other areas of the business were aware of what was happening.
I moved to regular releases of the system - down to a 3 week period instead of 12 months - and because there was no clear product owner I fulfilled that role as well as taking on the role of ScrumMaster.

Instead of instituting a change for the better I managed to institute anarchy. Filling both roles of ProductOwner and ScrumMaster led to conflicts of interest over what was best for the company over what was best for the team - I broke the rules of Scrum myself by allowing constant chopping and changing of direction based on need and failed to communicate reasons for change with the rest of the business. Obviously these failures are my own fault and lack of time and business growth compounded the errors I was making.

Unfortunately by not enforcing the rules of scrum, I allowed ad-hoc requests to be actioned, allowed staff to be pulled off for other non development work and as a result failed to get the team to own the process and to own the delivery of results.
I was doing the classic "tell you what to do and tell you when to do it" instead of letting the team self manage.
Effectively the daily meeting was a "report activity to Richard" session and was not acting as a team synchronisation meeting. Delivery commitments were estimates only with no repercussion for failure. The business saw the product backlog as just another way of tracking outstanding work requests but having no other value to the business.
The worst result was that without the team owning the product or the process they felt no responsibility for delivery. "Richard will tell me what to do next" was the common thought, and that was the last thing I wanted. It meant all responsibility was on my shoulders, made me the bottleneck for performance of the department, and prevented me from being able to scale up the team size.

As an aside to this, the organisation itself hardly changed. They still worked as before, promising customers unrealistic dates and expecting me to meet them. I failed to get through to the stakeholders that the team can only produce as much as they are willing to commit to. I failed to communicate the rules of no interference to them, and that it's "what to do, but not when to do it". The stakeholders were still pulling priorities all over the place and as both product owner and ScrumMaster I had to try and balance all of these competing issues.

In a nutshell it was a disaster.

If you are planning on implementing Scrum you need to do it treat it as an all-or-nothing change. I talked with my CEO about organisational change and the mess that I made and he told me that if I wanted things to change I had to "go to war with the company. State what I wanted to do and take no prisoners in implementing the change." The only real benefit of the anarchic state things were now in was that any change would be an improvement and likely to be received well by the organisation.

Learn from my mistakes. Scrum cannot be implemented in a piecemeal manner.

Update: Find out what things are like 9 months later

8 comments:

  1. Thanks for this Richard.
    I'm working on a project for the ScrumAlliance to collect success and failure stories, lessons learned, etc.

    May I reproduce your story on the scrumalliance site?

    -- Victor Szalvay

    ReplyDelete
  2. Sure. Anything to help others avoid the same pitfalls :-)

    ReplyDelete
  3. Well written and informative (although I have lots of other concerns about Scrum working on most projects) - a brave article and very honest.

    Marc

    ReplyDelete
  4. Thanks dude, I really took your advice in consideration, I have a meeting tomorrow to tune the shitty scrum we have at my organization, you are my evidence man

    ReplyDelete
  5. Thanks a lot. I am starting to implement SCRUM and was considering doing exactly as you had done (slowly transition into it). I have to rethink it now!

    ReplyDelete
  6. Thanks for this Richard.
    I am a Masters student doing a research on failed Agile stories, the reason for their failure and successful Agile stories, the reason for their succes.

    I have two requests to ask you -
    1. May I use your story for my research?
    2. Can you give me more details regarding the failed project?
    Like - size of the team; whether specialists were involved; Nature/Domain of the project; Quantity of code written; What steps in Scrum did you follow this project- (pair programming, collective ownership.etc); Number of iterations; Whether quality of code improved?

    I would really appreciate it if you can answer my questions..

    Thanks!!
    Paddy

    ReplyDelete
  7. I never said a software project failed. I said that the way I tried to introduce scrum to the organisation failed (at least in the first attempt). If you look at other blog posts you'll see that attempt #2 was a better experience.

    ReplyDelete
  8. I've had mostly success in implementing SCRUM, but there is a great deal of confusion out there about what SCRUM is for. SCRUM is just a process for managing the development effort. It does not handle all aspects of a project: release, deployment, testing, training, research, requirements gathering, architecture etc. To understand this; one needs to realize that the history of SCRUM comes from Software Development in Software Companies......not application development inside an IT department. There is a scaling difference here. Frequently the person leading these development efforts in a software company is a tech lead or architect. In an IT dept it's usually a PM. This nuance is frequently missed in our industry....not just in techniques, but also in tools.

    ReplyDelete