Over the last 6 months I've been trying to bring about organizational change in order to improve the product development process, change how product development interacts with the rest of the business and to improve the productivity of my department.

As you would expect there were two central ideas to this strategy. Firstly I needed to change engineering practices and secondly I needed to change the management of those practices. It was time to kiss goodbye the traditional waterfall/gantt chart based project management time to move to agile methods.

I started by migrating the team to agile development methods (based on the fundamentals of extreme programming) whilst still retaining the documentation needed to support our ISO accreditation. This has been quite successful. The team understands the need to improve, and the actual methods of producing software are fairly easy to alter since it is an internal process (ie internal to my department). The only areas I still need them to improve relate to refactoring and code reviews. This will come with time and I am quite happy with things being at the stage they are at this point in time.

I've also been changing the management of the development process to Scrum. I tried to do this gradually for a number of reasons, the main one being that I was new to the organisation and didn't want to upset the apple cart or be seen as the newbie trying to change something when they didn't understand the corporate culture.

Well, suffice to say, my attempts at the gradual adoption of scrum have been a complete disaster. Why? Because I only implemented portions of Scrum under the thinking that I need to give people time to get used to the new methods before implementing more change, and that too much change too quickly would upset the organisation. For example, I instituted the daily meetings to foster open communications between the team and I set up the prioritisation of tasks (the backlog) so that the other areas of the business were aware of what was happening.
I moved to regular releases of the system - down to a 3 week period instead of 12 months - and because there was no clear product owner I fulfilled that role as well as taking on the role of ScrumMaster.

Instead of instituting a change for the better I managed to institute anarchy. Filling both roles of ProductOwner and ScrumMaster led to conflicts of interest over what was best for the company over what was best for the team - I broke the rules of Scrum myself by allowing constant chopping and changing of direction based on need and failed to communicate reasons for change with the rest of the business. Obviously these failures are my own fault and lack of time and business growth compounded the errors I was making.

Unfortunately by not enforcing the rules of scrum, I allowed ad-hoc requests to be actioned, allowed staff to be pulled off for other non development work and as a result failed to get the team to own the process and to own the delivery of results.
I was doing the classic "tell you what to do and tell you when to do it" instead of letting the team self manage.
Effectively the daily meeting was a "report activity to Richard" session and was not acting as a team synchronisation meeting. Delivery commitments were estimates only with no repercussion for failure. The business saw the product backlog as just another way of tracking outstanding work requests but having no other value to the business.
The worst result was that without the team owning the product or the process they felt no responsibility for delivery. "Richard will tell me what to do next" was the common thought, and that was the last thing I wanted. It meant all responsibility was on my shoulders, made me the bottleneck for performance of the department, and prevented me from being able to scale up the team size.

As an aside to this, the organisation itself hardly changed. They still worked as before, promising customers unrealistic dates and expecting me to meet them. I failed to get through to the stakeholders that the team can only produce as much as they are willing to commit to. I failed to communicate the rules of no interference to them, and that it's "what to do, but not when to do it". The stakeholders were still pulling priorities all over the place and as both product owner and ScrumMaster I had to try and balance all of these competing issues.

In a nutshell it was a disaster.

If you are planning on implementing Scrum you need to do it treat it as an all-or-nothing change. I talked with my CEO about organisational change and the mess that I made and he told me that if I wanted things to change I had to "go to war with the company. State what I wanted to do and take no prisoners in implementing the change." The only real benefit of the anarchic state things were now in was that any change would be an improvement and likely to be received well by the organisation.

Learn from my mistakes. Scrum cannot be implemented in a piecemeal manner.

Update: Find out what things are like 9 months later