How I learned to love the “Agile Industrial Complex”
Now that agile adoption has gone mainstream, there is a growing sense of unease around how larger organisations have implemented it. In particular, there is a tendency towards centralised control that can be at odds with the agile preference for individuals over process.
Some of the more influential early practitioners of agile bemoan what it has become. Ron Jeffries is particularly critical of enterprise agile “roll-outs” that impose “correct” process on teams. He refers to “Dark Scrum” as the result of impatient and poorly-supported Scrum implementations that don't allow self-organisation to emerge.
Jeffries has suggested that developers should "abandon agile" by detaching their thinking from any framework or method. Instead, they should focus on those technical practices that empower developers to overcome the impediments to building great software. This means working software, incremental releases and clean design.
Martin Fowler has suggested that many organisations use the word "agile" without adopting the values and practices it is supposed to represent. He referred to the "Agile Industrial Complex" as the collection of consultants, experts and managers who try to assert "best practice" rather than letting teams decide how best to do their work.
Problems of vague definition
Fowler's argument is that the principles matter most, yet the principles outlined in the agile manifesto are vague value statements. The ideas on the left are "favoured" but the ideas on the right are not excluded. What is the correct balance between working software and documentation? How do you weigh up interactions and process in a shop with hundreds of developers and millions of lines of code?
The obvious retort is that "the team should decide", but this can feel a little impractical. If the team chooses its own path, how can we be certain that this dovetails with the wider organisation's needs? How do you know when a team is working effectively and how do intervene when something has gone wrong?
The notion of "self-organising" teams can be taken a little too literally. An organisation also has needs that are not addressed by the more team-centric principles of the agile manifesto. It will need to achieve some economies of scale, facilitate large-scale integrations, avoid repetition, attempt to share best practise and monitor progress across dozens of different self-organising teams.
Allowing teams to be self-organising assumes they have the skills, experience and perspective to make the right choices. They also need the maturity and self-awareness to be able to when change and improvement is required (i.e. all the time). This is not always the case, hence the growth industry of “agile coaches” to mentor teams in their journey towards agile nirvana.
The need for guidance rather than control
The word "agile" has become a morally loaded term. It has inspired more than its fair share of unhelpful zealots and obstructive cant. We should not be framing the problem in terms of the struggle of humble development teams against the machinations of ruthless management. It has more to do with the need to balance the interests of the organisation with the desire to create a productive environment for development teams.
The "Agile Industrial Complex" does have an important role to play in ensuring that the needs of the organisation are catered for. It just needs to resist the urge to assert common process in favour of guiding teams towards realising approaches that suit their unique technical and commercial context.
For example, many teams are not developing the kind of small to medium-sized applications that lend themselves to agile. It is not always possible to identify discrete features that can be delivered in the context of a regular cadence. The meat and drink of larger corporate organisations is more likely to be bug fixing on a code base old enough to drink alcohol or slogging through a complex yet lucrative data integration for a corporate ERP system.
Development process can also be shaped by the wider commercial context. Iterative delivery can be a hard sell to customers who are reluctant to accept what they perceive to be greater uncertainty. Organisations with a small number of highly influential customers can be put under intolerable pressure to provide guarantees around scope and delivery timelines.
Agile purists often interpret the agile preference for "customer collaboration" as never having to agree a deadline. A more liberal reading of the agile manifesto allows for flexibility so teams can accommodate the wider commercial reality. This can be difficult for inward-facing, technically-orientated development teams to grasp without some guidance.
Attempts at providing guidance are often interpreted as the actions of a management class who are unwilling and unable to relinquish control. This rhetoric can be unhelpful. Sure, there are occasional control freaks out there, but more often than not, this is an attempt to bring a wider organisational perspective to bear on insular development teams.
Improving the lot of developers
Making it easier for developers to produce working software is the central concern of agile. Perhaps what's missing from large-scale agile implementations is a focus on the developer experience.
Developers have come to associate agile with excessive bureaucracy and time wasted in too many meetings. This was never the intention of agile, which always sought to remove the impediments to doing great work.
I have seen teams energised by agile processes where they seize control of their work and find solutions for long-standing problems that always eluded their managers. It really does work on a team level, though does not offer any solutions to the problems of organising development at scale.
This is where the "Agile Industrial Complex" comes in. It is trying to solve some very important problems, it's just that these don't have much to do with the team-centric concerns of agile.