Agile velocity is not a measure of productivity
Agile development does not necessarily lend itself to the kind of dashboards and reports beloved by managers. The few metrics it produces make little sense outside of the context of team planning. Given this shortage of usable information, it can be tempting to re-purpose velocity as a measure of productivity. After all, it’s a measure of team capacity so it follows that changes over time could be used to indicate overall shifts in productivity?
The problem with this approach is that velocity is a planning tool rather than a specific measurement. It's an estimate of relative capacity that tends to change over time and these changes don’t necessarily indicate a shift in productivity. It’s also an arbitrary measure that varies wildly between teams. There’s no credible means of translating it into a normalised figure that can be used for meaningful comparison.
You get what you measure
Given that velocity is such an arbitrary measure it is easy to game. By equating velocity with productivity you create a “perverse incentive” to optimise velocity at the expense of developing quality software. Consciously or not, teams will start trying to demonstrate an increase in productivity by massaging velocity upwards. Worse still, they may start cutting corners to deliver things with fewer story points. This can lead to a build-up of technical debt that will undermine future productivity.
You see similar behaviour if you distribute burn-down charts to senior management. Every sprint, the progress line magically starts to intersect with the delivery target. The shape of the line can vary wildly, but somehow there’s a speeding up or slowing down in the second half of the sprint to meet the expectation.
Velocity is a measure of the amount of work that a team can do. This is not the same as measuring the value or impact of this work. Velocity may actually be relatively stable in a successful and well-established team as the amount of raw effort available for each sprint remains constant. In this case, putting artificial upwards pressure on velocity will only serve to distort estimates.
Can you really measure software productivity?
Productivity is determined by looking at the inputs and outputs of an activity. It’s easy enough to measure the inputs for software development, but it’s actually very difficult to measure the outputs in any consistent way.
A raw, quantitative measure such as numbers of lines of code does not provide any reasonable guide. It is too dependent on wildly variable factors such as coding style, development language and implementation approach. It can also be counter-intuitive as well-written code that has taken time in crafting often requires fewer lines of code.
Systems based on function points or relative complexity can fare a little better, but they are still very much correlated to the code that is being written rather than the value that is being delivered. They can also add considerable overhead to the development process in terms of gathering and interpreting data.
Are business outcomes enough?
If you derive a measure of productivity from velocity then you will see a statistical improvement. This is not the same as successful development. Ultimately, agile process should establish a continuous feedback loop between development teams and the commercial context. A more meaningful measure of success should take this into account and focus on real-world benefits rather than abstract measures or normalised indicators.
The problem is that these measures tend to vary wildly between teams making any kind of cross-team comparison all but impossible. There will never be a consistent real-world measure as web teams might focus on increasing visitors and conversions, while infrastructure teams might have more technical targets around throughput and legacy.
A leap of faith is required here as team productivity cannot be meaningfully reduced to a normalised score. Teams should be collecting their own data and measuring their own productivity with a view to managing their own continuous improvement. A focus on concrete benefits is more likely to resonate with teams and provide something they can collectively work towards. It will also provide a more consistent measure over time as real-world benefits are less prone to variations caused by team composition or development approach.
Metrics are not a bad thing, but the wrong ones are
Metrics are not just a management indulgence. They help you to identify problems, decide on corrective action or figure out how to evolve in the future. However, it is difficult to arrive at a productivity measure that can be reasonably applied across the board for software development. By satisfying the on-going management hunger for reports and dashboards you risk creating meaningless overhead and distorting development.