Given that most development leverages external frameworks and libraries, engineering organisations often try to build their own shared libraries in the hope of achieving some economies of scale. The expectation is that these libraries will save time and improve the quality of solutions by preventing teams from re-inventing the wheel.

The problem is that most initiatives are based on the fallacy that sharing code makes development more efficient. Given that sharing data or resources is often regarded as a bottleneck in system design, why should we be so keen to share code?

Shared libraries tend to undermine engineering productivity by creating coupling between teams. Applications can become indirectly tied into specific versions of dependencies that cause all sorts of compatibility headaches. Breaking change or regression can cause havoc without careful and conscientious management.

There is a trade-off between coupling and code reuse. Manual code duplication can be a decoupling mechanism that allows applications to evolve at a different rate. It also helps to promote code stability as applications will not constantly need to absorb updates caused by shared dependencies.

When libraries add value… and when they don't…

That's not to say that shared code is always a bad thing and there are situations where internal shared libraries can add value. For example, front-end developers often use shared libraries to implement design systems that assert a common look-and-feel across applications.

Many shared libraries have a less distinct purpose, especially those words like “common”, “utility”, or “helper” in the project title. These generic libraries have vague responsibilities where it's hard to maintain any clear boundaries for them over time. Given that terms like “common” are so subjective, these libraries tend to become dumping grounds for anything that engineers happen to think might become reusable at some point.

Generic libraries typically re-implement language features or address solved problems, such as authentication, serialisation, data access, and encryption. Not only do they create an unnecessary maintenance burden, but they add to the learning curve for newcomers by peppering code with unfamiliar function calls.

Other internal libraries are little more than anaemic wrappers around third party dependencies. These libraries are often justified in terms of making it easier to change the underlying dependency, even though this will never happen in practice. Most wrappers end up obscuring the interface and limiting the underlying functionality without adding anything worthwhile.

Another common form of internal library is the “best practice” implementation. This is often an opinionated attempt by one set of engineers to assert implementations on other engineering teams. The implied lack of trust is not a sign of a healthy engineering culture.

The more intrusive and opinionated a library is, the less useful it becomes. Libraries that define “blessed” patterns can serve to restrain engineers and create an unhealthy level of coupling between implementations. For the most part, “best practice” can be encouraged through examples rather than attempts to abstract them away into a library.

There are also those libraries that seek to share domain data and logic between services. These blur the boundaries between applications and create coupling through shared schemas. It eventually becomes impossible to make changes to these shared domain libraries without having unpredictable effects on the various applications that depend on them. It's normally better to duplicate shared domain data, implement it as a separate service, or rethink your service boundaries all together.

The shared library lifecycle

The biggest problem with shared libraries is that they are difficult to write and even harder to maintain. Shared libraries often suffer from an absence of tangible requirements which makes them more difficult to design and test.

Developing for reuse also requires specific skills and knowledge of design patterns that is not commonly found in engineering teams. It takes experience to get the level of abstraction right without making components too specific for general use or too generalised to add any real value.

Like any other code base, shared libraries tend to suffer from entropy over time and require consistent effort to maintain. Most organisations have some unloved shared libraries lurking in their code repositories. Just keeping the dependences up to date can be enough of a challenge, let alone evolving the functionality in line with the shifting systems landscape.

It can be difficult to free up sufficient engineering resource to maintain shared libraries, especially where engineering teams are organised around systems or commercial verticals. This causes shared libraries to get forgotten about and fall into disrepair once the initial enthusiasm for them fades away.

Orphaned libraries are a common outcome where engineers have long since stopped maintaining them. Others become reliant on a small body of enthusiast engineers who can skew development towards a very narrow set of interests.

An inner source approach could help to establish a community-driven approach to shared code. However, this is only likely to be successful where you have a large pool of skilled engineers that have experience of operating in open source communities. Even then, this will not spare you from having to explicitly devote engineering effort to designing, building, and maintaining shared libraries.

What makes for a “good” shared library?

No matter how you organise development, shared libraries will always need clear and consistent leadership. You need somebody to ensure that the library remains focused on its main objectives and that the code base remains in rude health.

There may be the need for governance to help determine what should be considered “shared”. This can take many forms, but a collaborative approach is more likely to gain lasting traction over “top down” governance and mandated libraries. One indicator of a successful shared library is the level of spontaneous adoption among engineering teams – i.e. is it really making their lives easier?

Successful shared libraries tend to have certain characteristics. They tend to be small and focused on a very narrow range of concerns, making it easier to maintain focus and defend boundaries over the time. They will have well managed code infrastructure, including a test suite that helps to guard against regression and unexpected breaking change. They will have efficient build and release automation that supports the easy integration of new code. There should also be clear and consistent documentation that makes the library easy to adopt.

Above all, there needs to be sufficient commitment around the organisation to ensure that enough time can be devoted to their ongoing development. If this isn't in place, then internal code libraries have little chance of adding any real value.

Filed under Agile, Development process, Strategy, Rants and Favourite posts.