When should you write your own message endpoint library?
Enterprise messaging patterns are complex beasts that often warrant a common implementation across your endpoints. This will assert consistency and prevent developers from wasting time on generic concerns around message handling and formatting.
In the .Net ecosystem this has given rise to a selection of libraries that abstract away the underlying transport, such as nServiceBus, MassTransit and Rebus. These libraries all serve pretty much the same purpose, i.e. they take care of all the nuance of message sending and handling so developers can focus their efforts elsewhere.
Given that these libraries tend to enforce an opinionated set of conventions onto the way you design and implement messaging, it can be tempting to build your own. After all, it'll give you direct control over your integrations and allow you to implement something that directly fits the unique demands of your environment.
There is a very strong argument that you… just… shouldn't… do… this…
The health warning
The problem is that implementing basic messaging patterns is much harder than it looks. There are a bunch of concerns to figure out, including routing, serialization, logging, transactions, retries, exceptions and scheduling. That's just the basics you need to get pub-sub messaging going.
Getting something genuinely battle-tested up and running is quite an undertaking. The kinds of bugs you'll get will be extremely difficult to spot. They will often be nuanced, involving unpredictable asynchronous behaviour, and may only be visible at scale in your production environment. You may grow old and die long before your new library is fully hardened.
Asserting a shared library between all your services can be quite a burden. Who would be responsible for maintaining it? What tends to happen is that the original authors of these libraries move on, leaving a legacy of impenetrable code behind them. It's hard enough getting your service boundaries and event design right without adding the extra overhead of plumbing.
Custom messaging implementations can also be very limiting. If you want to get into more sophisticated patterns such as sagas, you'll find yourself handicapped by an implementation that won't play along without a huge investment of work.
You shouldn't be writing a messaging library simply for NIH reasons, but can it ever make sense to roll your own?
Well, occasionally, yes…
Technology stack
Messaging endpoint libraries tend to assume a homogenous ecosystem. All the .Net options are predicated on running a recent Microsoft development stack. They do not address the problem of event-based messaging across diverse technical stacks.
Other eco-systems have similar libraries that abstract away the underlying messaging transport. For example, Java has Spring Cloud Stream while Python has Celery. None of them provide any kind of cross-platform support. If you want to get mixed ecosystems to participate in messaging patterns, you'll need to roll an implementation of your chosen library's message format.
This is hard as there is no consistency in this area. Rebus writes more than a dozen special headers to handle semantics such as correlation identifiers, timestamps, routing slips and content encoding. nServiceBus has its own set of headers, while MassTransit seems to push a lot of this stuff into the message body.
Larger ecosystems tend to be messy affairs, often growing through a combination of acquisition and haphazard decision-making. You could find yourself trying to connect a legacy ASP.Net website with Java Spring applications or containerised microservices written in Node.js. In this case, a common technical implementation won't be available, and you will need to focus instead on asserting a common messaging format.
Getting locked in
Every technology decision involves some element of lock-in. At some point you need to hold your nose and get some work done. However, messaging is something that needs particular care.
You are talking about one of the most important parts of your infrastructure, i.e. the kit that regulates service collaboration. Do you really want to be tied into an opinionated set of patterns and abstractions that you don't control? Your choice of messaging library can be a strategic decision that you'll be bound by for some time.
When I say that messaging is a common concern, it isn't really a generic one. It's not like using logging libraries that are effectively interchangeable and can be configured to push out pretty much any output. Given the lack of any common formats or standards, if you choose one of these abstractions then you're with it for life.
This can lead to some uncomfortable restrictions. All these libraries were very sluggish about making the jump from the .Net Framework to .Net Core. Mass Transit seems to have a decent community around it now, but it did have something of a lost year in 2013-2014. Some early adopters of nServiceBus were a little disconcerted with it moved from open source to a commercially licensed model.
Bloat
You're not just taking on a simple abstraction around pub\sub. There is a tendency for these messaging libraries to become bloated with features.
What if you just want a simple wrapper around your chosen pub\sub transport? This doesn't necessarily mean you should adopt sagas or accept pre-canned solutions for task scheduling, visual modelling and monitoring?
There is a trade-off around productivity and control going on here. The existing libraries may be gloriously battle hardened, but rolling your own simple implementation isn't as hard as you might think. The trick is to keep the scope small and controlled.
More modern transports such as Azure Service Bus and RabbitMQ do a lot of the hard work for you. Features such as transactional semantics and pub\sub distribution will already be available in the AMQP-based APIs. There is a case for abstracting away the underlying transport, but this is more for test isolation than flexibility of implementation. In the real world you are about as likely to change messaging transport as you are to change your relational database.
It's worth pointing out that nServiceBus and Mass Transit both started life in the days before AMQP to build integration patterns on top of MSMQ. They have expanded out to provide so much more than their original remit, but this has given rise to some very weighty frameworks. You don't necessarily need this much scaffolding on top of modern messaging transports.
When does it make sense to roll your own?
Rarely, to be honest.
Most of the arguments in favour of rolling your own library are comprehensively outweighed by the overheads it incurs. Worse still, there's the hell of trying to maintain a library created by developers who have long since left the building. You really shouldn't do this to your fellow developer.
The arguments shift a little at scale, particularly if you are trying to integrate a large estate based on different technology stacks. This generally means a mix of non-Microsoft or very old Microsoft technology. A messaging endpoint library is not going to help you here - you need a common format.
Messaging libraries feel like application-level solutions. The nServiceBus licensing costs certainly suggest something that is geared around a distributed system rather than a distributed enterprise. In a larger development shop it makes more sense to agree the “rules of the road” in terms of protocols and formats rather than trying to assert common technical implementations.