Cache Synchronization With Jabber

Yesterday afternoon Suneth and I took on a research project to see how feasible it was to keep server caches up to date by using XMPP to notify the other servers in the cluster of a change. Imagine a web server with some latency between it and the resources it’s serving (eg: it’s using S3), to speed up performance you’d want to cache the recently used or most commonly used resources locally on the server, but if you need to scale up to a cluster of servers and the resources are being changed, that cache becomes a problem.

The simplest solution to this is to set limits on the length of time that resources can be cached but I’d prefer to avoid latency in updates as much as possible.

The other alternative is to have some kind of notification between the server cluster so that all servers rapidly find out when a change has been made – much like database replication. This is what Suneth and I wanted to play around with and we thought XMPP would be a quick way to get such a notification system going.

It’s worth noting neither of us really knew anything about XMPP other than that it was the protocol underlying Jabber instant messaging and it was a generic XML messaging system. We figured we could use the generic nature of it to send whatever messages we needed and just take advantage of the existing servers to do the marshalling. Fortunately, our lack of knowledge led us right into a much simpler solution – just use Jabber.

We grabbed the Smack API and started playing with it and quickly discovered that sending and receiving messages was ridiculously easy. It turns out that the absolute simplest way you can minimize stale data in your caches is to simply have all the servers join a preconfigured chat room. Whenever they save a change to a resource they send a message to the room with the unique ID of that resource and whenever they receive a message from the room they assume it’s a unique ID and remove any cached versions of that resource.

It also makes testing easy – you simply open a jabber client and join the chat room yourself.