Papers
Event Sourcing
Martin Fowler · martinfowler.com (article), 2005
The problem: knowing where we are doesn't tell you how we got here
Fowler starts from a simple observation. Querying an application's state answers well the question of how the world is right now, but there are situations in which we also want to know how we got there. Event Sourcing's proposal is to ensure that every change to the application's state is recorded as a sequence of events. With that record, besides querying the events, it is possible to reconstruct past states and to use the log as a basis for automatically adjusting the state when retroactive changes arise. The article's example is a ship tracking system. In the simple model, a service receives port arrival and departure calls and directly updates the ship's location, so that only the final state is stored. With Event Sourcing, the service creates an event object that describes the change and processes it to update the ship. Seen only as processing, this looks like a useless layer of indirection; the difference shows up in what persists. Two things are now stored: the application state and the event log. Fowler admits that seeing where the ship has been is a small gain, because a port history on the object itself or an ordinary log file would also provide that. The decisive point is a different one: all changes to the domain objects are initiated by the events, and it is this guarantee that makes it possible to build features on top of the log.
What the event log enables: rebuild, temporal query, and replay
Since every change originates from an event, the article lists three facilities built on top of the log. The first is complete rebuild: the application state can be discarded entirely and rebuilt by re-running the events from the log on an empty application. The second is temporal query: to know the state at any point in time, you conceptually start from a blank state and re-run the events up to a certain instant or a certain event. Fowler suggests going further and considering multiple timelines, analogous to branches in a version control system. The third is event replay. If we discover that a past event was incorrect, we can compute the consequences by reversing that event and the later ones and then reapplying the corrected event and the following ones; the alternative is to discard the state and replay the whole log with the correct event in place. The same technique works for events that arrived out of order, a common problem in systems that communicate through asynchronous messages. Fowler cites version control as a current example: it uses temporal queries often and, in the case of Subversion, complete rebuild when doing dump and restore between repositories. He says he knows of none that does replay, since they are not interested in it. Enterprise applications with Event Sourcing, he says, are rarer, although he has seen a few.
Snapshots and where to keep state
The simplest way to use Event Sourcing is to compute any requested state by starting from an empty state and applying the events. Fowler notes that this is slow, especially when there are many events. Since, in many applications, the most common request is for recent states, a faster alternative is to store the current state and build the special capabilities of Event Sourcing on top of it. Because the state is fully derivable from the log, it can be cached anywhere, in memory or on disk. The article describes a system used during the workday that starts the morning from a snapshot of the previous night and keeps the current state in memory. If it crashes, it replays the events from the nightly storage. At the end of the day a new snapshot can be generated, and new snapshots can be taken at any time, in parallel, without stopping the running application. There is also a choice about the official system of record. It can be the event log or the current state. In the second case, with the state in a database, the events serve only for auditing and special processing. In the first, the log is the official record and the databases can be built from it whenever needed.
The costs: rollback, external systems and code changes
A large part of the article deals with complications. First, reversing events: it is simpler when the event is expressed as a difference (add 10 to the account) than as a final value (set the account to 110), because in the latter case there is no information to recover the previous value. If the event is not a difference, it must store whatever is needed to reverse it, such as previous values. Fowler points out that all of this can also be done by going back to a past snapshot and replaying the events; reversal only matters for efficiency, when undoing a few events costs less than reapplying many. Second, external systems: since replays happen freely, update messages sent to external systems, which cannot tell real processing from replay, would go wrong. The solution is to encapsulate those systems in a Gateway that knows whether the processor is in replay and, in rebuilds and temporal queries, it is usually enough to disable it, without the domain logic noticing. External queries also affect the result: when replaying an event from December 5, we want the exchange rate of that day, not the current one. Options are to query historical data or to make the gateway remember the responses. Third, code changes: new features and bug fixes in what is internal can be applied by reprocessing the events (with external gateways, you need to track the difference between what happened with the bug and without it), but rules that change over time must go into the domain model itself. Fowler warns that bitemporal behavior can get very messy.
When to use: auditing, debugging, and architecture with many readers
Fowler acknowledges that packaging every change as an event is an interface style that not everyone is comfortable with, so the technique is not a natural choice and is only worth it if there is a payoff. One obvious payoff is the audit log, also useful for support, which can track what a user did; the author notes, however, that ordinary logging mechanisms would also solve this. Another use is debugging: replaying the real events in a test environment to see exactly what happened, with the ability to stop, rewind and replay, as in a debugger, including for parallel testing before putting an update into production. The article states that Event Sourcing is the foundation of Parallel Models and Retroactive Events, patterns that require it, and that it is very hard to retrofit onto a system that was not born that way; so if those patterns are likely, it is worth adopting it from the start. There is also an architectural possibility: a system with many readers and few writers could be a cluster with in-memory databases updated by an event stream, while updates go to a master system that applies them and broadcasts the resulting events. The trade-off is that readers may lag behind the master. Fowler also warns that the text is draft material from the mid-2000s, with no later corrections.
See the ideas at work
Practice the designs that use this paper in the System Design school.