Audit trails, or activity logs, are the ever-growing shadow of data. Every time you click ‘submit’, snapshots are taken and pushed down into a virtual underworld. As time passes, it gets deeper and darker: sometimes it can turn into a black hole, suddenly sucking out your system resources.
It does not have to be like this of course: a distributed architecture will allow you to separate the dynamic, latest state of operational data from the shadows of past activity. The main hurdle, especially for smaller applications, is that it could turn out to be costly or overly complex. At Krescendo, we decided to go for a commoditised services approach: multiple applications benefiting from shared facilities such as reporting, file management, messaging and audit trail.
A centralised audit trail offers paths towards both lightness and light. Lightness because moving the audit trail away from the main database means getting rid of a constantly accumulating burden. It is also a path towards the light because using a dedicated infrastructure and storage technologies optimised for this kind of data – such as NoSQL databases – makes it possible to carry out analytics that would have been previously too onerous. In short, ‘healthier’, leaner applications and new data mining opportunities.
If you look at large consumer internet platforms, the role that historical user activity plays is instantly obvious from the way the system knows you, guiding you in a personalised way through your navigation experience. Corporate systems should be no different, and this means treating the audit trail not as a ‘dumping ground’ to simply comply with regulatory requirements, but as an application in itself, providing real-time intelligence to the systems that feed it.
As ever, there are challenges – especially with organisation and people: dependency on commoditised services means that the teams that are responsible for them must deliver service levels that match the requirements of the most sensitive application supported; also, being able to extract value from previously ‘dark’ data comes with a number of questions in terms of ‘how best to’, and ‘at what incremental cost’, requiring new skills or altogether new professional figures.
Escape from the shadows indeed, but watch out for the blinding light.