Development Diary #6

IT professionals often have an idealised view of how business processes and systems should work. This is apparent in the typical approach to data quality.

To make sure the data in the system is right, IT professionals add controls to prevent incorrect data being added. If a phone number is expected then the system will only allow you to save information that contains something that looks like a phone number.

Theoretically this should work perfectly. But the real world is messy. When someone is faced with an exception they can’t enter “This person has no phone” or “they have ordered a phone but don’t know the new number yet”. Instead they type in some fake data like 123-456-7890 – so they can proceed.

Forcing people to enter fake data is particularly unhelpful as it looks correct to the system. It could be a one-off exception. It could be a frequent issue. It could be just a question of timing where the information will be known but not by the person entering the initial data. e.g. there is a new project but we don’t know yet who the project manager will be.

With LiveDataset we encourage a permissive approach to data validation. There will be myriad reasons why the data entered is not correct but we should allow people to continue, highlighting the data with a warning so that the data can easily be corrected later.

Latest Release

Multi-Column Uniqueness

Checking each value in a column is valid is sometimes insufficient. There are often cases where you need to check the combination of multiple combinations is unique. In this example for sales forecasts there should not be more than one forecast row for any combination of country and product.

Two forecasts entered for the same country & product combination.

Restore Values

The change log not only shows who made changes and when, but also allows you to undo changes by reverting to a previous value.

Warnings Counter

Data that falls outside expected values is highlighted with a warning. The warnings counter shows at-a-glance if there are any potential data errors.

Restore previous values using the change log

Warnings counter

Warnings View

A view can be defined to show any data with warnings, making it easier for an administrator to resolve.