Never Mask Bad Data with Logic
Bradenton, Florida, October 2024
(This is the first article in my series, “The 00001010 Commandments of Data”. You can read the introduction to this series here.)
BAD DATA.
It describes so many problems people face at work, but nobody agrees on what to do about it. Executives complain about it, analysts override it in spreadsheets, programmers bury it in code, software companies monetize it, and report writers get blamed for it. Everyone complains, everyone has an opinion, but it doesn't go away.
The best way to eliminate bad data is to fix that data, but ironically, that’s usually the last thing people think of. Fixing bad data doesn’t cost anything, doesn’t require new software, and doesn’t require approval. But it does require you to let go of all your neglectful shortcuts and change the way you work. If you want to avoid cavities, you’ll need to brush your teeth every day, and if you want to avoid bad data, you’ll need a habit of fixing it at the source.
Shortcuts – like masking bad data with report logic – must stop.
Atomic Habits
In his best-selling book, Atomic Habits, James Clear argues that you don’t change outcomes by changing your goals, but rather by changing your routines. I love his title because it shows how good results usually come from the cumulative effect of many small changes that add up to a huge impact.
Clear offers four strategies to change your habits. Let’s apply those to your data.
Make it obvious. To get the data right, don’t hide it. Let it flow right up to the top-level reports that leaders see. There’s nothing better than full visibility to the problem to motivate people to make a change.
Make it attractive. You’ll need to explain the business case for fixing data. You might think that shouldn’t be necessary, but most people don’t see the big picture and can’t connect the dots between what data they’re neglecting and the problems that it causes.
Make it easy. Getting people to fix data means doing the work to define business rules. When you, the data strategist, do this right from the beginning, the people responsible for fixing the data don’t have to think about it when they make the corrections. You’ve made their job easy.
Make it satisfying. This is where leadership comes in. When you, the executive, recognize the value of these “atomic habits”, you’ll be able to give credit where credit is due. That’s what really motivates people, and that’s what communicates to your whole company that quality matters.
Plateaus and Potential
When people try to change a habit, they often get discouraged because they have such a long way to go (like training for a marathon). The same thing happens with corporate data: you have so many errors that everyone assumes that it can’t be fixed without software. James Clear calls this the “Valley of Disappointment”.
But stick with it long enough and you’ll break through what he calls the “Plateau of Latent Potential”, when suddenly bad data is the exception, not the rule. I’ve seen this happen over and over, where an entire company “gets it” and people naturally know what to do when they see bad data.
So, never mask bad data with data tools. Always fix it at the source.