Try Centralizing Your Data Team
CIO Journal
If you think your company’s data problems are complicated, you haven’t seen enough.
A large biotech company I worked with had 12 ERP systems, 100-plus legal entities in dozens of countries, and three divisions managing data independently. As they acquired more companies and their data platforms multiplied, the company’s team of analysts also grew larger and larger.
When we discussed their data challenges, a senior executive said to me, “Oh, now we’ve come to the age-old question: could we ever centralize our data?”
This exec saw the real consequence of decentralized data solutions: technical debt. Yet he wondered, could getting out of debt motivate his company to centralize its data teams?
A Binary Bargain
Does this conversation sound familiar?
IT leader: “We can reduce costs by consolidating data work into a single central team.”
Business leader: “My group has special needs. We need control of the data.”
These competing concerns cancel each other out, and nothing happens. What if I told you that, even if you solved both problems, you’d still miss the biggest value of centralizing your data team?
When every division operates independently, it makes your overall enterprise data fragile. My client’s company had created separate customer masters, different methods for determining tax codes, and competing ways of calculating demand.
They might have loved their decentralized data solution, but it didn’t love them back.
Getting Out of Debt
The very structure of your org chart has the greatest influence - good or bad - on the amount of technical debt your data teams create. Consider how the relationships between the people on data teams impact the solutions they create:
Centralization Aligns Incentives. Most technical debt gets created in response to a help ticket. Centralizing your data team gives you better options - it allows you to separate the analysis of business problems from development work:
Software developers solve problems by writing code. They’re incentivized to say “yes” to a request to change the system. If they say “no,” they worry people might think they’re just being obstinate.
Business analysts solve problems by showing people how to change their work procedures to align with the system; they’re incentivized to say “no” to a system modification and “yes” to solving problems with a business process change.
With a centralized data team, you can separate these functions into different roles within the same organization.
Centralization Increases Visibility. A central team sees the big picture, the overall flow of data across your company. Your best engineers solve the biggest problems.
Data architects on a centralized team can focus on changes that benefit everyone, like speeding up data movement across all systems. More importantly, visibility into different business processes allows a central data team to compare them, identify differences, and propose standardizing them before writing code that locks them in.
You’re Not Alone
Even some of the most advanced companies experience the downsides of decentralized data.
When I presented my Frictionless Data solution to a former Amazon executive, the first thing he asked me was, “Should business data be centralized or decentralized?” Amazon famously follows a decentralized approach to analytics, something I knew from a close friend who worked there as a business analyst. That friend also told me about the chaotic, frustrating churn he experienced while hunting for data at Amazon.
Who is to blame for the technical debt that results? Nobody, really. Thousands of independent decisions create it. It’s a problem that only a leader can solve.
If that’s you, then ask yourself: Do you want to avoid piling up more technical debt? Try centralizing your data teams.
To remind you of this week’s data concept, enjoy Der Kommissar by After The Fire, from the Frictionless Data Spotify playlist.
No, centralise the core data governance team, those who manage a common semantic understanding of your terms, models and usage across your domains, ideally a core ontology team that owns the conceptual ontology, and distribute data teams close to each domain that use the conceptual model to harmonise towards the conceptual model.
You get speed, agility, ownership, democratisation in the data teams, and a common semantic, controlled, small, conceptual ontology to manage data governance, including tech debt.
Centralising all data teams create a bottleneck for all the data domains.
Centralizing everything comes with it's own set of problems. I would argue that those teams aren't really incentivized to say "Yes". What I have seen is teams that lose the muscle memory of the business and friction builds up in the middle where people try to translate their needs to a data team that has been optimized on technical skills. Data people work best when they are close to their business partners.
In a large Enterprise there is no perfect answer, but I think hub and spoke style federation tends to give the best outcomes with the least friction.