Turn Your Data Ship Around

Shaping Your Data Team, Part 1: Innovation

I use a few simple indicators to check if a data strategy is failing, but if I share them with you, it could make matters worse. Just ask Captain David Marquet.

In his bestselling 2012 book Turn the Ship Around, Marquet tells the story about how he and his crew transformed the nuclear submarine USS Santa Fe from the worst-rated, most poorly-managed ship in the nuclear fleet into the most effective, best-prepared sub in the US Navy. When Marquet first took command, the sub was failing by almost every measure. And it wasn’t because of incompetent sailors. Instead, it was a leadership model that reduced everyone to rule followers. 

Marquet realized this when he asked a crew member what he did on board. The sailor replied, “Whatever they tell me to do.” 

Marquet writes, “The overwhelming sense on the ship was that we needed to avoid problems: avoid drunken driving citations, avoid liberty incidents, avoid physical fitness failures, and avoid a reactor problem.” He realized that all their key performance indicators (KPIs) were really just management’s attempts to avoid problems, not tools to inspire real, proactive improvements. 

When you’re just trying to avoid mistakes, it’s hard to aim for excellence.

Turning Your Data Ship Around

Captain Marquet turned the ship around by turning the entire leadership model of the crew upside-down. He replaced their top-down, metrics-driven “leader-follower” approach with a “leader-leader” model focused on empowerment at every level. He created a culture where every crew member fully-owned the decisions that their own job required. 

In true military form, Marquet offers this “mechanism” to change the direction of any organization: 

- Achieve excellence, don’t just avoid errors.

I think you’ll see many similarities between leading a submarine on its mission and leading a data strategy at your company. Most data teams work like Captain Marquet’s crew before he took command. They’re just responding to tickets, checking boxes on project plans, and aimlessly applying prescribed solutions. 

I once asked the leader of a data team if I could see the road map for all his projects, and he gave me a list of help desk tickets. So I asked him how he decided which ticket to work on first. He really didn’t know, so he just responded to the requestor who complained the most. 

Like that sailor on the USS Santa Fe, he just did whatever they told him to do.

Domain Architects

A data team works a lot like a submarine crew in the sense that every member owns some technical responsibility that the fate of the entire solution depends on. It took me a long time to recognize this: my staff wasn’t just there to help me do my own job better. Rather, they were an interdependent team whose work and ideas were greater than the sum of the parts. I was a member of the team, and we all made each other better.

Marquet’s “leader-leader” model for decisions on a submarine translates directly to your data strategy. More specifically, every member of a data team should work as a “domain architect” in their role. There’s no way that one person can understand (and manage) all the details of a technical architecture, so you don’t have to pretend to be an expert on everything. Instead, you can define domains and subdomains within the architecture and allow everyone to own the solutions in their area.

The Solution They Always Knew

This approach once led to my team solving the most common failure point (and the biggest technical headache) of every data team: batch processing.

“Batch processing” is the activity of moving data out of business systems and into a data platform. It’s the most time consuming, expensive, corruptible, and failure-prone task of a data team. Everyone struggles with it. Some teams solve it by hiring lots of people (at the cheapest rate they can find) to watch these processes execute. You might think I’m making this up, but many data teams think it’s their job to just restart system processes when they fail. Other companies just ignore failures. 

A major ERP system vendor offered a solution to this problem, but they required you to buy their new, multi-million-dollar data platform. That didn’t make sense to my team, but it did inspire new ideas. They already had a deep understanding of database fundamentals, so when they researched the vendor solution, they realized that the vendor was simply using a standard database feature to move the data. They researched the licensing and found that there were no restrictions on using this same feature without buying software. 

They always knew how to solve this problem, but nobody had ever given them permission to suggest that solution.

My team didn't just find a way to make this process more efficient; they eliminated batch processing altogether. It was like replacing a diesel-powered sub with a nuclear sub. The hardest part was convincing management that a game-changing solution like this could be done fast and free. The solution completely transformed the system (and business) landscape.

When people know that you truly trust them to make the critical decisions needed to do their job, you might find that they’ve got a lot better ideas than you. 

Constant Innovation

Business teams didn’t know why things suddenly got faster and more reliable, but like the USS Santa Fe, they knew the data crew was working beneath the surface to protect them with constant innovation and readiness. A data team that works together like this constantly thinks about innovations, finding solutions to problems that the business teams don’t even know exist. I could tell you about dozens of other ideas that they developed to make work easier and open up new solutions for business problems. 

That’s what happens when a data team aims for excellence instead of avoiding failure.

#frictionlessdata #datateam #leadership

.