Listen:
Check out all episodes on the My Favorite Mistake main page.
My guest for Episode #195 of the My Favorite Mistake podcast is Ward Vuillemot. Ward is a seasoned C-suite executive with over six years of experience leading fully remote teams while building technology organizations from the ground up for companies with 150 to 650 employees in size and $50M to $125M in revenue across the Americas and Europe.
He is currently Chief Product Officer and CTO at RealSelf. He is a technical advisor with his own company, where he advises startup founders and CEOs on technical roadmap and technology organization along with lean approaches to finding market signals quickly.
I invited Ward because of this Forbes article about celebrating errors.
In this episode, Ward tells his favorite mistake story about launching “Amazon Tote” and why there was “too little friction” in user experience. What did he learn about understanding the customer experience? In a separate story, what was Ward's epiphany about seeing an ant on a bus?
Questions and Topics:
- Innovation is doing something others haven’t done before
- Tell us about the Celebration of Error (CoE) concept – and practice…
- Chicken and egg between psychological safety and CoE?
- How much Psychological Safety is necessary and how does CoE build more PS?
- From Correction of Error (Amazon) to Celebration of Error?
- Are all errors created equally in terms of what to celebrate?
- Discovering mistakes that had been there for years
- As a person who is “high-functioning autistic” – is it ever a mistake to disclose something that personal?
- From mindset to document?
- IMPACT of the error on business – send to whole company?… why it matters, not why it happened
- RESOLUTIONS — short-term and long-term (countermeasures) – fire out, then prevention
- ROOT CAUSE – “show your work”
- When to use a CoE?
- Mark's firefighting A3 cartoon
- People “NEED” to make mistakes to hit ever-greater goals?
- Taking an impersonal, non-blaming approach — easier said than done? Fighting the instinct to blame?
Find Ward on Social Media:
Watch the Full Episode:
Quotes
!["[I tell my teams] I don't mind you making a mistake. I really don't like it when someone else from the part of the organization, or worse our customer, tells us about the mistake. If you tell me about it, that's OK."
- ward vuillmot](https://www.markgraban.com/wp-content/uploads/2023/01/Ward-Vuillemot-My-Favorite-Mistake-Quotes-3.jpg)
Subscribe, Follow, Support, Rate, and Review!
Please follow, rate, and review via Apple Podcasts or Podchaser or your favorite app — that helps others find this content and you'll be sure to get future episodes as they are released weekly. You can also become a financial supporter of the show through Anchor.fm.
You can now sign up to get new episodes via email, to make sure you don't miss an episode.
This podcast is part of the Lean Communicators network.
Other Ways to Subscribe or Follow — Apps & Email
Ward Vuillemot On The Power Of Celebrating Errors And Understanding Customer Behavior For Business Success
In this episode, our guest is Ward Vuillemot. He's a seasoned C-Suite executive with several years of experience leading fully remote teams while building tech organizations from the ground up for companies 150 to 650 employees in size and from revenue ranging from $50 million to $125 million across the Americas and Europe.
He's Chief Product Officer and CTO at RealSelf and is a technical advisor with his company. His website is WardVuillemot.com. Ward advises startup founders and CEOs on technical roadmaps and technology organizations along with using what we sometimes refer to in this show as Lean approaches to finding market signals quickly. Ward, welcome to the show. How are you?
I’m excellent. Thank you for having me, Mark.
Ward is joining us from Washington. We'll talk about this after his favorite mistake story. There was an article written in Forbes about Ward about what he calls CoE or Celebration of Errors. We're in the right place to have a discussion about that. Before we talk about that concept and the article, I won't let you off the hook. Looking back at the different things that you've done, what would you say is your favorite mistake?
In your career, mistakes are fundamentally the hallmark of growth in so many different ways and even where innovation comes from, which is something I've written about. With ignorance and mistakes, that’s how we learn. The earliest one where I thought I knew the answer and I learned something along the way was a small startup program that we were doing out at Amazon. It was a sister program to Amazon Fresh. This was before Amazon Fresh went national. We were still in the Seattle area. I was a part of that.
I was asked to go over and be part of the launch team for something called Amazon Tote. Briefly, Amazon Tote was the idea that you could get free deliveries to your home. Order any item on Amazon and we would deliver it like your best friend. Imagine if you called me up and said, “Could you go down to the mall and pick me up an iPod?” I put it in a little tote bag and I drop it off at your door. That's how you got it. To get rid of all the boxes, you would get these totes so we go by Amazon Tote.
I remember in the very beginning, we were working with Doug Herrington. Doug Herrington was into this working backward from the customer mindset. It’s very much of that Lean mindset. It's not a mistake but this is to give you some idea of how we were thinking about it and I'll get into the mistake. I hacked his browser with his permission, to be clear. He would go to the Amazon website and there was everything. There were Cheez Whiz or Cheez-Its. He would order it and go through what we thought was going to be the launch experience. I then would get an email based on that little hacked browser experience and I would put the Cheez-Its in the tote bag. I'd walk down the hallway to his office and would deliver the Cheez-Its.
[bctt tweet=”Mistakes are fundamentally the hallmark of growth in so many different ways, and even where innovation comes from.” via=”no”]
We learned some interesting things about having too little friction in the user experience, which was fascinating to us. We didn't have an interstitial at that point. It was too frictionless. A good, interesting learning from Amazon Tote was the lack of friction. You need some amount of friction. We always talk about reducing friction for consumers. There's such a thing as too little friction. That was an important lesson. There were lots of lessons there. We didn't even write code. We were learning this by doing some low-level, low-fidelity mockups.
The one that I had done before the launch was this idea of multi-quantity. You could get multiple items. We had a cart like everyone else but you could only order one. Let's go back to that original example. Let’s say you wanted to buy an iPod for yourself and someone else. The way we envisioned it was that you would add the iPod to your tote bag and then go back to the detail page and add it again. This was to do with some ways that we were hacking Amazon's inventory system. We knew we could only guarantee a quantity of one at any one time when we put it in. We couldn't do a promissory greater than that.
I did a bunch of research on consumer experience. We knew that, generally, consumers might buy multiple items but not multiple quantities. You could have an iPod and a book but very improbable, you would've 2 iPods and 1 book. We knew that by not having that multi-quantity, at least from a consumer behavior standpoint, we weren't missing out on an opportunity. We felt confident going to launch with this thing without those multi-quantity dropdowns.
We launched that and every week, we meet with our customer support folks. Lo and behold, the number one complaint, as you might imagine because of the story, was, “Where the F is your multi-quantity? Aren’t you a bunch of effing idiots?” They were not plated for us. They were like, “Aren’t you Amazon? How could you not have this thing?” It went every week. I kept on going to customer support. I was like, “I got the data. It's not going to change behavior.” We knew this was a fringe but it was the number one complaint in terms of volume and even quantity.
We were hacking the inventory system. This was meant to be this grand experiment in customer behavior. For me, to put a multi-quantity dropdown, I potentially had a race condition where you could ask for more than one item and I might only deliver you less than what you asked in terms of quantity. There was exposure to the consumer experience on the other end, which is hard to communicate via the website.
We did some more research. We finally came up with a way that we felt like, “We can probably cover for the race condition or at least we know how to dive and save for it.” I measured out before I had great data leading up to this list launch, which is multi-quantity. We launched multi-quantity. Lo and behold, we went back. Nothing happened except for the complaint. The complaints went down.
The consumer behavior did not change but the complaints disappeared. We did not open up. Some people were hoping that somehow, this was magically going to change consumer behavior. We had reams of data that wasn't it. In the Amazon Tote world, I learned a lot about too little and too much friction. We were always thinking to remove more friction but there's no such thing as removing. It was that Goldilocks of friction that was lesson number one.
Lesson number two is there's a big golf between quantitative and qualitative understanding of the customer experience. If you want to create the best experience, you need to have both. There was a subset of our customers who looked down on us and who weren't happy with us. It also created a lot of operational friction for our customer support folks too. Instead of being able to work on real problems, they were working on an optics problem of constantly dealing with these upset customers.
It’s recognizing that there's a world that a lot of people get into. Reduce friction at all costs and also let the data entirely drive itself. That's the only thing that you should pay attention to. There is Goldilocks in everything. There is a right balance. You have to look at a lot of different dimensions. Even when you know the answer is correct on one side of the equation, you have to look at other dimensions to come back with a holistic view of the customer experience. That was a big epiphany and a big eye-opener for me, which I've carried forward in how I think about the customer experience. I’m always working back and carrying multiple lenses along the journey.
There's a lot to dig into. I was going to ask about the learning. Here in the show, we try to emphasize. We're not shaming anybody for making a mistake. We're celebrating the learning that comes from it. Can you think of a situation later on where you were reminded, “Don't focus only on the quantitative data about the customer needs,” for example?
There's a greater bit to that too. I'll share a slightly different story. I've written about it before. It's about me and an ant. It was a previous place of work. I was on this bus. I commuted every day for an hour and a half one way. It was a three hours a day commute. I'm not the kind of person who generally goes out of my way to kill insects but I don't have a problem killing them.
I'm sitting on this bus and I see this ant. I have this moment of like, “I'm not going to kill this ant.” It's crawling down around by my toe. It’s a little bit of karma maybe. I had another thought. It's like, “I've been on this bus for 45 to 50 minutes. This ant probably came with me or someone else.” It's not like this ant's going to get off this bus, go find a new nest and say, “I’m moving in.” This ant’s effectively dead. It doesn't know it's dead. It's a matter of when. It’s because of the nature of ants. They're away from their people.
I had a moment of like, “This place I'm going to work is in the same way. I am going to die.” For me, it was a very toxic environment and aggressive. You and I could only have coffee and we'd go into a meeting. The next thing you know, you would open up a 12-gauge shotgun to me in that meeting. That was part of the course. I’d be like, “I thought we were on the same team.” You'd be like, “I'm taking care of my career and you're in my way.” It was heralded as a hallmark of the culture. They like that aggressive culture.
It was a long journey of understanding that I'm a high-functioning autistic person. Furthermore, I’m recognizing that I have a binary relationship with certain kinds of environments. I decided to change how I was behaving at work. Part of that's because I grew up in an environment where there was a sense of professional self versus personal self. You hear that a lot at work. You show up to work and act in a way that is “professional.” That's sometimes very counter to how you behave outside of work. There should be some differences. You get into this like, “You're a loving, amazing human being but inside of work, you're a different person. You’re unrecognizable.” It’s that kind of difference.
What I found for myself and this is like that ant, was I was trying to ape those behaviors that I thought were “professional.” Here is an environment that's very aggressive and in-your-face. It’s not how I want to reflect on myself. It's not how I want to act personally. Since I'm autistic, acting in a way that's inconsistent for me is like doing personality suicide. I couldn't handle it.
I showed up to work the next day after having this quite cathartic crying on the bus with this ant, as crazy as that sounds. I had a moment where I was crying. I don't think the ant cried but I was crying on behalf of the ant myself. I showed up to work. I try to be a warm and caring person. I would go into a meeting sometimes and would look to the person to the left or the right, especially if I worked with them closely and say, “I appreciate you.”
I could tell that they had a bad meeting and I might take a moment as we were meeting like, “I love you. I care about you.” It’s not something that most people would expect to find at work. It went about as well as you might expect it in that environment. A lot of people did not like it. The majority are probably indifferent. What I found was there were a few people that I resonated with and they were the people that I cared to work with the most.
That was the other piece of this whole mistake. I followed a career based on ego and chasing after the most interesting intellectual problems but I also wasn't solving the other aspect of myself, which was the interpersonal connections that I needed at work that I was not looking for when I went and interviewed for companies.
That was the moment and epiphany. That was the pivot in my life where I started looking at my jobs, where I worked and whom I worked with from additional dimensions. It’s not whether it is intellectually interesting but, “Are these people that I care about? Do they care about me? Are we solving a problem that I cannot intellectually but spiritually and emotionally get behind?” That was a complete game-changer.
As I would look for other jobs, people would extend offers. I would say, “Thank you but no, thank you. I've interviewed you. You're not the right company for me. Intellectually, I could do the job. I could knock it out of the park but I don't think I would enjoy working for you and I don't think you would enjoy the way I work. I don't need you to change to make me happy any more than, in some weird way, I'm going to make myself unhappy to change for you. I'm not going to do that. I'm not going to make that sacrifice.” That was a major change.
That's how I ended up in Central Washington with my wife. You start understanding what the most important things in your life are. One of the biggest mistakes in my life in my 20s and 30s was pursuing my career at all costs without putting it in context to the greater purpose or intent of my life. I don't think it's a zero-sum game or a mutually exclusive game that you can't have a great career and also work with great people that are right for you. That's why I want to be gentle with that statement.
Sometimes, we take overly strong proclamations, especially in the news with the way we talk about companies. There are companies and places I know I would never want to work or even some great companies. For example, Apple. I grew up with Apple. I've been programming since I was five starting with Apple. I always thought I was going to work at Apple.
I love Apple but Apple's very much an in-person. You got to be hands-on. You got to be down with the Apple company physically and I don't want to move. My wife and I do not want to locate to that. I would probably never be able to work for Apple or Disney, for example. They want that always in-person setting. That's not right for me but that doesn't make them a horrible company.
I worked at Amazon. Amazon's an amazing company with some amazing people but I can tell you for a fact that I'm not well fit or suited for their culture. I know a lot of people who it's great for. They love that culture. Who am I to turn to them and say, “You're wrong?” They should know what's best for themselves even if I would say it's not best for me.
A bad fit for me doesn't mean bad for all. Thank you for sharing both of those stories. There's something you said that I jotted down. It's a great phrase. It connects to a lot of the stories people have told here on the show. I've come to understand better from having these conversations. You said, “I thought I knew something and then I learned.” It’s the difference between whether we think we know something versus being able to recognize it's more of a hunch, an assumption or a hypothesis.
I think back to even this connection to Lean. Former Toyota people I've worked with, one of the things they love to ask if you make some statement is, “What do you know? How do you know it?” It’s helpful to be able to distinguish knowing something versus an assumption. Sometimes, assumptions are necessary. When you test an assumption with that thought process of, “I could be wrong,” that leads to better results instead of being stubborn, doubling down or refusing to be wrong. We're going to be wrong. Let's learn from that.
If you think about what innovation is, innovation is doing something that someone has not done before you. To go a little bit off in the weeds and I'll tie it back in, I came up through probably a culture and a belief in the worship of the written word or declarative knowledge. Someone else sat down. They gathered all the information together. They put it down in a formula. You could read it and study off of it. You'd go get a four-year degree.
You and I have maybe talked about this prior but just because you have a four-year degree doesn't make you an aerospace engineer. I have advanced degrees in Aerospace Engineering. What it meant though was I was qualified to go start at a company and become an aerospace engineer after another 10 or 15 years’ worth of experience. We have perverted what it means to acquire knowledge.
We have an overdependence on declarative knowledge, which is the knowledge written down. It is not innovative knowledge. Those are “unknowns” when most of the work we do is innovative, which means it's procedural knowledge acquired by doing. You're operating off assumptions and hypotheses all the time under the guise of, “I don't know but I will figure that out.”
I like to talk about fate or destiny. Your fate is predetermined by the sister's fate if you believe in that or God has ordained for you if you want to go down that route. The idea is that at some point, your fate has already been decided for you. To me, destiny is the idea that I make for myself and my future. I would argue your fate and your destiny come from the same place.
If you look at, “I know,” and, “I don’t know,” you can create this little 4×4 or 2×2 grid. It’s a quadrant. You can come up with, “I know,” which is the stuff you've already learned either procedurally or decoratively. The, “I know. I don't know,” is the stuff that you intentionally didn't decide to go learn. You didn't take that class in Accounting. The, “I don't know. I know,” is oftentimes, as we get older and you and I are probably about the same age, we start forgetting stuff but it's internalized to us. It's so deep in us. We don't even recognize that we have that as pattern recognition.
Our destiny or fate comes from that last box. Does the future of a company, if it's being innovative, try to solve problems? It’s, “I don't know what I don't know.” That box is disproportionately larger because that's the universe of actual existing possible knowledge out there to which we only know. Maybe I'm being generous by saying 1% but it's probably 0.0001%.
Most things that could be understood or are known are in this box of, “I don't know.” That is our ignorance. That's where our innovation comes from. If you do not have a culture or you are not set up personality-wise to go, “I fundamentally normally don't know,” then you'll never truly be successful as an innovative leader nor will you be able to create innovative teams. That gets back to that celebration of errors, which I took from Amazon. They called it the correction of errors. The idea is how do you create a mechanism to create a culture that's psychologically safe to make mistakes and learn? That is what the celebration of errors is all about.
[bctt tweet=”How do you create a mechanism to create a culture that’s psychologically safe to make mistakes and learn?” via=”no”]
Fundamentally, what we're trying to do is acknowledge, “I don't know but it's okay.” Please go figure it out. Go make a ton of mistakes because that's how you're going to learn. The mistakes are going to teach you. Through those mistakes, you go on and find new interesting mistakes to make. That flywheel of innovation comes from, “I don't know.” You trip, fall, pick yourself up and go, “Now, I know.” You do it all over again. You find another thing to go trip up over.
The last thing that I learned and I'll put this gently, was when I was at my first company, Boeing, I met a lot of people. I was coming out of grad school at that point. I was in my late 20s, coming into my early 30s. I would meet people at their anniversaries at work and they'd been there for 30 years, which still blows my mind that someone would be at one company for longer than I had been alive at that point. It's an amazing testament.
I'm using an example from Boeing but this isn't specific to Boeing. This is true of anyone. This is something we should all aspire to avoid. They got very used to being experts in a single subject and they got brittle emotionally and intellectually around not knowing something. What they did is they tended to avoid things that they didn't know and they focused on the things that they knew as experts. I found that that created a gap in their ability to grow and be innovative as engineers.
They tended to retread the same space right over and over again in part because the culture did not promote, “It’s okay to say you don't know. Go out and figure it out.” That was another learning that, through people, I realized I wanted to become an expert but I also did not want to have my intellectual self and curiosity stamped out. My ego, which was my read on it, would not allow me to say, “I don't know,” because a culture wouldn't allow you to say, “I don't know.” You’re expected to know.
You touched on a couple of important aspects of psychological safety in the workplace. There's a book that I love that I read. It is The 4 Stages of Psychological Safety. We’ll go through it quickly. You'd be interested in this. First off is inclusion safety. Do you feel accepted and valued as yourself? The one expression Tim Clark uses is, “Is it expensive to be yourself or not?” You've touched on this. There are environments where you feel fully accepted for the human being that you are, not the resource who's contributing whatever work.
The second stage is learner safety, which includes, “Is it safe to say I made a mistake? Is it safe to say I don't know? Is it safe to try and learn something new?” This is a helpful exercise. Go force yourself to go take a class on something that you don't know much about to remember what it's like to be a new learner. That builds safety for people that you're hopefully working with.
You have inclusion safety, learner safety and contributor safety. Are you safe to do work that makes a difference? That's a human need. The fourth level that's harder to get to is challenger safety. Is it safe to challenge the status quo? That's where innovation comes from. It's all connected. If you have a culture where you're only allowed to do experiments that you know are going to succeed, that's not innovation. The primary objective though in a lot of workplaces is, “Never be wrong. Don't have a hypothesis that doesn't play out.” You could survive that way but you're not going to thrive.
Even if you argue 80% of what you should be doing, which is maybe more turn-the-crank work, you still need some amount of innovation in that to keep on growing, evolving and changing to the times. We as consumers are inherently fickle. How we behave and operate and what drives us are going to be different even in a span of 1 year or 2. Companies who know how to listen to their customers and use that to drive their innovation will always succeed. Those who keep on doing the same thing, at some point, will be left behind. We can see lots of companies that have gone that way. That's part of the nature of companies.
I love the idea of, “I get to first be myself. I get to acknowledge myself as flawed.” The next piece is, “Even as a flawed person, I can contribute.” The final one is the test. It’s like, “With all those things said, I can also come in and challenge.” That's as much a statement about what kind of leadership you have and the psychological safety of the leadership be it okay to say, “I don't know.” It’s also being able to share in the leading of the company.
That fourth stage is the deep one. That's probably the hardest for a lot of companies to get to because it requires everyone at the company, top to bottom, to buy into psychological safety. I've been in places where I can create a lot of safety for everyone around me but maybe my peers don't agree with psychological safety. The moment I allow someone to challenge me, even though I might be comfortable with it, my peers and my boss might see it as a weakness. They’re like, “Ward has his team undermining him. They question him so why don't I even have Ward?” They don't understand the value of psychological safety.
There are misunderstandings or mistakes. People think psychological safety is the safety of not being challenged. No. It’s creating safety for people to challenge you. If you're not a leader who wants to be challenged, then, for one, don't promise people that you're going to have a culture of psychological safety and this is a safe space. Saying that doesn't make it happen.
I want to go back to Celebration of Error or CoE. You said something that I want to explore. All errors are not created equally. Would you equally celebrate the error that comes from the part of the work where you're cranking out work versus an error that's made when you're trying to be innovative and doing something brand new? Do you need to have a culture that celebrates all the errors?
I prefer to see the more innovative. We were pushing ourselves over our skis. I always go back to when I was learning to ski with my father. I used to complain that I would fall flat down in the snow and how horrible I was as a skier. He was like, “No matter how good you are, if you don't fall at least once or twice a day, you're probably doing it wrong. You're not pushing yourself.” I want to see more of like, “We pushed ourselves to the edges. We were pushing on the envelope of what we could do.”
With that said, there's nothing wrong with it operationally. That can be a celebration too. I want people to own their processes. What does Lean teach us? It advocates for pushing decision-making down to the lowest level in the organization of who is most informed by the decision. Celebration is it is also letting them know, “It's okay if you made mistakes.”
There are a few things that we go through. I always ask my leaders, “Should we celebrate everything?” We write a CoE. At some point, there are diminishing returns to writing everything up. You should be focused on, “Am I going to learn something from this?” If you don't think as a leader, don't spend the time and energy to do the root cause analysis and look for short-term and long-term solutions going forward.
Also, make sure you follow through. That's the other thing. I don't want a bunch of CoEs where we say, “We could do this in the future to avoid this mistake.” If we are going to do a CoE and put a long-term fix in it, we are committing ourselves to follow through. That's the other piece. I want to teach the follow-through.
There are lots of things we do with CoEs but a lot of it is about ideally trying to build better processes that protect us and give us more opportunity to spend on the edges. This is no disrespect to anyone. In RealSelf, when I joined, a lot of the mistakes that we discovered in production had been there for 3, 4 or 5 years. No one was paying attention or looking. Now, we’re down to about fifteen minutes if something sits in production normally before we find out about it.
Over time, through the CoEs, we've added better instrumentation monitor and learning. We know what to look for and the teams know how to monitor those things more proactively. It has incidentally driven a lot more trust from the rest of the organization back into the technology org that the technology org is taking care of things.
The worst thing I always tell my teams is, “I don't mind you making a mistake. I don't like it when someone else from the other part of the organization or worse, our customer tells us about the mistake. If you tell me about it, that's an okay thing. Otherwise, we should all look at it like, ‘We failed our partners at the company because they had to tell us that we made a mistake in production.’” The worst one would be our customer coming back to us after a month and saying, “Did you know you have a bug in your code?”
[bctt tweet=”I tell my teams I don’t mind you making a mistake. I really don’t like it when someone else from the organization, or worse, our customer, tells us about the mistake. If you tell me about it, that’s okay.” via=”no”]
Being psychologically safe doesn't mean being challenged. It means being challenged and okay with that. It means being accountable without being fearful of the consequences of that. How do you build a culture around that accountability and the ability to communicate outward and keep people informed? In our weekly business reviews, we share out our CoEs.
The rest of the organization uses CoEs in the same way as a broadcast mechanism. It’s like, “Customer support, all those complaints you're getting about, engineering knows about it. Here's what we're doing about it.” We also have mechanisms, if that's truly the case, to escalate so customer support doesn't wait until next week to find out about this because that oftentimes happens.
Something happens in the system, especially in software, where customer support is on the other end of the phone getting these people screaming at them. They have no idea what's going on because they're not involved in the day-to-day nuances of production releases. It's important for engineering to immediately go off the customer support and say, “All those calls are from us. We know about it so you can tell customers we're aware of it. Thank you. Here's our ETA or ETC of when we're going to get this fixed. We'll tell you once we get it fixed.” That makes customer support a great ally in that case.
In other companies, if you don't do that, customer support starts to hate the engineers because the engineers keep on breaking stuff and not telling them. They end up being the poor souls at the end of the phone call getting yelled at with no recourse and no way to communicate. They never get to feel like heroes, which is a shame. How you build that connective tissue in your organization is another reason why we do CoEs formally.
The underlying question is I try not to prejudge CoEs. I'm trying to maximize learning. I don't want to say, “Your organization isn't at the edge of innovation. Therefore, I don't care if you make a mistake.” I want you to improve your processes. I want you to feel ownership over it and not be fearful that your process isn't working the way you want it to. This is a deeper Lean learning.
One thing when I was working with Shingijutsu, we always talk about how it's always the process over people. People don't make mistakes. Processes do. People don't show up to work and go, “I'm going to have a bad day today. I'm going to ensure that something goes wrong.” You can go find me an exception case but that is an anecdotal one versus 99.999% of people who do show up and care about the quality of their work.
There's a difference between sabotage and errors. Sabotage is pretty rare.
That's another way to say this. What you're talking about is sabotage. What we're talking about is human error. Human error drives processes, communications and setting expectations. There are lots of things but it's not the person that made the mistakes. The CoE is also meant to create psychological safety. It’s like, “It's not you. It's us. Somehow, we failed you as a team. How do we, as a team, go ensure that this doesn't happen again? Since it happened to you, could you also be a part of the process? You probably know best.” You’re like, “That wording and that little line there was confusing.”
Sometimes, they are super easy things but if we don't enculturate people to go fix it, then people won't. They’ll go, “That’s someone else's job.” It's all our job. If you're touching that, go fix it. That's what Lean always teaches. It is to make those decisions at the lowest level and have the people on the line own the line. CoE is another means for exactly that philosophical belief.
With that philosophy, there are a lot of important mindsets here and then there's the CoE document. When you refer to writing these up, is there a template available on your website? Is the template even not as important as covering the main components as that Forbes article described?
Forbes goes over the template. I'm happy to share out. It's super simple. It’s how you talk about it. I even have an hour-long lunch and learn. I'm happy to share out links to that. If anyone wants to, I'll come and talk to them about it. The format is the vehicle floor but the top of it is an impact statement. I use that to teach, especially more junior leaders, how to communicate more at an executive summary level. Can they write that in 3 to 5 sentences using quantitative data? How many customers were impacted? When did it start? When did we observe it? How long was it in production? Was there a monetary impact? It’s quantifying.
This can be useful. For example, I'll take engineers. Oftentimes, engineers are the most distant from a business acumen standpoint in understanding the impact of a line of code on the impact of revenue or cost. When they write that CoE, they're like, “That one line of code that took down the site for 15 minutes maybe cost a company $5 million. That's a lot of money. That's a lot of responsibility that I have as an engineer.”
It’s that versus if you look at it as lines of code, they'll go, “It’s one line of code.” I’m like, “It's one line of code that took down the site.” That's okay at some level but please don't distance yourself from that. Please understand and connect yourself to how much impact you can have on this business positively and negatively. In this case, not so great but let’s use it as learning. Let's use that as an investment in your education about how important that one line of code is. That's impact. We do 3 to 5 sentences. It’s good to learn how to be succinct.
The next section is resolutions. I do short-term and long-term. I normally break it into short-term and long-term. The short-term is, “Put the fire out.” We had a fire. That's the impact. It’s the extent of the fire and how much damage it made. It’s then short-term resolutions like, “Here's what we did to put the fire out. We went and started the server back over. We got customers back getting their orders. Fundamentally, the error still is this but at least, we no longer have that issue out in production.” It could also be, “We rolled back the code and fixed it.”
The long-term solution would be maybe it was a simple bug fix or it might be something deeper. As you're getting through the root cause analysis, you might go, “Architecturally, we're set up incorrectly. We have a major probability of this reoccurring. Should we go re-architect that?” I get into like, “Please do not pontificate on the shoulds. Either do it or don't do it. I don't want to hear about what we should do.”
I pick on engineers. I deal with them a lot. I see this a lot. They'll use it as a parking lot to gripe about massive tech debt that no one's ever going to pay down. It’s like, “Let's not talk about that because it doesn't engender anything helpful. I want you to be proactive about what you will do, not what you could do. Don't moralize the problem. Be very much proactive.”
At the bottom, it is the root cause. A lot of people will go, “CoE is a root cause analysis.” I go, “The bottom section is the dumping ground for your root cause analysis.” That's when we get into the five whys but I use that as homework. What I carry as an executive is the first two sections, which are impact and solutions.
I look at the root cause largely to make sure that I have faith in those first two sections. I almost don't need that third section of, “What was the root cause as an executive?” I use that as a fly-by like, “Did you dig into the code? Did you talk to other people? Do you understand what you're proposing or were you trying to slap this problem shut and go away?” That sometimes happens. If I don't quite believe in the impact or feel like the solution is keeping at the root of it, I will look at the root cause. That's where I can interrogate the team's thinking there. The format is pretty easy. I try to keep it to 1 or 2 pages. The first 2 things should be 1 page.
That Forbes article is titled Celebrating Errors Creates Psychological Safety in the Workplace. When you talk about impact resolutions and the root cause, it sounds like that's the order of documenting it. It’s where the thought process is thinking through the root cause so you know what the long-term resolution or the long-term countermeasure is. That's where you show your work.
That’s the document's deductive. To your point, inductively, you would probably need to write the first piece, the bottom piece or the root cause first. Though I'd argue the impact does not need to be done with the root cause. That’s the one thing a lot of people trip up on. They’re like, “I haven't done the root cause analysis.” The patient is dead on the table. You don't need to understand all the mistakes made. The patient's dead. Those are your 3 to 5 sentences.
I'm using an extreme example from Shingijutsu. We were always talking about the surgeon. It’s like, “There's a defect. There is a problem.” What is that defect? How big is it? That has nothing to do with how you got there. That's describing like, “Here is the outcome of that.” That's to inform people. That can be written normally within 24 hours of a defect or an issue being found, especially if it has a material impact on the business that other people need to be aware of in short order. I would recommend no later than 24 hours from the time when you start writing the CoE.
Once you get that resolved, then go spend time doing the root cause. Your short-term resolutions might also be written and done and executed in the same amount of time. A lot of times, you're trying to make the mistake go away. Put out the fire. Short-term or long-term, make sure this particular fire doesn't happen again. I find that framework useful for people.
A lot of times, they try to solve it all and do it all at once. It's like, “Put the fire out.” We bought ourselves a little bit more time to breathe and go, “What would we need to do to help us make sure this problem doesn't happen again?” If we can't do that in a lot of instances, especially in an engineering world, how do we monitor an alert for ourselves to make sure that if it does happen again, the meantime to response is faster and/or ensure that it's not a customer telling us but it's our on-call person getting alerted at 2:00 AM if we think it's important enough?
I'll be looking for those kinds of things. I’m like, “Was this a manual detection? How do we automate the detection? Were these 7 hours from injection or 7 days or 7 months? How do we get it to be 7 minutes or 7 seconds? Better yet, how do we get it in unit testing before we even get it to production so that we don't even ever see it and it becomes part of the course?” You’re always trying to improve things even if you can't necessarily eliminate things.
You can use a CoE as a very good way to gauge the healthier organization and along the way, teach all these other things. It's all rooted in psychological safety. You won't get accountability, communications or growth if people don't feel safe. It's as simple as that. They got to feel safe. We all have to feel safe. Top leaders, too.
I was going to ask you one final question before we wrap up. Our guest has been Ward Vuillemot. You can find his website at WardVuillemot.com. We can talk about firefighting and the root cause analysis. It's a visual joke because it was a cartoon that I created with somebody. It’s a couple of firefighters. There is a burning house and they're grabbing the hoses. The one turns to the other with this piece of paper and says, “Have you started doing your A3 yet?”
This is a very nerdy Lean Toyota production system problem-solving kind of joke. The point is a serious one. You got to put the fire out first and then we can think about defining the problem. The problem was right in front of us. We can do the root cause analysis once everyone's safely out of the building and the fire is no longer spreading.
This is the other question I was going to ask. It's unfair to ask you this question on the way out because this could be a whole 30-minute discussion in and of itself. When you talk about the safety to be yourself and being who you are and you disclosed under your website and mentioned earlier that you are autistic, is it ever a mistake to disclose something that personal with people you're working with?
It's only a mistake if you think it's a mistake. Every decision has a cost. There's nothing that's costless. For me, by sharing that I'm autistic, I'm sure some people don’t want to work for me or work with me or don't want to hire me. The way I look at it is I'm okay with that because it's more important for me to be true to myself than necessarily having that opportunity.
A different person might decide differently. Their calculus might be different. It's hard for me to say, “Maybe that one opportunity or that job you wanted, you're not going to get.” I would argue it's not worth the job if it means I can't be my true self but that's me. That was something that I learned. I try to take a weak form of opinion. For me, this is what I know. You can use that framework to make a different decision for yourself. I rarely will have a strong opinion. I don't want to work in a world where people have to hide because I don't want to work in a world where I have to hide.
When we think about inclusivity, as you talked about even from CoE but I even think of diversity, equity and inclusion, that's a part of allowing people to come as they are. I quip that the T in CTO, people are like, “What is it? Technology?” I sometimes say, “It's Chief Therapy Officer.” Much of my job as a people leader is helping other people become more comfortable and integrated with themselves. I find that a little ironic because I am autistic but probably because I've spent the last couple of years trying to integrate with myself.
I'm a pretty good observer of human behavior. I, at least, have some ideas for helping people. I do spend a lot of time helping people become more integrated versions of themselves. That's important to me. That might not be important to everyone. Some people might be like, “That would be my nightmare if I had to work in that environment.” That's great. There are lots of companies that aren't like that. You should go work for those.
He is the Head of Titan Casket. When I interviewed him from Amazon, the last thing I said as we were ending the interview was, “I will oftentimes tell you I love you. If you're uncomfortable with that or uncomfortable telling your team, you probably shouldn't come work with me.” He said that was the thing that cinched the deal. He wanted to come work with me because of that. I am true to myself and I want people to be true to themselves too. What more can we ask of anyone?
We can't ask anything more of you. Thank you so much for sharing your stories, experiences and insights here. We appreciate it.
Thank you. I appreciate it.
We've been joined by Ward Vuillemot. Go check out his website, WardVuillemot.com. He has a great article. At first, I almost made the mistake of attributing the authorship to Ward but it was written by somebody else about the CoE method. It’s titled Celebrating Errors Creates Psychological Safety in the Workplace. Ward, thanks again.
Thank you.
—
Thanks again to Ward for being our guest. As always, I want to thank you for reading. I hope this show inspires you to reflect on your mistakes and how you can learn from them or turn them into a positive. I've had the audience tell me they started being more open and honest about mistakes in their work. They're trying to create a workplace culture where it's safe to speak up about problems because that leads to more improvement and better business results. If you have feedback or a story to share, you can email me at MyFavoriteMistakePodcast@Gmail.com. Our website is MyFavoriteMistakePodcast.com.
Important Links
- RealSelf
- WardVuillemot.com
- The 4 Stages of Psychological Safety
- Celebrating Errors Creates Psychological Safety in the Workplace
- MyFavoriteMistakePodcast@Gmail.com
- I WANT TO WRITE MY BOOK
- Follow – Mark Graban, Subscribe to My Favorite Mistake
- Apple Podcasts – My Favorite Mistake
- Podchaser – My Favorite Mistake
- Become a financial supporter of the show through Anchor.fm
- Sign up to get new episodes via email
- Lean Communicators Network
- https://Anchor.fm/favorite-mistake/Support