Here is a hypothetical meeting that you can probably relate to: The boss calls everyone together because the company is losing money on freight. He solicits ideas for what can be done from the group. Ideas are written on a whiteboard. Sales people tell stories about “this one time this one customer told me this one thing” that makes them believe the company should change course based on one data point. Some ideas are to lower cost. Some ideas are to raise revenue.
At the end, everyone votes on their favorite idea. When the winning idea is picked, resources are assigned, marching orders are given, and schedules are set for a full implementation of the idea.
What’s wrong with this picture? Absolutely no data has been collected on the potential benefits of any of these ideas. The group has only their best guess as to which idea is the best. It is a lazy process to make a decision without putting forth any effort to collect data on the problem and potential solutions. If the first idea does not work, you have wasted time and resources and will probably just repeat the same process of going all in on the second most popular idea and hope that it works out.
Executives and business owners make decisions all day long. The best of them are correct slightly more often than they are wrong. Businesses are complicated machines with too many variables to understand absolutely. Hundreds of products, thousands of customers, customized pricing rules, changing market conditions, competitor strategies, supplier performance, etc. all interact to make things unpredictable at best and unfathomable at worst.
Failure is not a disappointment in that it serves to point the way forward by eliminating paths that should not be taken.
Those executives that have the humility to accept the fact that they really don’t fully understand their own business are the ones who demand experiments and data before making decisions. They understand the bullets to cannonballs concept of Jim Collin’s book, Great by Choice. Bullets to cannonballs essentially boils down to the fact that you should constantly be running many little experiments to see what works and what does not. When you find something that works, double down on it and go all in to maximize the impact. When you find something that does not work, that is valuable as well because you can quit talking about it in meetings because you know it doesn’t work.
Amazon has a goal to get into the healthcare space with a new, disruptive model. They launched a telehealth and primary care service for employees in 2019 called Amazon Care that they talked about growing into a national healthcare provider. Recently, they shut down Amazon Care without achieving that goal. A few years ago, Amazon, JP Morgan, and Berkshire Hathaway started a project called Haven to fix the ills of healthcare. Haven was dissolved a short while later without any tangible impact.
Does this mean that Amazon is done with trying to disrupt healthcare? For those of you who are terrified of Amazon having your health information along with your purchase history and your streaming preferences, the answer is unfortunately “no”. Amazon is pursuing the purchase of established telehealth and primary care service companies to replace Amazon Care. They have acquired pharmacy companies with unique models and rolled them up into Amazon Pharmacy. Amazon’s goal of disrupting healthcare has not changed. They are simply in the process of trial and error to determine what their model is going to be. One of the secrets to Amazon’s success is that they seek the answer through experimentation. Failure is not a disappointment in that it serves to point the way forward by eliminating paths that should not be taken.
The outcome of our hypothetical meeting should have been the list of all of the ideas that we can test rather than going all in on one idea. Not all ideas are easy to test via experimentation. Amazon can start up or buy companies to test concepts. Not all of us are so lucky. But the nice thing about parts distribution is that there are a lot of transactions, customers, and products that make it relatively easy to design and execute experiments that will quickly have results. Here are a few tips on how to design experiments to understand your business.
Keep it Small and Cheap
In order to comply with the principles of “bullets to cannonballs”, you have to keep it small. The concept of a control group vs. an experimental group is a good way to measure the impact of a change in an environment with a lot of variables. The other benefit of the experimental group is that it can be relatively small in order to keep costs down. If you are considering adding a tranche of parts to stock that meet a certain sales criteria, you shouldn’t just buy them all and measure sales before and after. You should buy somewhere between 20% to 50% of the parts you want to stock and measure the before and after sales of both the stock and non-stock group to isolate the impact of stocking the parts.
Modern e-commerce tools enable A/B testing which can be helpful in quickly testing messaging and offers, but there is plenty you can do without involving your IT department.
In a call center environment, don’t underestimate the power of a well-designed stick tally form to document offers and responses. It is pretty easy to document how many times a special offer was made and how often it was successful.
If you need to mark a customer who is part of a test group that needs special treatment, just add an asterisk after the company name in your system rather than code up some pop-up indicator or create a separate field. Your contact center folks will see it if they need to execute a specific script or offer. Your warehouse people will see it on the packing list if they need to execute a special pick/pack process or marketing insert. When the experiment is done, go back and delete the asterisk.
If You Can Undo It Then Try It
If you find yourself in a meeting where people are debating whether some change or initiative will work and there are opinions on both sides, ask yourself if the change is reversible. If you can try something and undo it if it doesn’t work, don’t waste your time talking about it and just try it.
You should default to “yes” for any experiment that is small, cheap, and reversible. Pricing changes are the most obvious example of something that is easily reversible. You should always be running small pricing experiments to optimize your profits.
If you think you could make more money raising the price on a brand of parts you stock more aggressively than the competition, then try it. If you think you could increase sales significantly by lowering the price on a set of fast-moving parts, then try it. If you think you could lower your cart abandonment by shaving a couple bucks off of your ground shipping price, then try it.
If you can try something and undo it if it doesn’t work, don’t waste your time talking about it and just try it.
You should be able to tell the result of your experiment pretty quickly. If it doesn’t give you the result you want, switch it back and know that you have learned something about your customers.
If lowering the price doesn’t give you the result you want, try raising it just to see what happens. It is extremely valuable to really understand how price sensitive your customers are by the different groups of parts you sell. If you can raise your price without losing sales or lowering conversion rates, you have just made more money for doing the same amount of work. If it doesn’t work then just change it back and take that knowledge you have gained with you into your future decisions.
Sometimes It Is Easier to See if You Can Make the Problem Worse
Sometimes the change you are pondering might be very expensive or disruptive to your business. If you want validation that you are doing the right thing for your business before committing the time and resources, sometimes it is easier to prove your hypothesis in reverse.
As an example, back in college I worked as a mechanical engineering intern at Fermi National Accelerator Lab. For those of you who are not familiar, Fermilab is a Department of Energy research facility that smashes matter and antimatter together to see what subatomic particles make up matter. I worked for a man named John Grimson who was the head of the mechanical engineering group. John basically built the place and was a truly great boss who taught me things that I often found myself passing on to many of the folks who worked for me over the years. John had a problem at the Collider Detector. The physicists were not getting the results they expected from an experiment. The way the experiment worked is that they run millions of collisions and measure what happens. The same thing doesn’t always happen when you smash matter and antimatter together, so the experiment counts on statistical analysis of the results to determine the probability that the standard model of matter is correct.
The hypothesis was that the steel beam pipe the matter and antimatter ran through was too thick and therefore obstructing the path of the subatomic particles that came out at low angles after the collision. Passing a subatomic particle through a steel pipe is like hitting a golf ball through a tree. You can see sunlight through the tree so you know that there is a path, but good luck hitting your golf ball on that exact path without hitting at branch or leaf.
The beam pipe looked like a steel pipe, but it was actually a highly engineered vacuum chamber that contained the matter and antimatter while keeping everything else out that could disrupt the beam. Needless to say, it would be really difficult and expensive to change out the beam pipe with a thinner design. You would definitely want to know that what you are doing would solve the problem before committing to the disruption to make it happen.
John’s solution to this dilemma was to try to make the problem worse. If making the beam pipe thinner would fix the problem, then making the beam pipe thicker would make the problem worse. John had the machine shop cut some stock aluminum tubing in half to make a shell and then had those shells cable tied to the exterior of the beam pipe. The physicists ran another set of collisions with the “thicker” beam pipe. Sure enough, the results got worse and the hypothesis that the thickness of the beam pipe was a problem was confirmed.
My other example of this method is how we accidentally discovered how powerful an impact inventory availability has on parts sales. We didn’t start by spending a lot of money on inventory to see what happened to sales. Instead, I zeroed out all the inventory on our website over a weekend while moving facilities and was surprised by the result.
Our process was to move slow-moving parts during the week to the new facility. The new facility was a separate warehouse ID in our ERP system whose inventory was not being fed to our website because we weren’t shipping from there yet. On the cutover weekend, we moved all of the remaining fast-moving parts. On Sunday night, we switched the inventory feed to our website from the old warehouse to the new warehouse. But the result was that we showed zero inventory on all of our parts for the entire weekend. Normally, we would get about 80 orders through the website over the course of the weekend that we would process on Monday morning. Over this weekend where we showed no available inventory on the website, we got 20 orders. This triggered a lot of investigation into the relationship between inventory availability and sales that ultimately made us a lot of money.
Don’t Let Perfect Be the Enemy of Good
Fermilab was the only place I have ever worked where the engineers were the cool kids. That’s because the other clique was the particle physicists. For as socially awkward as engineers might be as a group, particle physicists take it to another level. John Grimson had a favorite joke that contrasted the engineers and the physicists:
There was a mixer between the all male technical college that taught engineering and physics and the all female college down the road. When the DJ noticed that there was not much mixing going on at the mixer, he had all the men line up on one side of the room and all the women line up on the other. He told the men to walk halfway across the room. When this was done, he again told the men to walk halfway across the remaining distance towards the women. One of the physicists asked the DJ if they were going to keep doing this all night. The DJ said “yes”. All of the physicists turned around and left because they realized that they were never going to reach the women. All of the engineers stayed because they knew they would get close enough for all practical purposes.
The lesson here is that your experiment doesn’t have to be perfect and all inclusive. There is such a thing as “close enough”. If you have an offer or promotion that you want to test, don’t delay or not do it because you can’t execute it through every channel. If you can’t code the promotion quickly on your website, just run the promo through an inbound call script in your contact center. The faster you start learning the better. You might find the promotion isn’t worth doing on the web based on customer reactions in the contact center. Or better yet, you might refine the promotion through your learnings in the contact center so that you have a better promotion to launch on your website.
There might be parts categories or manufacturers for which you can’t execute a particular program. There might be customer segments that you can’t reach easily. It is OK to run experiments and test things on less than the whole and extrapolate those conclusions to the rest of the universe of customers and parts.
If you have multiple elements to the initiative that you want to try, don’t wait until they are all ready to roll them out. Do the easy stuff quickly and start learning. The information you gain will inform the way you roll out the more difficult steps of the process.
Your business is like Fermilab. In both places, there should always be experiments running. As a parts distributor, you have a lot of transactions, parts, customers, pricing rules, stocking profiles, and manufacturers that interact to make it difficult to know what the impact of a particular change is going to have on your business. Run experiments to help you understand the levers in your business that you can pull to really make a difference. Rather than brainstorming and voting, default to action and try as many small experiments as you can to determine the path you need to take to optimize your business results. When you find the few changes that really make a difference, roll them out more broadly to maximize their benefit.