Nothing can be better for your software than to have a new release go through a beta program where customers (even prospects) put the software through its paces in a real-world environment. When the release is announced, not only are you confident that most major problems have been identified and rectified, but you have case studies for your marketing collateral, endorsements for your press releases, and best of all, positive references ready for interviews with the media.

However, while those who make quality their career may clearly understand what beta software means in terms of number and type of bugs and other issues, this is a murky area in the minds of your beta users and the executives they report to. It is critical that you set expectations with your beta program participants if you want the experience to leave a good impression. This lays the foundation for acceptance in the marketplace, and can mean the difference between good reviews in the trade journals and getting trashed for problems that were never serious in the first place.

In my experience with beta software, I have seen a common pattern of problems, and responses to those problems, that has led me to set expectations in a very specific way with beta customers in order to leave a lasting impression of good quality software. This approach is described below.

Painting the Picture

When I have signed up a customer for our beta program and the time comes to deliver and install the software, I usually give a speech along the following lines to those who will be the beta users and their direct managers so that they know what to expect:

“Beta software has been put through extensive testing at our company. In addition to testing individual functions of the application, we put together scenarios reflecting common ways the customer base uses the software. The beta program takes testing a step further by putting the software through its paces in real-world situations at your company.

In our experience, we don’t encounter many serious bugs or issues during beta, however, there will always be some problems that must be fixed. Problems generally follow a pattern, namely:

• The very first time you go to use the software, the software may not work at all, usually because of a minor problem or setting. While this is dramatic, it’s important not to panic, because the solution is usually a small change, and then the software is up and running fine.
• Once the software is running, there are usually one or two major modules or capabilities that don’t work. Again, this is usually fixed by minor changes or settings, after which the features work as expected.
• As you work with new capabilities, we generally encounter a couple important problems with them that require fixes to make the feature run more smoothly or effectively. Sometimes certain buttons or functions simply don’t work. When these are fixed, you generally won’t encounter further serious problems with the feature.
• Finally, there will be a number of minor problems, as well as suggestions from you to make things easier, faster, or more convenient. We try to fix as many bugs as time permits and also either put in enhancements or schedule them for the next release. The vast majority of these are issues that most customers can live with for the time being, considering the benefits they get out of the new features.”

Each stage of this pattern is discussed separately below.

Flipping the Switch: the Whole Thing’s Broken!

This one is the big deal. Invariably, when you go to install the beta software at the customer site, there is a setting or variable set long ago which slipped everyone’s mind. Even as the software was reinstalled repeatedly during testing, with all the code wiped out each time, nobody bothered with this setting.

And the setting causes the whole product, or some major component of it, to fail to launch.

This kind of problem looks disastrous, but usually after a little head-scratching by the developers, they figure out what to tweak in the new environment and everything runs smoothly.

But the last thing you want is for someone to run to the top manager and say “The whole thing’s broken! They can’t even turn it on!” Unfortunately it’s the simplified version of things that gets remembered and passed on. There are two things to do to try to minimize the danger.

1. If at all possible, try to expose the installation to a restricted number of people at the customer site, preferably in the IT group, where they put technical glitches into perspective. When plants have been started out in a greenhouse, they need to be gradually exposed and hardened to outside conditions. They can’t just be uprooted and planted outside into the temperature changes and weather. Your beta software is like those hothouse flowers. The ideal situation is to have a team from your company install the software and test out the major areas on site before showing it to a customer.
2. The other tactic is to set expectations up front that this may happen, so that when the switch is flipped and nothing happens, nobody panics and goes running to their boss, who simplifies things even further to their boss.

At some point, you hope the media will be talking to the top contacts at your beta customer. Successfully dealing with this kind of dramatic — and minor, in terms of the fix — beta problem will mean the difference between a headline that says “New Release Overcomes Serious Problems to Go Commercial” and “Latest Release Provides X, Y, and Z Capability.”

Next: I Can’t Believe That Doesn’t Work!

Once the overall application is up and running, you may very well find another problem almost as dramatic as the first one, but limited to one component. The component that happens to be the most critical one in the customer’s eyes.

The typical reaction is: “I can’t believe that doesn’t work! After all, that’s the heart of what your software’s supposed to do!”

Again, judiciously setting expectations will tone down the reaction and head the detractors off at the pass. Often this kind of problem is just a simple and quick code change away from being solved.

Later: I’ll Forgive, But I Won’t Forget!

When all the major components are up and running, beta users will encounter areas of major confusion, or buttons that don’t work as expected, or come up with suggestions to streamline and improve the features as they use them. Most users will be willing to wait while these changes are made. They might even be okay if changes are put off till the next release. But they’re not going to forget about them, and you’re going to have to follow through on commitments. They may very well be mentioned, if only in a vague way, during an interview, and you want to make sure that your beta customer comes across as comfortable that they can live without the features for now and confident that they will be in the product soon.

Finally: No Biggie! This Works Great!

All along the way, there will be a number of minor bugs or improvements identified. It’s important that you take them seriously, and record them, and if possible report back on the schedule for their incorporation into the product. They may not be scheduled for certain until the next release. Or fixes may show up during beta.

As long at the beta users feel that the new capabilities bring an improvement into their lives, many of these issues may even be forgotten. If you’ve gotten to this point, you can be certain that you have a beta reference who can give customers, prospects, and journalists a comfort level with the value of the new release. They may even help reassure other customers who encounter some of these issues.

Expectations Met

When you work to set the expectations among the beta users, their managers, and the executives above them, so that typical and inevitable problems arising during beta testing are seen in the proper perspective, you’ll have the momentum needed to come out of the launch at full speed.

— Jacques Murphy, Product Management Challenges

ProductManagementChallenges.com