Situation: A CEO has a new app that her company wants to test online. The principal challenge is avoiding a “catastrophic success” – success that ramps so quickly that the company is unable to deliver the quality or responsiveness expected by users. How do you best test a new app online?

Advice from the CEOs:

• The challenge is similar to that faced in the massively multiplayer game space.
  • Creators target a small number of known enthusiasts (sneezers) with the message that they are special. The creators ask them to preview a new game and provide feedback that will help the creators produce the best game possible.
  • Never apologize for an Alpha or Beta test. Let enthusiasts know that they are getting the first peek at what will be the greatest thing since sliced bread. Enthusiasts will tolerate Alpha conditions – as long as the company responds quickly to their suggestions for service or performance improvement.
• For initial live tests hype the coolness and uniqueness of early availability and adoption.
  • Don’t lower expectations – manage them by responding very rapidly and fixing any glitches. This is why Web companies are 24-hour, eat and sleep in the office affairs during launch and for as long post-launch as needed to assure success.
  • Continually hype the coolness of being involved early.
  • Use the current version as the early test. When the company is ready to spread beyond the very first users, reward them for sneezing the app to other users.
    • For example, as a Beta Testers, users get 10 free 1-year plug-ins to give to their friends. For each additional user that they bring on-board, they get an additional 10 free 1-year plug-ins.
    • This technique supports the coolness of having been a Test participant because it makes the participants cooler with their close circle of contacts. The really smart ones will give free plug-ins to other sneezers and influencers. Reward this latter group for bringing on additional users.
• Using lessons from the gaming market:
  • Shake out all issues pre-Alpha Test.
  • Conduct automated testing of the software via server farms that are set up for this.
  • Be prepared for upgrades – both in the software and in the server farms. Typically upgrades are conducted while the software and systems are live.
  • Create test localities to pre-test any upgrades to assess the impact on performance and service prior to deployment. This minimizes disruption to the broader audience.
  • Recruit, alert, and reward those who assist with these tests.
• It is possible to conduct an unsophisticated Alpha Test, but this can’t be risked in Beta Tests.
  • Alpha testing is usually conducted as an internal exercise and lasts until all of the bugs have been identified and worked out.
  • The Beta test is then planned, with a known number of sites or users.
• Concerning IP Protection:
  • Threats will come from two sources:
    • The iTunes types who may perceive the new offering as a threat to their markets – ones with deep pockets to keep the company busy defending its legal position.
    • International teams who rapidly clone any new technology that they find for a variety of motives. These groups tend to work from locales where IP protection is difficult to impossible.
  • IP is not secure until tested in courts. Often this involves the most innocuous aspects of the IP or software offering. In addition, big players may seek injunctions to halt service until courts resolve claimed IP conflicts.

[like]