news@sun.UUCP (07/09/87)
[There's been a some interest in my thoughts on setting up a Beta Test "service" with USENET. This article is an offshoot of that -- I got into a discussion of beta testing and how to run it with a Mac software publisher on the net, and decided it was of general interest, so I cleaned it up and am now posting it for those folks who can use this kind of information -- chuq] >It's always difficult finding *good*, reliable beta testers. We have >had problems in this area. Our product is a good example. Looking >back at it now, it seems to me that a large number of the "beta-testers" >signed up to test the product were really only interested in getting a copy >of the program for their own use, way before any one else would get it. >Several individuals *did* do a good job of reporting bugs, giving >detailed descriptions and follow-ups. But this was a *significant* minority! >Most we didn't even hear from! >Another problem we have run into is the blatant disregard for a product's >copyright. I know that a *great many* copies of our product out there >today are bootleg "beta" copies! >The result of all this? I'm justifiably wary of people purporting to >be "beta-testers". This is not to say that it can't work - just that it >must be well organized and well supported. I am (in case you didn't know) among other things a professsional Beta Tester -- before a new release of Unix goes to Sun customers, I get to try to break it first. There are really two types of Beta testers, the technical folks, whose job is to give feedback and help solidify the product, and the special customers, who get early releases to gripe about. Do you just hand out software and tell people to give you feedback? You're asking for problems. Here is generally what I would suggest to anyone doing a Beta Test. I could probably be convinced to help someone design a Beta Test on a consulting basis if they really want me to, but this should help folks avoid most of the pitfalls.. o Decide how many beta testers will be enough. And stick to it. You'd be surprised how many extra copies go out sometimes. Then again, maybe not. o If your marketing folks insist, let the marketing folks support them, and stick only to the technical sites. The others generally aren't worth the time spent poring over their reports. o If you need more than 10 beta sites to get coverage of the product, you're choosing the wrong beta sites. o Build up a prospective list of your customers and whoever else you know that might qualify. Pass that list around the company and get feedback, especially from the developers, and support groups. If anyone on the list is flagged as a problem customer, get them off the list (this means if ONE person flags them -- blackballs are appropriate). You want customers who are working with and for you -- and only those people. o Interview them. Get them to explain why they're qualified, how much time they plan on dedicating, and what they hope to accomplish. o The folks left over are the candidates. They get to sign non-disclosures. They can't talk about the beta software. They can't admit to having it. They can't talk about it. They can't write reviews about the pre-release software. [you do NOT want to have happen to you what happened to Microsoft with Word 3.0 -- all those authors who said "neat toy!" and are now looking silly are overreacting and saying "horrible garbage!" -- primarily because the authors blew it and are trying to save face. Word was neither as good as they say nor as bad as they are trying to make it -- what really got screwed up was a lack of journalistic professionalism throughout the magazines. Unfortunately, Microsoft is getting the brunt of it because it seems the reporters aren't willing to admit they blew it and are blaming Microsoft for many of the self-generated problems Did you, by the way, see what MacUser did to Pagemaker 2.0? Their pre-release software came with a "no review" clause, so they didn't review it -- they simply talked about the new features and how they planned on reviewing it when they got the real software. That's borderline legal, depending on the wording of the contract -- but it is still slimy. This is a great example of why this needs to be in writing, so that people don't get cute on you. And why they can't talk about it as well as review it. It may sound hardnose, but there are folks who've proven that hardnose is necessary. It is also the major factor in why I'm nor renewing my subscription -- if they've gotten this bad in their reporting, I don't need them.] They can't give it to friends. They are there to test software. If they aren't willing to put that in writing, they shouldn't be Beta sites. And make sure it is in writing, and that they understand, that if you find out they've broken the contract, you are going to have them ship back the software. Have your lawyer write up a Beta Test contract that spells out the rights and responsbilities clearly. I can't stress this step enough -- words and a handshake DO NOT WORK. o Part of the beta agreement is an agreement to do a weekly written report. Every beta tester will send in a weekly report of problems and comments, or a piece of paper that says "nothing sigificant found" or full of recipes or something. They should also have a contact in the company to call as soon as a problem is found, but the written report is backup that something didn't get dropped on the floor. It is also a good check that the beta tester is beta testing. If they miss two reports, pull the software or get a good reason. This sounds hardnosed, but the practical fact is that most beta sites are useless, either because the wrong testers were chosen for the wrong reasons, the information returned was ignored, lost, or generally discounted, the information wasn't returned, or the schedules are unrealistic. A few well known things to avoid in beta testing: o do not beta test until all of the software is ready. Trying to do development AND bug fixing generally doesn't work. There is a good purpose for an alpha (pre-beta) test, but it needs to be handled differently (and accepted by all concerned as different). o do not beta test until the manuals are written. Don't try to shorten schedules by testing the software separately -- the manuals tend to need at least as much beta testing as the software. o Do not scrimp on testing time. Anyone who spends a year developing a piece of software should realize that it can't be tested in four weeks. There is a definite tradeoff between exhaustive testing and getting to market, but if you try to make up a late schedule (which implies hurried development) by cutting testing, you might as well slit your throat. Beta testers need time to install the software, read the manuals, get familiar with the software, and really start beating on it. Figure 8 weeks, minimum. Hopefully 12. Let me re-emphasize this. If the development schedule slips a month, and you try to make up the time by cutting the length of the Beta Test, you will live to regret it. I would even suggest that the later the product, the longer the Beta Test, but I would probably give everyone in Marketing a heart attack. In practical terms, you have a choice -- you can either test it before shipping it, or let your customers test it for you -- and expect to have to do an emergency second release shortly after. That's expensive -- if you don't believe me, ask Microsoft what Word 3.1 is going to cost them. o Developers can not test. The people who write the software can not successfully test a product. While this may sound illogical, they are too familiar with the product, and know how it is supposed to work. It is generally the places where the developer didn't think about using it that things get a bit creaky -- and if the developer had thought about it, he would have programmed for it. o finally, get a complete novice. If the product is designed for the general marketplace, get a complete novice and let them free. call Kelly and hire a temp, put them in front of a machine with a manual and watch. If they get confused, your customers will, too. Is the program confusing, the manual incomplete, do you need a (better) tutorial? Give the person a week of completely unstructured test and see what happens -- that is the best view of what will happen once you ship that I can think of. chuq Chuq Von Rospach chuq@sun.COM Delphi: CHUQ Touch Not the Cat Bot a Glove -- MacIntosh Clan Motto
billc@blia.BLI.COM (Bill Coffin) (07/10/87)
We had a problem with Beta software proliferating uncontrollably.
Here's something that helped a lot: Set up all pre-release
software to print a banner on startup. The banner should say
a lot of intimidating things like "Beta Pre-release" and/or
"for TEST only". Remove the banner (or make it prettier) when
doing the final release.
A lot of people that just want the software are not willing
to have the end users see messages like these, and will wait
for the real product. (Dedicated pirates can always find and
remove the banner (unless you go to absurd lengths) but 99% of
the bozos will ease off.) I recommend this technique very highly.
--
W.H.Coffin. billc@blia.BLI.COM (ucbvax!{mtxinu|ucsfcgl}!blia!billc)
>> the usual disclaimer about my employer and my wretched opinions. <<
>> the usual witticisms that swell netnews to ridiculous proportions. <<biff@nuchat.UUCP (Brad Daniels) (07/15/87)
In article <2937@blia.BLI.COM>, billc@blia.BLI.COM (Bill Coffin) writes: > Here's something that helped a lot: Set up all pre-release > software to print a banner on startup. The banner should say > a lot of intimidating things like "Beta Pre-release" and/or > "for TEST only". Remove the banner (or make it prettier) when > doing the final release. This will work fine for commercial sites, but I don't think they're the problem in most cases. If you're producing PC software, you have to worry about the weekend pirates. I don't know about now, but when I was in high school, it was considered quite a coup to get software that said "Beta Pre-Release" all over it..... - Brad -- Brad Daniels ...!soma!eyeball!biff Now that I have my own account, biff@tethys.rice.edu I don't NEED a disclaimer. ...!uhnix1!nuchat!biff