Practical ideas for beta tests in the small business

The value of listening

As the last stage of testing a product, beta tests usually involve sending the product to customers outside the company for real-world exposure. From a marketing standpoint, listening to your customer is one of the most important aspects of doing business. Beta tests are one of the most effective ways to listen.

Quality does not happen by accident; it has to be planned.

Beta testing saves you money:

  • Reduces redundant testing.
  • Reduces redundant support calls.
  • Provides a thorough review of your documents or online Help.
  • Can provide free advertising for your product.
  • Reduces the risk of interim releases.

A good measure of the value of a beta test is the quantity and quality of the feedback that the beta testers provide.

Consider what not testing your product might cost. Neglecting to test can cost much more than the beta test itself. Current studies suggest that a single technical support call costs a minimum of $20; finding a single defect that generates five calls is worth $100.

General guidelines for effective beta tests

  • The best beta test simulates the actual release of a product. A beta test is the final rehearsal before releasing your baby into the world.
  • Your installation and removal routines should be rock solid.
  • Perception equals reality. Listen to all your beta testers, even the grumpy ones.
  • A beta test is most effective if it is conducted with target customers, not in a lab environment. An internal quality test, using company employees, usually discovers more bugs than the average external beta tester and is most useful for quality testing. However, an internal test is not a substitute for the external beta test. External beta tests provide a real-world scenario for the actual release of your product as the product is tested on a wider variety of systems.
  • A public beta is a beta test in which a general call is posted to a public audience indicating that anyone can participate if they download the beta product. These types of beta tests usually involve a larger number of beta testers. You might consider not conducting a public beta for the first release of a product. Public betas are most effective for revisions of a product or products derived from an established product.
  • You can get the most results out of usability testing if you do it early in the development process, before beta testing. However, you might consider building 1-2 usability tests into your beta test workflows.
  • Small companies sometimes benefit from starting a beta test early in the development process to assist in verifying issues. This approach works best with more technical products that have been previously released and are relatively stable. However, you might need to make a special effort to maintain beta tester incentives and keep your beta testers interested. Note that this type of beta does not fit the usual recommended process of a beta test.
  • For the typical beta test that is conducted right before the final release, consider fixing every known bug before beginning the beta test. A known bug can chew up time from several different folks during your beta test. If a developer fixes a known problem before the beta test, fixing the bug costs only the time of one person. During beta testing, the time spent on the one bug is multiplied many times by every person who gets involved in the reporting, testing, and fixing.
  • For small businesses, getting good feedback about all aspects of the product can be one of the biggest challenges. Testers tend to report on features that they requested. You can try giving out incentives, such as a free released product, for consistent, quality feedback on all aspects of the product to motivate your testers. For more information, see Designing incentives.

A practical process

Pre-planning

As a small business owner, you might want to consider the following needs that are not necessarily included in your written plan.

  • People If you are the owner, you are probably the beta test administrator.
  • Equipment You will need different systems to test your product and duplicate issues that testers report. You probably already have these, so this cost is usually negligible. You also might need to have additional storage for product builds, test materials, and data reported from the test. You might possibly need to expand your web hosting resources to accommodate distributing and gathering beta test information.
  • Space In a small company, you might not need a separate lab. However, you will probably need an adequate area for storage: shipping, packaging, sample product, test incentives, and additional hardware.
  • Cost
    • Shipping of software or hardware.
    • Incentives to offer beta testers, such as a free product, caps, mugs, and t-shirts.
    • Baseline value of your product, including cost of manuals, packaging, and so forth, if these apply to your situation.
    • Cost implementing and maintaining your beta web site with bulletin boards and so forth.
    • Possible cost of legal assistance in providing non-disclosure agreements (NDAs) and enforcing them. This cost is rarely needed.

Step 1: Write the plan

  • The plan is good information to communicate to your beta testers. The plan can contain much overview information that they might need. You can include much of the information from your plan in the beta test package that you send the testers.

Part 1: Summary

  • Write one paragraph that provides enough information for a beta tester to know what product is being tested, when, and what you hope to achieve in the test.

Part 2: Goals

  • The goals are the most critical part of the test plan. They can be specific, yet flexible.
  • Include the following types of goals:
    • What data the test will attempt to collect.
    • What parts of the product must be assessed.
    • What specific tasks need to be achieved—these are the key functions and tasks that must be evaluated.
    • Platforms and configurations to test.
  • As you write, consider the following examples of goals:
    • Test the technical accuracy and usability of the documentation.
    • Test for operation on the supported operating systems.
  • Make sure that you include specific workflows that you want beta testers to work through:
    • Include a test that compares your product with competitor products on the same computer. Some of your beta testers might own competing products.
    • Have someone test registering the product, obtaining technical support, and using your web site.
    • Consider including at least one or two usability tests, if at all possible.
    • Consider requesting that a tester check the interface for spelling, appearance, and usability.
    • Ask several testers to review and test the documentation or Help file.
    • Ask testers to review user preference settings and customizable options. Consider placing tremendous emphasis on testing these features.

Part 3: Specific requests

  • Include specific test goals or testing of new features requested by customers, sales affiliates, and so forth.

Part 4: Budget and materials

  • Think through the logistics of your beta test and do some future planning. You might decide at this point you cannot afford certain things.
  • Do not go into depth. Just list common associated costs and what you need to budget.
  • Include things like hardware for testing. You can leave out incidentals such as disks or packaging.
  • Include the cost of a bug tracking system that meets your needs, if you do not already own one. If you have more than five beta testers, you probably need something more than a spreadsheet. For more information, see Step 2: Gather your test materials and set up your reporting system. For the purposes of this document, a bug tracking system stores data about both issues and bugs.

Part 5: Number of beta testers

Part 6: Reporting information

  • Describe how beta testers will report issues and bugs.
  • Describe how you will report the results of the test.
  • Include how beta testers can reach technical support.

Part 7: Participant profile design

  • Describe the intended audience for the product. Include specific items that focus on the demographic that you want to focus on for the product.
  • Describe some of the general technical criteria that you will use for selecting your beta testers, such as the operating system they must own.

Part 8: Communication

  • Describe a process for your development team to get information to and from testers. Give general guidelines and provide phone numbers, web addresses, and so forth.

Part 9: Schedule

  • This is the fictitious part of the plan. :)
  • One possible way to determine how long the beta test might take is to consider how long you would take to fully evaluate your product and then multiply that figure by three.
  • Include a general timeline including tentative distribution dates, test performance, and closure dates.
  • Include an Internet address for testers to obtain an updated schedule during the beta test.

Part 10: Closure

  • Describe how you will distribute incentives to the testers.
  • Describe how you will gather test material, archive data, and distribute test result information to marketing, development, and other folks, if applicable to your company.
  • Consider adding a statement that the document is subject to change and state where or how updates will be distributed.

Step 2: Gather your test materials and set up your reporting system

  • You might get updates to some test materials after you initially gather them or add test materials later. This means that you need to set up a tracking system for your test materials. Use a simple numbering system and tie a description or version to that number.
  • As you give out test materials, such as test files, track which beta tester has the materials. If you have more than ten beta testers, you might consider using a web-based form for testers to enter information before they download materials. Otherwise, keep track of the location with a spreadsheet. This information helps you later in resolving issues and bugs.
  • One of the greatest challenges is getting feedback from your testers. Try to use the simplest methods available for beta testers returning comments, either email or a web form, which can be fed into a bug tracking system or database.

Reporting system

  • Decide how you want to track your bugs and issues and set up that system. Most bug tracking systems fall into the following categories:
    • Spreadsheet—The spreadsheet includes columns such as priority, status, assignee and resolution. You are limited to one user at a time.
    • Microsoft® Access database—This solution is slightly better than a spreadsheet. You have a benefit in being able to add an interface for data input. This solution does not extend to cross-platform development and has the added constraint of requiring all users to run Microsoft database run-time files.
    • Home-grown utility—You can get a solution totally customized to your needs—at the price of your time.
    • Third party software—You might want to consider the following items in selecting software.
      • Do you need to run the server on multiple platforms or a specific operating system?
      • Should the system be web-based or fat client? Do you need to support clients on multiple platforms? Do you want to install clients everywhere access is required? Do you need to support remote developers or customers?
      • Do you have the expertise for a database or does a file-based repository better suit your needs?
      • Can access to the bug tracking system be controlled to the level that you need, such as on a user, module, or bug basis?
      • Does the customizable reference data, such as priority and status, meet your needs? Does the software allow additional fields for specific attributes?
      • Does the software allow you to search on criteria, such as status or priority?
      • Does the software allow the reports to be exported to other software, such as Microsoft Excel, for more analysis?
      • Can users be notified by email when an issue or bug that they reported is changed? Can the format be customized?
      • Does the software license allow for an unlimited number of users, bugs, issues, or projects?
      • Does the software allow an audit trail of all changes to a bug?
      • Is integration with your SCM system, for source code control, such as SourceSafe, available?
      • Is integration with your IDE, such as Netbeans/Forte, Eclipse, or Jbuilder, available?
      • Is the input of non-ASCII characters, such as accented characters and multi-byte characters, for Kanji-based languages, supported? Can the user interface be displayed in languages other than English?
      • Can the bug tracking system run out of the box or does the system require significant customization?
      • Can users attach files to their reports?
  • There are many bug tracking products available. You can find a comparison list here: http://www.aptest.com/resources.html#bug. You can also find free tracking systems:

Custom settings in your reporting system

  • Consider assigning an identifier number to each type of system that the beta testers own. This can help make investigating bugs and issues go more quickly. The identifier also saves your beta testers from entering information every time they report an issue.
  • Consider assigning settings that determine whether a product can ship with the reported data. The development team can decide the level during the beta test:
  • High The issue causes crashes, loses data, or corrupts or damages other products.
  • Moderate A workaround is available.
  • Low The issue involves cosmetic changes.
  • Consider creating a four tier system in your bug tracking system that allows you to determine how much time to spend on an issue or bug.

    Tier one This tier includes critical bugs that cannot be overlooked and that can severely impact product’s release or company reputation. A small percentage of test results fall into this tier. Bugs in this tier require immediate communication to the development team.

    Tier two This tier includes bugs that should be reviewed by the development team.

    Tier three This tier includes issues that concern repetitive data or items of debatable importance. These issues are often exaggerated suggestions from testers, extraordinary circumstances, or unexpected claims.

    Tier four This tier includes issues with little or no significance. This group should be reviewed after beta test is closed.

  • Consider giving each bug some type of closure setting in your bug tracking software:
    • Fixed.
    • Work-around.
    • Postpone.
    • Investigation.
    • Consideration—Less serious than a bug that warrants investigation.
    • No action—Do not designate this until the end of the beta test cycle. In the explanation field for the issue, phrase the wording in a manner that shows that, while you have reviewed and investigated the issue, you do not see a viable way to fix the issue at this time.

Step 3: Select your beta testers

  • Selecting beta testers can take awhile, possibly a month or more. Try to get quality testers, not a large quantity of testers.
  • The least desirable beta testers are potential sales leads, customers who never complain, friends, or random people.
  • The best beta testers are sales affiliates or customers who request additional features. The ideal candidates are professional consultants, who make their living giving recommendations about products and services.

Determine how many beta testers that you need

  • Consider the following factors when determining the number of beta testers that you need:
    • Copies of the software, with registration codes.
    • Number of people—Consider the number of testers that you think you, or your beta test team, can support and respond to effectively.
    • Product type—A simple driver might need many testers; a complex product might need fewer testers.
    • Complexity of the tester profile—A complex profile might mean testers are harder to locate.
    • Duration of the beta test—The longer the test, the less likely you can sustain interest, so you need a larger number of testers. If the test is short, a large number of folks consume too much time and you cannot address their issues.
    • Recommendations of folks who work with you.

Create an application form for beta testers.

  • To speed up the selection of applicants, you can create an application form for each test applicant to fill out. You can implement the application through email or a web-based form.
  • Consider including the following sections in the application form:
    • Contact information Include email, street address, and phone number.
    • Profile Include the occupation, company, interests. This information is invaluable for marketing your product.
    • System information This information is important. You might consider getting an inexpensive utility that allows users to get this information and send it to you. If applicable, request information about whether the applicant owns third-party applications that must be installed to use your product.
    • Prerequisites to using product Put this part in a different, emphasized section of your application and make sure applicants understand they must have this type of system to use your software. This saves you time later in reviewing applications. You can eliminate applicants who do not have the equipment to use your product.
    • Privacy statement Make sure that you tell applicants that the information is private and only for the beta test and for use within the company.

Get your legal documents

  • Depending on your situation, consider whether you need the following documents and make these available to applicants to sign during the selection process:
    • Non-disclosure agreement (NDA) This document protects your test results and gives you a way to enforce secrecy. When testers sign this document, they are agreeing to stay quiet about the testing experience. They must remain quiet about everything: the product, documentation, and emails that are transmitted. This confidentiality is important because any word of failures could affect product sales before the product gets to market. The NDA specifies a period of time that information cannot be shared.
    • Software license agreement You should have this, even if your software is protected from copying or pirating. This is a legal agreement between the company and the tester. The license agreement overlaps slightly with an NDA and can be combined with one.
    • Test contract Rarely used, this document specifies the rules of the test. A lawyer usually prepares the document. This document obligates a beta tester to participate. It outlines the repercussion of not participating in a test, test duration, ownership of the beta product when the test is complete, and warranty or servicing requirements.

Call for applicants

  • Make sure your first contact with the applicant, such as a posting to a newsgroup, includes the following information:
    • Timeframe of the test.
    • Product description.
    • System requirements, like processor speed.
    • List the most important requirement first. Describe each requirement with enough details that only qualified people respond. However, try to keep the list short so that you do not discourage people from applying.
    • How to apply, such as going to a web site.
    • Benefits of participating in the beta test.

      • For example, beta test participants can see the product before anyone else. Be sure to point out that testers are volunteering and there is no charge for participation. Describe the product benefits, but try not to make promises or list incentives in this section.

    • Your contact information.

Select the applicants for the beta test.

  • After collecting enough applications, you can narrow down the number of candidates to about twice the number of beta testers you want. For example, 30 final testers = 60 applicants.
  • To narrow down the applications to twice the number of final beta testers, consider the following guidelines:
    • Look at applicants’ responses. First, consider those who respond immediately selling themselves and their skills. Secondly, consider the applicants who send a short, but specific, message of interest. Remove candidates who respond with a single word.
    • Set up 3-5 criteria that every candidate must have to participate. Do not throw away applications as these might be useful for other tests.
    • Consider demographics. You might want to select a wide variety of applicants.
    • Candidates with the most experience with your product should be your first consideration.
  • Narrow down to the best quality testers
  • When contacting the final candidates, use calls and emails in a three phase approach to eliminate candidates:
    • Phase 1 In an email, state clearly that you are still considering candidates, but have not selected the final ones. Ask for responders to express further interest. Give priority to first responders. Eliminate 10-20 % of the applicants at this point.
    • Phase 2 Send out a message that candidates must complete the NDA by a specified date or they will be dropped from consideration. Give priority to first responders. Notify remaining candidates that they are not selected for the test—offer to put them on a later test or some other reward for their response.
    • Phase 3 Select your final candidates by ranking them according to the following criteria:
      1. Test parameters.
      2. Qualifications.
      3. Responsiveness.
      4. Communication.
      5. NDA completed.

For more information about NDAs, see Get your legal documents.

 

Notify the candidates

  • This letter is a simple letter notifying the candidate that they have been selected. Consider keeping it short. Thank the testers for their time and let them know they will be receiving a beta test package in the near future.

Step 4: Establish your test parameters

  • Test parameters are test guidelines given to the actual beta test participants. These guidelines give structure to the test. These can be a complex series of tests or a simple framework for test progression. The test parameters can include the goals from Part Two of your beta test plan. For more information, see Part 2: Goals.
  • These parameters might change many times prior to beginning a beta test. Just stay focused on the key objectives of your product and build from there.

Step 5: Determine whether the product is ready to beta test

Establish that the product is ready.

  • As you are evaluating your product, create a checklist of key features. If any of the key features are not working, the product probably does not need to go into beta test. These are guidelines, not pass or fail criteria.
  • Evaluate each feature that is not working as expected based on these points:
    • How likely is it for a beta tester to see the issue?
    • Will the customer experience with the product be negative based on the product’s current state? What is the potential impact the problem will have on the tester’s computer? Your guideline is to protect the customer.
    • Will the product perform without these features? Will the beta test be effective without these features?
    • How long will it take to fix the problem?
  • Try to fix known problems before the beta test. The best outside beta testers cannot match the defect counts for internal testing. Each defect that an outside beta tester finds costs plenty of time to your organization. A few defects can make that time quickly turn into staff-months.
  • Try not to send a product whose feature set is not complete. You will get remarks on the missing features. Secondly, after you add the feature, it will get more testing than the rest of the product.
  • Review your installation program carefully. Both your installation and removal, or un-installation, routines have to be rock-solid. Otherwise, later beta versions will not be installed on clean systems. If a program cannot be installed at all, you will lose an entire beta cycle for each tester affected by the problem. If the product seems to install correctly, but leaves out something crucial, testers will report problems that do not really exist in the core product. Finally, if the product does not uninstall correctly, later beta cycles may fail to identify problems with missing files or configuration settings.
  • Make sure that you have included adequate documentation on how to use the product, such as a Help file. The Help file does not have to be perfect as beta testers can give excellent reviews of the documentation to make it more accurate. A Help file is also a lower cost item than a printed manual.
  • Set up your distribution site, if you have not already done so.

Conduct a last-minute review

  • This review should focus on verifying that known bugs are corrected and that new features work as expected. This review does not focus on evaluating the entire product, which is time-consuming.
  • Double-check your files to ensure that you are distributing the correct build, modules, and accompanying files.
  • Ensure that your product is free of viruses.
  • Make sure you have included third-party components and documentation in the test materials.

Step 6: Make sure your legal documents are in order

  • If you have not already done so, make sure that all participants have signed and returned the necessary legal documents, such as the NDA. For more information, see Get your legal documents.

Step 7: Distribute your test materials and beta test package

  • Consider including the following items in the beta test package:
    • Welcome letter on company stationary—This is a nice touch. The welcome letter can include three parts:
      1. Paragraph welcoming the tester to the program and indicating that they have been specially selected to participate.
      2. Simple explanation of the product.
      3. Paragraph thanking the tester for participating.
    • Parameters document—Specify the following items:
      • Tasks needed to be performed during the test.
      • Basic timeline of events.
      • Contact information and web site locations.
      • General guidance on the type of feedback you want.
      • Details about incentives.
    • Test package—Include the following:
      • Manuals.
      • Third party software.
      • A return mailing label, to be used when you close the test.
      • Hardware accessories, if needed.
      • The product.
  • Consider sending the test package on CD because of the following:
      • A download reduces the perceived value of the product; a CD seems more valuable.
      • Sending a master on CD possibly reduces the size of needed upgrades.
      • A CD serves as a backup, in case the product becomes corrupted during the beta process.
  • You can find a good example of a beta test package at the following link: www.epri.com/eprisoftware/processguide/docs/betapkg.doc.
  • You might consider sending the initial beta test package by regular mail. Testers are possibly more likely to read printed documents more thoroughly. Not all of your users have a printer to print the documents that you send or a CD-writer to burn backups of your software on CD. Testers also might not read all the attachments that you post on your web site or send as an attachment in email.
  • Keep a log of every component distributed during the test in a simple spreadsheet or database. This helps you later in the beta process when trying to reproduce a bug or issue that your testers report.

Step 8: Manage your testers

Communication

  • At the beginning of the beta test, ask your beta testers to guard the privacy of the test product, through the NDA and through a reminder email. In the reminder, you can mention the following guidelines:
    • Do not mention unreleased product names in public.
    • Do not share test results, even casually, to anyone.
    • Do not mention schedules, timing, or anticipated release dates of any project.
    • Do not show a beta product to anyone outside the beta test program before or during the beta test.
    • Do not note to the press, public sites, bulletin boards, or chat rooms anything about past, future, or current tests.
    • Interactive communication, such as the phone or instant messaging, is more time consuming and can produce inconsistent communication about test results. However, for emergency needs, you can let your testers know how to call you or use instant messaging to reach you.
  • You need a written record of most communication. Consider a web-based bug tracking system in which testers can report bugs and also follow up on the report. For more information, see Step 2: Gather your test materials and set up your reporting system.
  • Do not send too much work up front and overwhelm your testers. Too many emails, documents, files, and test components might make them back away from testing.
  • Establish a forum for your beta testers on your web site, if at all possible. Do not use the forum for bug reporting. Just use the forum for discussing issues and workarounds.
  • Get detailed system data on their test platforms once, if you did not do this during the phase in which you selected your beta testers. For more information, see Create an application form for beta testers. Assign that system an identifier in your bug tracking system. Try to make sure that the tester includes the identifier in bug reports.
  • Respond quickly to your testers. They know they are important if you do. When you want to thank a tester, pick up the phone to do it. The tester will love you.
  • In most cases, you can ask your testers to use the product at least an hour a day, depending on the situation and incentives. In some cases, the tester will give much more than that.

Designing incentives

  • Incentives are ways to keep your beta testers motivated. They can include a free product, caps, t-shirts, mugs, and so forth.
  • Let testers know the criteria for earning the incentive. However, you might not want to say the testers must send 20 email messages during the test to be eligible. Rather, you might indicate that the testers must give constant and consistent feedback during the test period to be eligible.
  • Ensure that you do not make the incentives too difficult to obtain; beta testers might lose interest. You can possibly help keep tester interest high by posting reports to your web site that show which testers have reported bugs and whether the bugs have been fixed.
  • Deliver the incentive immediately upon completion of the beta test or when the tester meets the criteria for the incentive. Testers expect their reward immediately upon completing the test.
  • Consider substantial monetary awards for the top three providers of useful reports. Current studies suggest that a single technical support call costs a minimum of $20; finding a single defect that generates five calls is worth $100.
  • One product owner has suggested that, to ensure testers are actually using the product, you can insert a bug or message box that comes up during what would be a standard run through the product. The message or bug can have a codeword. Beta testers who return the codeword win a free license of the released product. If someone is interested enough to track down the code, then they have probably given the product a reasonable run through. If they have found any problems, they will probably let you know.

Identifying problem testers

  • If you notice someone is not being responsive, contact them immediately and remind them of their commitment.
  • Try to identify non-responsive testers as early as possible. Try to get them to meet the requirements of the beta test—or leave the test.
  • If a beta tester leaves the test, ask for your materials and software back. A poor product out there now might cause a public relations problem later by giving a poor impression of your product. In most cases, you cannot prevent a person who is attempting to defraud a company out of a product. You can only send a certified letter telling the tester they must return the product.

Step 9: Act on the test results

Communication

  • If you have a beta test team comprised of employees who communicate with your beta testers, set up communication standards. For example, a response to an initial test issue report from a beta tester can have a 24-hr. turnaround time.
  • Respond to your testers continuously and repetitively to ensure they know their feedback is important. Indicate to the beta tester that you are reviewing their feedback, not just that you received it.
  • Whenever a tester reports an issue, respond until the issue is resolved and marked with a closure setting in your bug tracking system. For more information about closure settings, see Custom settings in your reporting system.
  • Review the list of goals in your beta test plan and evaluate whether these goals are being tested. For more information, see Part 2: Goals.

Managing the test product during beta testing

  • Whenever possible, no more than one version of the product should be in circulation. Tracking builds and versions is critical to your plan. Use a spreadsheet or database to log what builds have entered test to allow the beta test team to track the history of an application through the test process.
  • Try not to update the product too often or not at all. Try to maintain a balance.
  • Consider the following levels of complexity in testing the software and ensure your beta testers have the necessary information for handling the following items:
    • Minor recent changes to the code.
    • Upgrades and patches, fixes.
    • Special issues affecting the operating system.
    • Product interaction with other software.

Step 10: Qualify the test issues as they arrive

  • You should try to determine the level of importance of each issue as soon as it arrives. This process saves time in handling the report. You can decide whether to route the problem to the developer or address it in the documentation.
  • Try to duplicate the issue as soon as possible.
  • When investigating the issue, consider the following:
    • Impact on product—Determine whether a product can ship with the reported data; involve development team in this one. Mark the following issues in your bug tracking system:
      • High—The issue causes crashes, loses data, or corrupts or damages other products.
      • Moderate—A workaround is available.
      • Low—The issue involves cosmetic changes.
    • Frequency—This is a measure of how often or how many beta testers report an issue. A high frequency means that the issue is severe and needs attention.
    • Experience or competence of the tester—Interview the tester and try to determine if the issue is the result of a lack of knowledge or unreasonable expectations on the part of the tester.
    • Interaction of hardware and software—You can find a list of common bugs and fixes to them at www.bugnet.com.
  • When folks see blunt, uncensored comments of a beta tester, the tendency is to ignore the comments. Do not ignore these comments as they often have some basis in fact.

Step 11: Evaluate the test data

  • When you evaluate the test data, you investigate the reasons that issue are occurring. At some point, you might escalate an issue to the status of a bug. A bug is anything that potentially has a negative impact on a customer’s experience with the product. An issue is unexpected or abnormal. A bug is more serious. If an issue is reported many times, it can become a bug and must be addressed.
  • Beta bugs can sometimes be vague and hard to quantify. Your technical criteria for a bug can vary. However, you must decide what an issue is and what is a bug and respond accordingly. Try the following process for evaluating the issue:
    1. Verify your data. Get as much detail from tester as possible.
    2. Check the frequency. Look for how many people are reporting this issue and how often they are experiencing it.
    3. Follow through. Make several attempts to duplicate the issue. If you cannot duplicate it, you should contact the tester, explain the situation, and emphasize the importance of their feedback.
    4. Periodically review your bug reports. Ensure that beta testers are following up to see if their bugs are fixed. They can indicate this in your bug tracking system.

Step 12: Close the test

“Testing never ends, it just stops.”

  • Tell beta testers a specific closure date only when you are actually ready to close the beta test. Ask the beta testers to return your software and test materials, using the shipping label you included in your beta test packet.
  • Distribute your incentives immediately.
  • Send a personal thank you note from the development team to each beta tester.
  • Write a closure document to refer to later when trying to make marketing decisions about your product. Write out what you want to remember, which might include the following items:
    • Lessons learned and the results of the beta test.
    • A project summary that includes the following:
      • Number of participants.
      • Duration of the test.
      • Description of the product.
      • Items tested.
      • Numbers about quantity of feedback gathered during the test.
    • List of the participants to use in future beta tests.
    • Description of the demographics that you tested.
    • Summary of the important issues. This section is difficult to write, but can be the most important part of the report.
    • Marketing summary that includes positive comments that came out during the test. This can be a list of quotes or customer success stories.
    • Material summary, which is a list of what materials were distributed during the test. You will need to know this when looking at test results. The material summary includes build dates, build numbers, and so forth. In addition to the software, you can document every manual, disk, and so forth. You hopefully were keeping a log of this during the beta test.
    • Statistical summary including the following:
      • Number of beta testers planned.
      • Number of testers who provided feedback.
      • Number of beta test bugs reported.
      • Total number of beta test bugs unresolved.
      • Severity levels of unresolved bugs, such as the number of minor, severe, and critical defects logged in your bug tracking system.
      • Number of suggested enhancements not incorporated in this version.

Step 13: Archive the test data

  • Keep your beta project summary and list of marketing suggestions. If applicable, distribute this information to your developers, marketing folks, sales people, and so forth.
  • Keep a list of the tests and workflows used, along with the files for the test product, of course.

One last suggestion

If you are going to manage a beta test program, consider participating in other beta test programs of small companies and see how they run their beta tests. You can find out about other beta tests at the following web site: www.betawatch.com.

Useful reading

www.epri.com/eprisoftware/processguide/docs/betapkg.doc

This link displays a good example of a beta test package that contains material to give your beta testers. This package includes a feedback form intended to be filled out manually. However, you can use this feedback form as the basis for web-based forms you design for your beta testers.

Testing Applications on the Web

By Hung Q. Nguyen, 2001.

Every tester can benefit from this book, and, if you are working in the Microsoft Windows environment, this book is more valuable. This book gives a lot of information that is useful for every tester.

Beta Testing for Better Software

By Michael R. Fine, 2002.

This book describes all nuts and bolts of running a beta program. The book is oriented towards managers in medium to large size businesses, but the book still has practical information for a small or one-person software businesses.

Lessons Learned in Software Testing

By Cem Kaner, James Bach, Bret Pettichord, 2002.

This book has 293 suggestions that can be used in everyday testing routines. Each lesson can be more or less useful or obvious, depending on your experience. Some of these lessons are very practical, others more theoretical.

Software Testing and Continuous Quality Improvement

By William E. Lewis, 2000.

This book describes the complete testing of a product from beginning to end.

Handbook of Usability Testing

By Jeffrey Rubin, 1994.

This book was published ten years ago, but still offers great advice for watching and evaluating how people use your product.

A Practical Guide to Usability Testing

By Joseph S. Dumas and Janice C. Redish, 1994.

This book is a great guide to the study of human factors and human interaction and how they can impact your product design.