Changes between Version 5 and Version 6 of BetaTest


Ignore:
Timestamp:
06/23/14 10:45:25 (3 years ago)
Author:
davea
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • BetaTest

    v5 v6  
    1 = Beta-test applications =
     1[[PageOutline]]
     2= Using volunteers to test applications =
    23
    3 It's important to test new applications on a wide range of hosts, since bugs may appear only with particular OS versions, memory sizes, display types, usage patterns, and so on. It's handy to use volunteers to do this testing, since they provide the needed diversity of hosts. One way to implement this is to create a separate test project. This has two disadvantages:
     4Testing new apps and app versions has two phases
    45
     6 * making sure they doesn't crash;
     7 * making sure they computes the correct results.
     8
     9It's important to test new applications on a wide range of hosts,
     10since bugs may appear only with particular OS versions, memory sizes, usage patterns, and so on.
     11It's handy to use volunteers to do this testing, since they provide the needed diversity of hosts.
     12
     13== Beta-test project ==
     14
     15One way to implement this is to create a separate test project,
     16attach your own hosts to it, and get a limited number of volunteers to attach to it.
     17
     18This is useful, but it has two limitations:
    519
    620 * There is overhead in creating and maintaining a separate project.
    7  * The credit accrued by testers goes to a different project.
     21 * The credit earned by volunteer testers goes to a different project.
    822
    9 BOINC provides a way to do beta testing in the context of your existing project. You can let users volunteer to run test applications, warning them in advance that these applications are more likely to crash. These users will get a mixture of regular and test results, and they'll get credit for both. Here's how to do it:
    10  * Upgrade to the BOINC server software of Oct 25 2006 or later.
    11  * Create a new [AppVersion application] using [XaddTool xadd]. Include `<beta>1</beta>` in the `<app>` element to designate it as a beta-test application.
    12  * Add a validator and assimilator for the test application.
    13  * Include the line
     23BOINC provides two ways to do beta testing in the context of your existing project.
     24You can let users volunteer run test versions,
     25warning them in advance that these versions are more likely to crash.
     26These "beta-test users" will get a mixture of regular and test results, and they'll get credit for both.
     27
     28To use these mechanisms, include the line
    1429{{{
    1530$project_has_beta = true;
    1631}}}
    17  in your html/project/project_specific_prefs.inc file. This will add a 'Run test applications?' option to your project-specific preferences.
    18  * Publicize this on your web site and wait for some users to set their preferences to allow test apps. (Note: this flag is stored in XML in the project_prefs field of the user table; scan for `<allow_beta_work>1</allow_beta_work>`).
     32in your html/project/project_specific_prefs.inc file.
     33This will add a 'Run test applications?' option to the project-specific preferences page.
     34
     35Then publicize this on your web site and wait for some users to set their preferences to allow test apps.
     36(Note: this flag is stored in XML in the project_prefs field of the user table;
     37to find beta-test users, scan for `<allow_beta_work>1</allow_beta_work>`).
     38
     39== Beta-test applications ==
     40
     41You can designate application as "beta",
     42and only beta-test users will be sent jobs for these apps.
     43To do this:
     44
     45 * Create a new [AppVersion application] using [XaddTool xadd]
     46   or the [HtmlOps admin web interface].
     47   Mark the app as "beta", e.g. by including
     48   `<beta>1</beta>` in the `<app>` element.
     49 * Add a validator and assimilator for the test application.
    1950 * Create application versions for the test application.
    20  * Create work (as needed for testing) for the test application. To prevent test work from dominating regular work, either use the -allapps feeder option (and give the test app a small weight) or make a work generator for the test app that maintains only a small number of unsent results.
    21  * Note that for beta applications, the scheduler ignores which apps a user has allowed to run in their project specific preferences (the <app_id> element in the XML). If a project has several beta applications, a user can only either run all or none.
     51 * Create jobs (as needed for testing) for the test application.
     52   To prevent test jobs from dominating regular work,
     53   either use the [BackendPrograms --allapps feeder option] (and give the test app a small weight)
     54   or make a work generator for the test app that maintains only a small number of unsent results.
     55
     56Note: for beta applications,
     57the scheduler ignores which apps a user has allowed to run in their project specific preferences
     58(the <app_id> element in the XML).
     59If a project has several beta applications, a user can only either run all or none.
     60
     61== Beta-test app versions ==
     62
     63You may want to verify that new app versions
     64compute the same results as old ones,
     65i.e. that your validator accepts combinations of old and new.
     66
     67You can do this by designating app versions as "beta".
     68These app versions will be sent only to beta-test users.
     69For results processed using beta app versions,
     70usually the other replicas will be processed using non-beta app versions;
     71you can examine these workunits and check whether they validated.
     72
     73When you create a new app version you can mark it as beta
     74by including <beta/> in its [AppVersionNew version.xml file].
     75You can toggle the beta flag for existing app versions using the
     76[HtmlOps admin web interface].