Changes between Version 5 and Version 6 of BetaTest

Timestamp:

Jun 23, 2014, 10:45:25 AM (10 years ago)

Author:

davea

Comment:

--

BetaTest

@@ -1 @@
-= Beta-test applications =
+[[PageOutline]]
+= Using volunteers to test applications =
 
-It's important to test new applications on a wide range of hosts, since bugs may appear only with particular OS versions, memory sizes, display types, usage patterns, and so on. It's handy to use volunteers to do this testing, since they provide the needed diversity of hosts. One way to implement this is to create a separate test project. This has two disadvantages:
+Testing new apps and app versions has two phases
 
+ * making sure they doesn't crash;
+ * making sure they computes the correct results.
+
+It's important to test new applications on a wide range of hosts,
+since bugs may appear only with particular OS versions, memory sizes, usage patterns, and so on.
+It's handy to use volunteers to do this testing, since they provide the needed diversity of hosts.
+
+== Beta-test project ==
+
+One way to implement this is to create a separate test project,
+attach your own hosts to it, and get a limited number of volunteers to attach to it.
+
+This is useful, but it has two limitations:
 
  * There is overhead in creating and maintaining a separate project.
- * The credit accrued by testers goes to a different project.
+ * The credit earned by volunteer testers goes to a different project.
 
-BOINC provides a way to do beta testing in the context of your existing project. You can let users volunteer to run test applications, warning them in advance that these applications are more likely to crash. These users will get a mixture of regular and test results, and they'll get credit for both. Here's how to do it:
- * Upgrade to the BOINC server software of Oct 25 2006 or later.
- * Create a new [AppVersion application] using [XaddTool xadd]. Include `<beta>1</beta>` in the `<app>` element to designate it as a beta-test application.
- * Add a validator and assimilator for the test application.
- * Include the line
+BOINC provides two ways to do beta testing in the context of your existing project.
+You can let users volunteer run test versions,
+warning them in advance that these versions are more likely to crash.
+These "beta-test users" will get a mixture of regular and test results, and they'll get credit for both.
+
+To use these mechanisms, include the line
 {{{
 $project_has_beta = true;
 }}}
- in your html/project/project_specific_prefs.inc file. This will add a 'Run test applications?' option to your project-specific preferences.
- * Publicize this on your web site and wait for some users to set their preferences to allow test apps. (Note: this flag is stored in XML in the project_prefs field of the user table; scan for `<allow_beta_work>1</allow_beta_work>`).
+in your html/project/project_specific_prefs.inc file.
+This will add a 'Run test applications?' option to the project-specific preferences page.
+
+Then publicize this on your web site and wait for some users to set their preferences to allow test apps.
+(Note: this flag is stored in XML in the project_prefs field of the user table;
+to find beta-test users, scan for `<allow_beta_work>1</allow_beta_work>`).
+
+== Beta-test applications ==
+
+You can designate application as "beta",
+and only beta-test users will be sent jobs for these apps.
+To do this:
+
+ * Create a new [AppVersion application] using [XaddTool xadd]
+   or the [HtmlOps admin web interface].
+   Mark the app as "beta", e.g. by including
+   `<beta>1</beta>` in the `<app>` element.
+ * Add a validator and assimilator for the test application.
  * Create application versions for the test application.
- * Create work (as needed for testing) for the test application. To prevent test work from dominating regular work, either use the -allapps feeder option (and give the test app a small weight) or make a work generator for the test app that maintains only a small number of unsent results.
- * Note that for beta applications, the scheduler ignores which apps a user has allowed to run in their project specific preferences (the <app_id> element in the XML). If a project has several beta applications, a user can only either run all or none.
+ * Create jobs (as needed for testing) for the test application.
+   To prevent test jobs from dominating regular work,
+   either use the [BackendPrograms --allapps feeder option] (and give the test app a small weight)
+   or make a work generator for the test app that maintains only a small number of unsent results.
+
+Note: for beta applications,
+the scheduler ignores which apps a user has allowed to run in their project specific preferences
+(the <app_id> element in the XML).
+If a project has several beta applications, a user can only either run all or none.
+
+== Beta-test app versions ==
+
+You may want to verify that new app versions
+compute the same results as old ones,
+i.e. that your validator accepts combinations of old and new.
+
+You can do this by designating app versions as "beta".
+These app versions will be sent only to beta-test users.
+For results processed using beta app versions,
+usually the other replicas will be processed using non-beta app versions;
+you can examine these workunits and check whether they validated.
+
+When you create a new app version you can mark it as beta
+by including <beta/> in its [AppVersionNew version.xml file].
+You can toggle the beta flag for existing app versions using the
+[HtmlOps admin web interface].