Changes between Initial Version and Version 1 of WorkGeneration

Timestamp:

Apr 18, 2007, 1:58:11 PM (17 years ago)

Author:

davea

Comment:

--

WorkGeneration

@@ +1 @@
+As described earlier, a workunit represents the inputs to a computation. The steps in creating a workunit are:
+
+    * Write XML 'template files' that describe the workunit and its corresponding results. Generally the same templates will be used for a large number work of workunits.
+    * Create the workunit's input file(s) and place them in the download directory.
+    * Invoke a BOINC function or script that creates a database record for the workunit. 
+
+Once this is done, BOINC takes over: it creates one or more results for the workunit, distributes them to client hosts, collects the output files, finds a canonical result, assimilates the canonical result, and deletes files.
+
+During the testing phase of a project, you can use the make_work daemon to replicate a given workunit as needed to maintain a constant supply of work. This is useful while testing and debugging the application.
+Workunit and result template files
+
+A workunit template file has the form
+{{{
+<file_info>
+    <number>0</number>
+    [ <sticky/>, other attributes]
+</file_info>
+[ ... ]
+<workunit>
+    <file_ref>
+        <file_number>0</file_number>
+        <open_name>NAME</open_name>
+    </file_ref>
+    [ ... ]
+    [ <command_line>-flags xyz</command_line> ]
+    [ <rsc_fpops_est>x</rsc_fpops_est> ]
+    [ <rsc_fpops_bound>x</rsc_fpops_bound> ]
+    [ <rsc_memory_bound>x</rsc_memory_bound> ]
+    [ <rsc_disk_bound>x</rsc_disk_bound> ]
+    [ <delay_bound>x</delay_bound> ]
+    [ <min_quorum>x</min_quorum> ]
+    [ <target_nresults>x</target_nresults> ]
+    [ <max_error_results>x</max_error_results> ]
+    [ <max_total_results>x</max_total_results> ]
+    [ <max_success_results>x</max_success_results> ]
+    [ <credit>X</credit> ]
+</workunit>
+}}}
+The components are:
+
+ <file_info>, <file_ref>::
+
+        Each pair describes an input file and the way it's referenced.
+
+ <command_line>::
+
+        The command-line arguments to be passed to the main program.
+
+ <credit>::
+
+        The amount of credit to be granted for successful completion of this workunit. Use this only if you know in advance how many FLOPs it will take. Your validator must use get_credit_from_wu() as its compute_granted_credit() function.
+Other elements  Work unit attributes
+
+Workunit database records include a field, 'xml_doc', that is an XML-format description of the workunit's input files. This is derived from the workunit template as follows:
+
+    * Within a <file_info> element, <number>x</number> identifies the order of the file. It is replaced with elements giving the filename, download URL, MD5 checksum, and size.
+    * Within a <file_ref> element, <file_number>x</file_number> is replaced with an element giving the filename. 
+
+A result template file has the form
+{{{
+<file_info>
+    <name><OUTFILE_0/></name>
+    <generated_locally/>
+    <upload_when_present/>
+    <max_nbytes>32768</max_nbytes>
+    <url><UPLOAD_URL/></url>
+</file_info>
+<result>
+    <file_ref>
+        <file_name><OUTFILE_0/></file_name>
+        <open_name>result.sah</open_name>
+    </file_ref>
+</result>
+}}}
+Result database records include a field, 'xml_doc_in', that is an XML-format description of the result's output files. This is derived from the result template as follows:
+
+    * <OUTFILE_n> is replaced with a string of the form 'wuname_resultnum_n' where wuname is the workunit name and resultnum is the ordinal number of the result (0, 1, ...).
+    * <UPLOAD_URL/> is replaced with the upload URL. 
+
+Moving input files to the download directory
+If you're using a flat download directory, just put input files in that directory. If you're using hierarchical upload/download directories, you must put each input file in the appropriate directory; the directory is determined by the file's name. To find this directory, call the C++ function
+{{{
+dir_hier_path(
+    const char* filename,
+    const char* root,       // root of download directory
+    int fanout,             // from config.xml
+    char* result,           // path of file in hierarchy
+    bool create_dir=false   // create dir if it's not there
+);
+}}}
+If you're using scripts, you can invoke the program
+{{{
+dir_hier_path filename
+}}}
+It prints the full pathname and creates the directory if needed. Run this in the project's root directory. For example:
+{{{
+cp test_workunits/12ja04aa `bin/dir_hier_path 12ja04aa`
+}}}
+copies an input file from the test_workunits directory to the download directory hierarchy.
+Creating workunit records
+
+Workunits can be created using either a script (using the create_work program) or a program (using the create_work() function). The input files must already be in the download hierarchy.
+
+The utility program is
+{{{
+create_work
+    -appname name                       // application name
+    -wu_name name                       // workunit name
+    -wu_template filename               // WU template filename
+        // relative to project root; usually in templates/
+    -result_template filename           // result template filename
+        // relative to project root; usually in templates/
+    [ -batch n ]
+    [ -priority n ]
+
+    // The following may be passed in the WU template,
+    // or as command-line arguments to create_work,
+    // or not passed at all (defaults will be used)
+
+    [ -command_line "-flags foo" ]
+    [ -rsc_fpops_est x ]
+    [ -rsc_fpops_bound x ]
+    [ -rsc_memory_bound x ]
+    [ -rsc_disk_bound x ]
+    [ -delay_bound x ]
+    [ -min_quorum x ]
+    [ -target_nresults x ]
+    [ -max_error_results x ]
+    [ -max_total_results x ]
+    [ -max_success_results x ]
+    [ -additional_xml 'x' ]
+
+    infile_1 ... infile_m           // input files
+}}}
+The program must be run in the project root directory. The workunit parameters are documented here. The -additional_xml argument can be used to supply, for example, <credit>12.4</credit>.
+
+BOINC's library (backend_lib.C,h) provides the functions:
+{{{
+int create_work(
+    DB_WORKUNIT&,
+    const char* wu_template,                  // contents, not path
+    const char* result_template_filename,     // relative to project root
+    const char* result_template_filepath,     // absolute or relative to current dir
+    const char** infiles,                     // array of input file names
+    int ninfiles
+    SCHED_CONFIG&,
+    const char* command_line = NULL,
+    const char* additional_xml = NULL
+);
+}}}
+create_work() creates a workunit. The arguments are similar to those of the utility program; some of the information is passed in the DB_WORKUNIT structure, namely the following fields:
+
+name
+appid
+
+The following may be passed either in the DB_WORKUNIT structure or in the workunit template file:
+
+rsc_fpops_est
+rsc_fpops_bound
+rsc_memory_bound
+rsc_disk_bound
+batch
+delay_bound
+min_quorum
+target_nresults
+max_error_results
+max_total_results
+max_success_results
+
+Examples
+Making one workunit
+
+Here's a program that generates one workunit (error-checking is omitted for clarity):
+{{{
+#include "backend_lib.h"
+
+main() {
+    DB_APP app;
+    DB_WORKUNIT wu;
+    char wu_template[LARGE_BLOB_SIZE];
+    char* infiles[] = {"infile"};
+
+    SCHED_CONFIG config;
+    config.parse_file();
+
+    boinc_db.open(config.db_name, config.db_host, config.db_passwd);
+    app.lookup("where name='myappname'");
+
+    wu.clear();     // zeroes all fields
+    wu.appid = app.id;
+    wu.min_quorum = 2;
+    wu.target_nresults = 2;
+    wu.max_error_results = 5;
+    wu.max_total_results = 5;
+    wu.max_success_results = 5;
+    wu.rsc_fpops_est = 1e10;
+    wu.rsc_fpops_bound = 1e11;
+    wu.rsc_memory_bound = 1e8;
+    wu.rsc_disk_bound = 1e8;
+    wu.delay_bound = 7*86400;
+    read_filename("templates/wu_template.xml", wu_template, sizeof(wu_template));
+    create_work(
+        wu,
+        wu_template,
+        "templates/results_template.xml",
+        "templates/results_template.xml",
+        infiles,
+        1,
+        config
+    );
+}
+}}}
+This program must be run in the project directory since it expects to find the config.xml file in the current directory.
+Making lots of workunits
+
+If you're making lots of workunits (e.g. to do the various parts of a parallel computation) you'll want the workunits to differ either in their input files, their command-line arguments, or both.
+
+For example, let's say you want to run a program on ten input files 'file0', 'file1', ..., 'file9'. You might modify the above program with the following code:
+{{{
+    char filename[256];
+    char* infiles[1];
+    infiles[0] = filename;
+    ...
+    for (i=0; i<10; i++) {
+        sprintf(filename, "file%d", i);
+        create_work(
+            wu,
+            wu_template,
+            "templates/results_template.xml",
+            "templates/results_template.xml",
+            infiles,
+            1,
+            config
+        );
+    }
+}}}
+Note that you only need one workunit template file and one result template file.
+
+Now suppose you want to run a program against a single input file, but with ten command lines, '-flag 0', '-flag 1', ..., '-flag 9'. You might modify the above program with the following code:
+{{{
+    char command_line[256];
+    ...
+    for (i=0; i<10; i++) {
+        sprintf(command_line, "-flag %d", i);
+        create_work(
+            wu,
+            wu_template,
+            "templates/results_template.xml",
+            "templates/results_template.xml",
+            infiles,
+            1,
+            config,
+            command_line
+        );
+    }
+}}}
+Again, you only need one workunit template file and one result template file.