Changes between Version 3 and Version 4 of JobSubmission


Ignore:
Timestamp:
Sep 22, 2009, 9:29:45 AM (15 years ago)
Author:
davea
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • JobSubmission

    v3 v4  
    1717that creates lots of jobs.
    1818
    19 == Input and output template files == #templates
     19== Input template files ==
    2020
    2121An input template file has the form
    … …  
    5353
    5454 '''<file_info>''':: describes an [BoincFiles#Fileproperties input file].
    55  '''<open_name>''':: the physical name of the file.
    56  '''<sticky/>''':: if present, the file remains on the client after job is finished.
    57  '''<nodelete/>''':: if present, the file is not deleted from the server after job is completed.
    58  '''<report_on_rpc/>''':: if present, report file in each scheduler request (sticky files)
     55  '''<open_name>''':: the physical name of the file.
     56  '''<sticky/>''':: if present, the file remains on the client after job is finished.
     57  '''<nodelete/>''':: if present, the file is not deleted from the server after job is completed.
     58  '''<report_on_rpc/>''':: if present, report file in each scheduler request (sticky files)
    5959
    6060 '''<file_ref>''':: describes [BoincFiles#Filereferences the way the file is referenced].
    61  '''<open_name>''':: the logical name of the file
    62  '''<copy_file>''':: if present, the file is copied into the job's slot directory
    63 
    64  '''<command_line>'''::
    65 
    66         The command-line arguments to be passed to the main program.
     61  '''<open_name>''':: the logical name of the file
     62  '''<copy_file>''':: if present, the file is copied into the job's slot directory
     63
     64 '''<command_line>''':: The command-line arguments to be passed to the main program.
    6765
    6866 '''<credit>'''::
    69 
    70         The amount of credit to be granted for successful completion of this workunit. Use this only if you know in advance how many FLOPs it will take. Your [ValidationSimple validator] must use get_credit_from_wu() as its compute_granted_credit() function.
     67        The amount of credit to be granted for successful completion of this workunit.
     68        Use this only if you know in advance how many FLOPs it will take.
     69        Your [ValidationSimple validator] must use {{{get_credit_from_wu()}}}
     70        as its {{{compute_granted_credit()}}} function.
    7171
    7272 '''<rsc_fpops_est>''' etc.::
    73         [JobIn Work unit attributes]
     73   [JobIn Job attributes] such has how much disk space will be used.
     74   BOINC will supply reasonable defaults for these,
     75   but you should supply the correct values;
     76   otherwise, for example, BOINC might try to run the job
     77   on a host with insufficient disk space.
     78
     79== Output template files ==
    7480
    7581An output template file has the form
    … …  
    9399
    94100Elements and tags must be on separate lines as shown.
    95 '''<file_info>''': describes an output file.
    96  <file_ref> pair describes an output file.
    97101The elements include:
    98  '''<name>''' and '''<file_name>''':: the physical file name.  Typically use <OUTFILE_0>, <OUTFILE_1> etc.; this will be replaced with a generated name based on the job name.
     102
     103 '''<file_info>''':: describes an output file.
     104 '''<name>''':: the physical file name.
     105  Typically use <OUTFILE_0>, <OUTFILE_1> etc.;
     106  BOINC will replace this with a generated name based on the job name.
     107
     108 '''<file_ref>''':: describes how an output file will be referenced by the application.
     109 '''<open_name>''':: the "logical name" by which the application will reference the file.
     110 '''<copy_file/>''':: if present, the file will be generated in the slot directory,
     111   and copied to the project directory after the job has finished.
     112   Use this for [WrapperApp legacy applications].
    99113 '''<generated_locally/>''':: always include this for output files.
    100  '''<max_nbytes>''':: maximum file size.  If the actual size exceeds this, the file will not be uploaded, and the job will be marked as an error.
    101  '''<url>''':: the URL of the file upload handler.  You may include this explicitly, or use '''<UPLOAD_URL/>''' to use the URL in your project's config.xml file.
    102  '''<open_name>''':: the "logical name" by which your application will reference the file.
    103  '''<optional>''':: if false, your application must create the file, otherwise the job will be marked as an error.
    104  '''<no_validate>''':: if true, don't include this file in the result validation process (relevant only if you are using the sample bitwise validator).
    105 
    106 Both elements may have other properties, see BoincFiles for the full list.
     114 '''<max_nbytes>''':: maximum file size.
     115  If the actual size exceeds this, the file will not be uploaded,
     116  and the job will be marked as an error.
     117 '''<url>''':: the URL of the file upload handler.
     118  You may include this explicitly, or use '''<UPLOAD_URL/>'''
     119  to use the URL in your project's config.xml file.
     120 '''<optional>''':: if 0 or absent, your application must create the file,
     121  otherwise the job will be marked as an error.
     122 '''<no_validate>''':: if true, don't include this file in the result validation process
     123  (relevant only if you are using the sample bitwise validator).
     124 '''<no_delete/>''':: if present, the file will not be deleted on the server
     125  even after the job is finished.
    107126
    108127Note: when a job is created, the name of its output template file is stored in the database.
    … …  
    115134(i.e. no more replicas will be created).
    116135
     136== Putting input files in the download directory ==
     137
     138To move an input file to the download directory, use
     139{{{
     140dir_hier_path filename
     141}}}
     142which prints the full pathname and creates the directory if needed (run this in the project's root directory).
     143For example:
     144{{{
     145cp test_files/12ja04aa `bin/dir_hier_path 12ja04aa`
     146}}}
     147copies an input file from the test_files/ directory to the download directory hierarchy.
     148
     149[[PageOutline]]
     150= Submitting jobs =
     151
     152To submit a job you must
     153
     154 1. Write XML 'template files' that describe the job's input and outputs (typically the same template files can be used for many jobs).
     155 1. Create the job's input file(s), and put them in the right place (as determined by the file name) in the [DirHierarchy download directory hierarchy].
     156 1. Invoke a BOINC function or script that submits the job.
     157
     158Once this is done, BOINC takes over: it creates one or more instances of the job,
     159distributes them to client hosts, collects the output files.
     160It [ValidateIntro validates] and
     161[AssimilateIntro processes] the results,
     162and deletes the input and output files.
     163
     164Typically, steps 2) and 3) are done by a [WorkGeneration work generator] program
     165that creates lots of jobs.
     166
     167== Input and output template files == #templates
     168
     169An input template file has the form
     170{{{
     171<file_info>
     172    <number>0</number>
     173    [ <sticky/> ]
     174    [ <nodelete/> ]
     175    [ <report_on_rpc/> ]
     176</file_info>
     177[ ... other files  ]
     178<workunit>
     179    <file_ref>
     180        <file_number>0</file_number>
     181        <open_name>NAME</open_name>
     182        [ <copy_file/> ]
     183    </file_ref>
     184    [ ... other files ]
     185    [ <command_line>-flags xyz</command_line> ]
     186    [ <rsc_fpops_est>x</rsc_fpops_est> ]
     187    [ <rsc_fpops_bound>x</rsc_fpops_bound> ]
     188    [ <rsc_memory_bound>x</rsc_memory_bound> ]
     189    [ <rsc_disk_bound>x</rsc_disk_bound> ]
     190    [ <delay_bound>x</delay_bound> ]
     191    [ <min_quorum>x</min_quorum> ]
     192    [ <target_nresults>x</target_nresults> ]
     193    [ <max_error_results>x</max_error_results> ]
     194    [ <max_total_results>x</max_total_results> ]
     195    [ <max_success_results>x</max_success_results> ]
     196    [ <credit>X</credit> ]
     197</workunit>
     198}}}
     199Elements and tags must be on separate lines as shown.
     200The components are:
     201
     202 '''<file_info>''':: describes an [BoincFiles#Fileproperties input file].
     203  '''<open_name>''':: the physical name of the file.
     204  '''<sticky/>''':: if present, the file remains on the client after job is finished.
     205  '''<nodelete/>''':: if present, the file is not deleted from the server after job is completed.
     206  '''<report_on_rpc/>''':: if present, report file in each scheduler request (sticky files)
     207
     208 '''<file_ref>''':: describes [BoincFiles#Filereferences the way the file is referenced].
     209  '''<open_name>''':: the logical name of the file
     210  '''<copy_file>''':: if present, the file is copied into the job's slot directory
     211
     212 '''<command_line>''':: The command-line arguments to be passed to the main program.
     213
     214 '''<credit>'''::
     215        The amount of credit to be granted for successful completion of this workunit.
     216        Use this only if you know in advance how many FLOPs it will take.
     217        Your [ValidationSimple validator] must use {{{get_credit_from_wu()}}}
     218        as its compute_granted_credit() function.
     219
     220 '''<rsc_fpops_est>''' etc.::
     221        [JobIn Work unit attributes]
     222
     223An output template file has the form
     224{{{
     225<file_info>
     226    <name><OUTFILE_0/></name>
     227    <generated_locally/>
     228    <upload_when_present/>
     229    <max_nbytes>32768</max_nbytes>
     230    <url><UPLOAD_URL/></url>
     231</file_info>
     232<result>
     233    <file_ref>
     234        <file_name><OUTFILE_0/></file_name>
     235        <open_name>result.sah</open_name>
     236        [ <optional>0|1</optional> ]
     237        [ <no_validate>0|1</no_validate> ]
     238    </file_ref>
     239</result>
     240}}}
     241
     242Elements and tags must be on separate lines as shown.
     243The elements include:
     244
     245 '''<file_info>''':: describes an output file.
     246 '''<name>''':: the physical file name.
     247  Typically use <OUTFILE_0>, <OUTFILE_1> etc.;
     248  this will be replaced with a generated name based on the job name.
     249
     250 '''<file_ref>''' describes an the output file will be referenced by the application.
     251 '''<open_name>''':: the "logical name" by which your application will reference the file.
     252 '''<generated_locally/>''':: always include this for output files.
     253 '''<max_nbytes>''':: maximum file size.
     254   If the actual size exceeds this, the file will not be uploaded,
     255   and the job will be marked as an error.
     256 '''<url>''':: the URL of the file upload handler.
     257  You may include this explicitly, or use '''<UPLOAD_URL/>'''
     258  to use the URL in your project's config.xml file.
     259 '''<optional>''':: if 0 or absent, your application must create the file,
     260  otherwise the job will be marked as an error.
     261 '''<no_validate>''':: if true, don't include this file in the result validation process
     262  (relevant only if you are using the sample bitwise validator).
     263
     264Note: when a job is created, the name of its output template file is stored in the database.
     265The file is read when instances of the job are created, which may happen days or weeks later.
     266Thus, editing an output template file can affect existing jobs.
     267If this is not desired, you must create a new output template file.
     268
     269You can safely remove a workunit template file after creating your last WU with it.
     270However, you can't delete a result template file until any WU that refers to it is completed
     271(i.e. no more replicas will be created).
     272
     273== Staging input files ==
     274
     275Before submitting a job, you must put its input files
     276into the proper place in the [ServerDirs download directory hierarchy].
     277BOINC provides both a command-line tool and a function for this.
     278The command-line tool is:
     279{{{
     280dir_hier_path filename
     281}}}
     282which prints the full pathname and creates the directory if needed
     283(run this in the project's root directory). For example:
     284{{{
     285cp test_files/12ja04aa `bin/dir_hier_path 12ja04aa`
     286}}}
     287copies an input file from the test_files directory to the download directory hierarchy.
     288
     289The function (a member function of the {{{SCHED_CONFIG}}} class) is
     290{{{
     291// convert filename to path in download hierarchy
     292//
     293SCHED_CONFIG::download_path(
     294    const char* filename,    // name of input file
     295    char* path               // path in download hierarchy where it should go
     296);
     297}}}
     298
    117299== Submitting a job manually == #creatework-tool
    118300
    119 To move an input file to the download directory, use
    120 {{{
    121 dir_hier_path filename
    122 }}}
    123 which prints the full pathname and creates the directory if needed (run this in the project's root directory). For example:
    124 {{{
    125 cp test_files/12ja04aa `bin/dir_hier_path 12ja04aa`
    126 }}}
    127 copies an input file from the test_files directory to the download directory hierarchy.
    128 
    129 To submit a job, run bin/create_work from the project root directory:
     301'''create_work''' is a command-line tool for submitting jobs.
     302Run it from the project root directory.
    130303{{{
    131304create_work [ arguments ] infile_1 ... infile_n
    132305}}}
    133306Mandatory arguments are:
    134     --appname name::
    135   application name
    136     --wu_name name::
    137         workunit name
    138     --wu_template filename::           
    139  WU template filename relative to project root; usually in templates/
    140     --result_template filename::
    141           result template filename, relative to project root; usually in templates/
     307 --appname name:: application name
     308 --wu_name name:: workunit name
     309 --wu_template filename:: WU template filename relative to project root; usually in templates/
     310 --result_template filename:: result template filename, relative to project root; usually in templates/
     311
    142312Optional arguments are:
    143     --batch n::
    144     --priority n::
     313 --batch n::
     314 --priority n::
    145315
    146316The following [JobIn job parameters] may be passed in the input template,
    … …  
    148318or not passed at all, in which case defaults will be used.
    149319
    150     --command_line "-flags foo"::
    151     --rsc_fpops_est x:: FLOPs estimate
    152     --rsc_fpops_bound x::
    153     --rsc_memory_bound x::
    154     --rsc_disk_bound x::
    155     --delay_bound x::
    156     --min_quorum x::
    157     --target_nresults x::
    158     --max_error_results x::
    159     --max_total_results x::
    160     --max_success_results x::
    161     --opaque N::
    162     --additional_xml 'x':: This can be used to supply, for example, <credit>12.4</credit>.
     320 --command_line "-flags foo"::
     321 --rsc_fpops_est x:: FLOPs estimate
     322 --rsc_fpops_bound x::
     323 --rsc_memory_bound x::
     324 --rsc_disk_bound x::
     325 --delay_bound x::
     326 --min_quorum x::
     327 --target_nresults x::
     328 --max_error_results x::
     329 --max_total_results x::
     330 --max_success_results x::
     331 --opaque N::
     332 --additional_xml 'x':: This can be used to supply, for example, <credit>12.4</credit>.
    163333
    164334== Submitting jobs from a C++ program == #cpp-workgen
    165335
    166 BOINC's library provides the functions:
    167 {{{
    168 // convert filename to path in download hierarchy
    169 //
    170 SCHED_CONFIG::download_path(
    171     const char* filename,
    172     char* path,
    173 );
    174 
    175 // submit a job
    176 //
     336BOINC's library provides a function for submitting jobs:
     337{{{
    177338int create_work(
    178339    DB_WORKUNIT& wu,