Changes between Version 11 and Version 12 of RemoteInputFiles


Ignore:
Timestamp:
01/30/17 16:06:58 (11 months ago)
Author:
davea
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • RemoteInputFiles

    v11 v12  
    11[[PageOutline]]
    2 = Remote management of input files =
     2= Job-based input file management =
    33
    44Input files of BOINC jobs must be available on a public web server.
    5 Usually this is the project's BOINC server.
    6 
    75For projects that use [RemoteJobs remote job submission],
    86job submitters don't have login access to the BOINC server,
    97so they can't store files there directly.
    10 Instead, BOINC provides two mechanisms that allow
    11 them to store and manage files on the BOINC server.
    128
    13 '''Job-based file management''': files are automatically transferred
    14 as part of [RemoteJob remote job submission].
    15 It's intended to be integrated with such a system;
    16 job submitters are not aware of it.
     9There are several options:
    1710
    18 '''Per-user file sandbox''': job submitters explicitly maintain,
    19 via a web interface, a set of files on the server.
     11 * Files are served from a publicly-accessible server,
     12  possibly other than the BOINC server.
     13  They must be managed, and file immutability enforced,
     14  by a mechanism outside BOINC.
     15 * [FileSandbox Per-user file sandbox]: job submitters explicitly maintain,
     16  via a web interface, a set of files on the server.
     17 * '''Job-based file management''': files are automatically transferred
     18  from the submission machine to the BOINC server via Web RPCs.
    2019
    21 Each mechanism deals with several issues:
     20This document describes the latter mechanism.
    2221
    23  * '''File immutability''': BOINC requires that a file
    24   of a given name can never be changed.
    25  * '''File cleanup''': files on the server must be deleted when they are no longer needed.
    26  * '''Authorization''': only users authorized to submit jobs
    27   should be able to move files to the server.
    28 
    29 '''Note''': both mechanisms upload files via a PHP script.
    30 PHP's default max file upload size is 2MB.
    31 To increase this, edit /etc/php.ini, and change, e.g.
    32 {{{
    33 upload_max_filesize = 64M
    34 post_max_size = 64M
    35 }}}
    36 
    37 == Job-based file management ==
    3822
    3923In this system, you must supply physical names of files
     
    4226
    4327File cleanup is based on file/batch associations.
     28You must create a batch (with create_batch())
     29before querying or uploading files.
    4430Each file can be associated with one or more batches.
    4531Files that are no longer associated with an active batch are
     
    5036the RPC handler is html/user/job_files.php.
    5137
    52 This mechanism requires you to create a batch (with create_batch())
    53 before querying or uploading files.
    54 
    55 === C++ interface ===
     38== C++ interface ==
    5639
    5740The following C++ functions are provided (in lib/remote_submit.cpp).
    5841They are to be called on the job submission host;
    59 the files must exist on that host,
    60 and their MD5s must have already been computed.
     42the files must exist on that host.
    6143{{{
    6244extern int query_files(
    6345    const char* project_url,
    6446    const char* authenticator,
    65     std::vector<string> &boinc_names,
     47    std::vector<string> &boinc_names,        // must be unique, e.g. by including content hash
    6648    int batch_id,
    6749    std::vector<int> &absent_files,             // output
     
    11799This will delete files that are no longer associated with an active batch.
    118100
    119 === Python interface ===
     101== Python interface ==
    120102
    121103The Python interface does both RPCs in one function (in lib/submit_api.py):
     
    136118}}}
    137119
    138 == Per-user file sandbox ==
     120== File size limits ==
    139121
    140 This mechanism allows job submitters to explicitly
    141 upload files via a web interface: PROJECT_URL/sandbox.php.
    142 
    143 Links to the files are stored in a "sandbox directory"
    144 PROJECT_ROOT/sandbox/USERID/.
    145 The entries in this directory have contents
     122'''Note''': This mechanism upload files via a PHP script.
     123PHP's default max file upload size is 2MB.
     124To increase this, edit /etc/php.ini, and change, e.g.
    146125{{{
    147 size MD5
     126upload_max_filesize = 64M
     127post_max_size = 64M
    148128}}}
    149129
    150 The actual files are stored in the download directory,
    151 under the name '''sb_userid_MD5'''.
    152 
    153 Currently, files in the sandbox are not cleanup up automatically.
    154 The web interface allows users to delete their files.