| 1 | = Remote job submission = |
| 2 | |
| 3 | A group from Universitat Pompeu Fabra |
| 4 | has developed a system for remote job submission and monitoring. |
| 5 | This system allows scientists to submit jobs (or groups of jobs) |
| 6 | from a convenient command-line interface. |
| 7 | |
| 8 | The system (Perl-based) is in boinc/rboinc/. |
| 9 | |
| 10 | '''Warning: this system has been used only by its developers. |
| 11 | It will take some work to get it working on other projects.''' |
| 12 | |
| 13 | Powerpoint slides describing the system are |
| 14 | [http://boinc.berkeley.edu/rboinc.pdf here]. |
| 15 | |
| 16 | == Notes == |
| 17 | |
| 18 | The software should be fairly self-explanatory, but installation may be tricky. Here's a general overview |
| 19 | |
| 20 | * boinc_retrieve_server, boinc_submit_server run as cgi. The former, actually, also handles all administrative requests (stop, purge). |
| 21 | * boinc_retrieve, boinc_submit, are the client components (ditto as above for admin requests) |
| 22 | * Exchange of files between client and server is done through WEBDAV http extensions (a scratch area needs be setup for this) |
| 23 | |
| 24 | * Wus naming is important and enforced like this: NNN-UUU_GGG-XX-YY-RNDzzzz where |
| 25 | * NN is the name of the workunit (sub-group) |
| 26 | * UU is the submitter id |
| 27 | * GGG is the group |
| 28 | * XX is the current step in the chain |
| 29 | * YY is the total n. of steps |
| 30 | * zzzz is a random number (not needed,actually) |
| 31 | * WUs are kept in a "workflow_directory", a subdir of the project dir, as per slide 22 of the Powerpoint. |
| 32 | * Inside each dir a "process" bash file is created, which is executed by the assimilator with the name of the assimilated WU as its argument. It will create_work the next step for execution. |
| 33 | * The main reason for using perl is that I preferred to use the XML::Simple module for (un-) xml-ing data structures over the network - it was useful for adding features on the fly keeping backwards compatibility |
| 34 | * I implemented basic functions for authentication, but this is not finished yet |
| 35 | * file storage is optimized through hardlinking and pooling. Network transfers are not (but they could be) |