Changes between Version 1 and Version 2 of ServerDirs


Ignore:
Timestamp:
Mar 26, 2014, 2:59:03 PM (10 years ago)
Author:
davea
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • ServerDirs

    v1 v2  
    3434The upload and download directories may contain large numbers (millions) of files. For efficiency they are normally organized as a hierarchy of subdirectories.
    3535
    36 == Hierarchical upload/download directories ==
     36== Upload and download directory hierarchies ==
    3737
    38 The data server for a large project may store 100Ks or millions of files at any given point. If these files are stored in 'flat' directories (project/download and project/upload) the data server may spend a lot of CPU time searching directories. To avoid this, BOINC uses hierarchical upload/download directories. Each directory has a set of 1024 subdirectories, named 0 to 3ff. Files are hashed (based on their filename) into these directories.
     38The data server for a large project may store 100Ks or millions of files at a given point.
     39If these files are stored in flat directories
     40the data server may spend a lot of CPU time searching these directories.
     41To avoid this, BOINC uses hierarchical upload and download directories.
     42Each directory has a set of 1024 subdirectories, named 0 to 3ff.
     43Files are hashed (based on their filename) into these directories.
    3944
    40 The hierarchy is used for input and output files only. Executables and other application version files are in the top level of the download directory.
     45The hierarchies are used for input and output files only.
     46Executables and other application files are in the top level of the download directory.
    4147
    42 This affects your project-specific code in a couple of places. First, your work generator must put input files in the right directory before calling [WorkGeneration create_work()]. To do this, it can use the function
     48Normally you don't need to know about these hierarchies.
     49[JobStage stage_file] automatically puts input files in the right place.
     50If for some reason you want to know what directory an input file is in,
     51run the command
     52{{{
     53bin/dir_hier_path filename
     54}}}
    4355
    44 {{{
    45 int dir_hier_path(
    46     const char* filename, const char* root, int fanout, char* result,
    47     bool make_directory_if_needed=false
    48 );
    49 }}}
    50 This takes a name of the input file and the absolute path of the root of the download hierarchy (typically the download_dir element from config.xml) and returns the absolute path of the file in the hierarchy. Generally make_directory_if_needed should be set to true: this creates a fanout directory if needed to accomodate a particular file.
    51 
    52 Secondly, your validator and assimilator should call
     56For output files, your validator and assimilator should call
    5357{{{
    5458int get_output_file_path(RESULT const& result, string& path);
     
    5963}}}
    6064to get the paths of output files in the hierarchy.
    61 
    62 A couple of utility programs are available (run this in the project root directory):
    63 {{{
    64 dir_hier_move src_dir dst_dir fanout
    65 dir_hier_path filename
    66 }}}
    67 dir_hier_move moves all files from src_dir (flat) into dst_dir (hierarchical with the given fanout). dir_hier_path, given a filename, prints the full pathname of that file in the hierarchy.