Changes between Version 1 and Version 2 of DirHierarchy


Ignore:
Timestamp:
Apr 25, 2007, 10:44:38 AM (17 years ago)
Author:
Nicolas
Comment:

Required manual changes to automatic conversion.

Legend:

Unmodified
Added
Removed
Modified
  • DirHierarchy

    v1 v2  
    11= Hierarchical upload/download directories =
    22
    3       The data server for a large project, may store 100Ks or millions of files at any given point. If these files are stored in 'flat' directories (project/download and project/upload) the data server may spend a lot of CPU time searching directories. If you see a high CPU load average, with a lot of time in kernel mode, this is probably what's happening.  The solution is to use '''hierarchical upload/download directories'''. To do this, include the line
    4 
    5 
     3The data server for a large project, may store 100Ks or millions of files at any given point. If these files are stored in 'flat' directories (project/download and project/upload) the data server may spend a lot of CPU time searching directories. If you see a high CPU load average, with a lot of time in kernel mode, this is probably what's happening. The solution is to use '''hierarchical upload/download directories'''. To do this, include the line
    64{{{
    75<uldl_dir_fanout>1024</uldl_dir_fanout>
    86}}}
    9       in your [ProjectConfigFile config.xml file] (this is the default for new projects). This causes BOINC to use hierarchical upload/download directories. Each directory will have a set of 1024 subdirectories, named 0 to 3ff. Files are hashed (based on their filename) into these directories.
     7in your [ProjectOptions config.xml file] (this is the default for new projects). This causes BOINC to use hierarchical upload/download directories. Each directory will have a set of 1024 subdirectories, named 0 to 3ff. Files are hashed (based on their filename) into these directories.
    108
    119The hierarchy is used for input and output files only. Executables and other application version files are in the top level of the download directory.
    1210
    1311This affects your project-specific code in a couple of places. First, your work generator must put input files in the right directory before calling [WorkGeneration create_work()]. To do this, it can use the function
    14 
    1512
    1613{{{
     
    2017);
    2118}}}
    22       This takes a name of the input file and the absolute path of the root of the download hierarchy (typically the download_dir element from config.xml) and returns the absolute path of the file in the hierarchy. Generally make_directory_if_needed should be set to true: this creates a fanout directory if needed to accomodate a particular file.  Secondly, your validator and assimilator should call
    2319
     20This takes a name of the input file and the absolute path of the root of the download hierarchy (typically the download_dir element from [ProjectOptions config.xml]) and returns the absolute path of the file in the hierarchy. Generally make_directory_if_needed should be set to true: this creates a fanout directory if needed to accommodate a particular file.  Secondly, your validator and assimilator should call
    2421
    2522{{{
     
    2825int get_output_file_paths(RESULT const& result, vector<string>& );
    2926}}}
    30       to get the paths of output files in the hierarchy.  A couple of utility programs are available (run this in the project root directory):
    3127
     28to get the paths of output files in the hierarchy.  A couple of utility programs are available (run this in the project root directory):
    3229
    3330{{{
     
    3532dir_hier_path filename
    3633}}}
    37       dir_hier_move moves all files from src_dir (flat) into dst_dir (hierarchical with the given fanout). dir_hier_path, given a filename, prints the full pathname of that file in the hierarchy.
     34
     35dir_hier_move moves all files from src_dir (flat) into dst_dir (hierarchical with the given fanout). dir_hier_path, given a filename, prints the full pathname of that file in the hierarchy.
     36
    3837== Transitioning from flat to hierarchical directories ==
    39  If you are operating a project with flat directories, you can transition to a hierarchy as follows:
    4038
     39If you are operating a project with flat directories, you can transition to a hierarchy as follows:
    4140
    42  * Stop the project and add <uldl_dir_fanout> to config.xml. You may want to locate the hierarchy root at a new place (e.g. download/fanout); in this case update the <download_dir> element of config.xml, and add the element
     41 * Stop the project and add `<uldl_dir_fanout>` to [ProjectOptions config.xml]. You may want to locate the hierarchy root at a new place (e.g. download/fanout); in this case update the `<download_dir>` element of config.xml, and add the element
    4342{{{
    4443<download_dir_alt>old download dir</download_dir_alt>
    4544}}}
    46       This causes the file deleter to check both old and new locations.
     45
     46 This causes the file deleter to check both old and new locations.
    4747 * Use dir_hier_move to move existing upload files to a hierarchy.
    4848 * Start the project, and monitor everything closely for a while.