Changes between Version 3 and Version 4 of LocalityNew


Ignore:
Timestamp:
08/14/12 17:17:04 (5 years ago)
Author:
davea
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • LocalityNew

    v3 v4  
    6060{{{
    6161batch
    62         // this table already exists; we may need to add fields to it
     62    // this table already exists; we may need to add fields to it
    6363
    6464batch_host              // batch/host association table
    65         host_id integer
    66         batch_id integer
    67         cursor_id integer
     65    host_id integer
     66    batch_id integer
     67    cursor_id integer
    6868
    6969locality_cursor
    70         batch_id integer
    71         expavg_credit double
    72                 // sum of expavg_credit of hosts in the team
    73         first_job_num integer
    74         last_job_num integer
    75         first_unfinished_job_num integer
    76                 // all jobs before this have been completed
    77         first_ungenerated_job_num integer
    78                 // we've generated workunit records for all jobs before this
    79         index on (batch_id, expavg_credit)
     70    batch_id integer
     71    first_job_num integer
     72    last_job_num integer
     73        // range of jobs to be done
     74    expavg_credit double
     75        // sum of expavg_credit of hosts in the team
     76    remaining_credit double
     77        // estimated credit of unfinished jobs
     78        // zero means all jobs finished
     79    first_unfinished_job_num integer
     80        // all jobs before this have been completed
     81    first_ungenerated_job_num integer
     82        // we've generated workunit records for all jobs before this
    8083
    8184workunit (new fields)
    82         cursor_id integer
    83         job_num integer
     85    cursor_id integer
     86    job_num integer
    8487
    8588}}}
    8689
    87 === Initialization ===
     90=== Creating a batch ===
    8891
    89 To initialize a batch:
     92To create a batch:
    9093
    9194 * create batch record
     
    97100==== Assign host to cursors ====
    98101
     102Define
     103{{{
     104est_time_left(cursor) = cursor.remaining_credit/cursor.expavg_credit
     105}}}
     106This is an estimate of the time needed to complete the cursor's jobs,
     107given its current team.
     108
    99109For each batch:
    100110
    101111If this is a new host (i.e. no batch_host record) then
    102  * assign host to cursor for this batch with least expavg_credit
     112 * assign host to cursor for this batch with least greatest est_time_left().
    103113 * create batch_host record
    104114 * add host's expavg_credit to cursor's expavg_credit
    105115
    106116Otherwise, consider moving this host to a different cursor.
    107 Let C = host's cursor.
    108 If C.expavg_credit > 2*lowest expavg among cursors,
    109 then move this host to that cursor.
    110 (This policy may need to be refined a bit).
     117Let C = host's cursor,
     118and let D = the cursor for which est_time_left() is greatest.
     119If est_time_left(C) < .5*est_time_left(D),
     120then move this host to D.
     121(This policy may need to be refined a bit to reduce moving hosts between cursors).
    111122
    112123==== Assigning jobs ====
     
    127138==== Deleting files ====
    128139
    129 If the host has a file that's used only by finished jobs,
    130 tell client to delete it.
     140If the host has a sticky file that's not used by an unfinished job in its cursor,
     141tell client to delete that file.
    131142
    132143Note: names of sticky files should encode the batch and file number.
     
    143154
    144155{{{
    145 if workunit.job_num == cursor.first_unfinished_job_num
    146         cursor.first_unfinished_job_num++
     156while cursor.first_unfinished_job_num is finished
     157    cursor.first_unfinished_job_num++
     158    update cursor.remaining_credit
    147159}}}
    148160