wiki:AutoFlops

Version 2 (modified by davea, 15 years ago) (diff)

--

Automated estimation of job and app version characteristics

Goals

  • eliminate the need for projects to supply FLOPs estimate for jobs
  • eliminate the need for projects to supply FLOPS estimates for app versions (in app_plan())
  • simplify credit-granting policy

Outline

Server

For each app, maintain

  • flops_avg: the estimated average number of FLOPs used by the app's jobs
  • flops_stdev: the standard deviation of the above

Note: if a project has different types of jobs for a given app, with widely differing durations, they should create separate apps for them. That will reduce the variance of the estimate.

Initially this is set to a high value (e.g. 1 GFLOP-day).

Update: whenever a complete job is reported, let x = duration * (host's flops_est). Update flops_avg in a way that favors decrease over increase. That way, hosts that execute the app efficiently (close to peak hardware speed) have a larger weight in the estimate.

Job completion time estimate: app.flops_avg / (host's flops_est for this app version)

client

for each app version, maintain

  • flops_est: dynamic estimate of the real FLOPS of the app version on this host.

Initially this is based on the peak hardware speed, i.e. ngpus*(GPU peak FLOPs) + avg_ncus * (whetstone). Update: when a job finishes, let x = (job.flops / duration). Update accordingly (but cap at peak hardware speed).

Note: this replaces "duration correction factor".

protocol

Request message: add flops_est for each app version

Credit

Grant each validated job credit proportional to app.flops_avg

Note: a user doesn't get more credit for a longer-than-average job. But it averages out in the end.