Changes between Initial Version and Version 1 of AutoFlops

Timestamp:

Aug 28, 2009, 12:32:48 PM (15 years ago)

Author:

davea

Comment:

--

AutoFlops

@@ +1 @@
+= Automated estimation of job and app version characteristics =
+
+== Goals ==
+
+ * eliminate the need for projects to supply FLOPs estimate for jobs
+ * eliminate the need for projects to supply FLOPS estimates for app versions (in app_plan())
+
+== Outline ==
+
+=== Server ===
+For each app, maintain
+ * flops_avg: the estimated average number of FLOPs used by the app's jobs
+ * flops_stdev: the standard deviation of the above
+
+Note: if a project has different types of jobs for a given app,
+with widely differing durations,
+they should create separate apps for them.
+That will reduce the variance of the estimate.
+
+Initially this is set to a high value (e.g. 1 GFLOP-day).
+
+Update: whenever a complete job is reported,
+let x = duration * (host's flops_est).
+Update flops_avg in a way that favors decrease over increase.
+That way, hosts that execute the app efficiently (close to peak hardware speed)
+have a larger weight in the estimate.
+
+Job completion time estimate:
+app.flops_avg / (host's flops_est for this app version)
+
+
+=== client ===
+for each app version, maintain
+
+ * flops_est: dynamic estimate of the real FLOPS of the app version on this host.
+
+Initially this is based on the peak hardware speed, i.e. ngpus*(GPU peak FLOPs) + avg_ncus * (whetstone).
+Update: when a job finishes, let x = (job.flops / duration).
+Update accordingly (but cap at peak hardware speed).
+
+Note: this replaces "duration correction factor".
+
+=== protocol ===
+
+Request message: add flops_est for each app version
+
+== Credit ==
+
+Grant each validated job credit proportional to app.flops_avg
+
+
+