indefinite suspension of computing when changing system clock

Wow, I really thought there was a ticket for this already.

Many problems appear when the system clock is changed. Most impossible to solve, or so hard it's not worth it.

For example, if you have your clock 1 month forward than the correct date, and contact a scheduler, the deferral time is stored as an absolute timestamp: when to contact the server again. If you then take your clock 1 month back (ie. to correct time), communication with that project will be deferred for a month and a bit.

I think there are three ways to mitigate this:

• Check for time conflicts during every server interaction. This would at least log a relevant message.
• Check every time against the current time looking for time-travel errors.
• Subscribe to time-change events from the operating system.

Sadly, there is no quick fix. None of these methods (and really we need all of them, not just one) are particularly simple to implement.

Replying to Nicolas:

Wow, I really thought there was a ticket for this already.

Well, I searched both trac and the message boards before posting, and I couldn't find it.

Many problems appear when the system clock is changed. Most impossible to solve, or so hard it's not worth it. For example, if you have your clock 1 month forward than the correct date, and contact a scheduler, the deferral time is stored as an absolute timestamp: when to contact the server again. If you then take your clock 1 month back (ie. to correct time), communication with that project will be deferred for a month and a bit.

I agree there are lots of problems, but this particular one seems to cause significant loss of scientific work (by halting computing) at one specific and clearly-defined point: the two or three seconds between

Running CPU benchmarks

and

[benchmark_debug] Starting floating-point benchmark

That would seem to be worth solving on its own, and shouldn't be to difficult to track down what it's waiting for.

I think I've got it:

File: cs_benchmark.C Routine: cpu_benchmarks_poll Line 309:

static double last_time = 0;

If benchmarks have been run in the future (as envisioned by changeset [12128], lines 247-248), this static variable will be pre-initialised to some time in the indefinite future. The test at line 312 will always be satisfied, and the application hangs, by indefinite looping.

Solution: discard variable last_time (or set it to zero) at all possible valid exit points from the benchmarking process.