OK, now we're sucking diesel - or diesel fumes, at any rate.

02/05/2021 16:12:11 | World Community Grid | Sending scheduler request: Project initialization.
02/05/2021 16:12:11 | World Community Grid | Requesting new tasks for CPU and Intel GPU
02/05/2021 16:12:12 | World Community Grid | Scheduler request completed: got 0 new tasks
02/05/2021 16:12:12 | World Community Grid | No tasks are available for OpenPandemics - COVID-19 - GPU
02/05/2021 16:12:12 | World Community Grid | Tasks for CPU are available, but your preferences are set to not accept them
02/05/2021 16:12:12 | World Community Grid | Tasks for NVIDIA GPU are available, but your preferences are set to not accept them
02/05/2021 16:12:12 | World Community Grid | Tasks for AMD/ATI GPU are available, but your preferences are set to not accept them
02/05/2021 16:12:12 | World Community Grid | Project requested delay of 121 seconds
02/05/2021 16:12:12 | World Community Grid | General prefs: from World Community Grid (last modified 02-May-2021 14:35:25)
02/05/2021 16:12:12 | World Community Grid | Computer location: home
02/05/2021 16:14:15 | World Community Grid | Sending scheduler request: To fetch work.
02/05/2021 16:14:15 | World Community Grid | Requesting new tasks for Intel GPU
02/05/2021 16:14:15 | World Community Grid | [sched_op] CPU work request: 0.00 seconds; 0.00 devices
02/05/2021 16:14:15 | World Community Grid | [sched_op] Intel GPU work request: 9504.00 seconds; 1.00 devices
02/05/2021 16:14:16 | World Community Grid | Scheduler request completed: got 0 new tasks

So far, so bad. I'll keep trying a few different combinations.

I've never used the simulator, except in its web-interface form. It often lags behind the master code: I had a problem involving an app_config.xml file, but I couldn't include it in the simulation run. That was added to the input page, but my next problem seemed to involve the use of two app_config.xml files. And so on. But if you can overcome those constraints by building locally, it may give you some insights. NB - it can't simulate the server response "... reached a limit on tasks in progress".

On the server side, your description is fine. But be aware it leaves out the dimension of time. For your req_seconds, the scheduler picks one candidate task, locks it, and performs a number of checks to see if it's right for your machine. If it is, it's assigned to you, added to a buffer, and the whole process starts again. Adding multiple tasks to the buffer - sequentially - takes time, and is optimised as far as possible.

The main optimisation is the use of a 'feeder'. The feeder preforms all the database lookups to populate a small pool of available tasks for allocation. On small projects, that pool contains maybe 100 or 200 tasks at a time, and is held in the fastest-access memory available. I shudder to think how many may be needed for WCG in this mode! That single pool has to be used by every scheduler instance currently running (multiple requests are processed in parallel). So, while your scheduler is plodding through your request, other schedulers are fishing in the same pool. For a non-trivial request, it's entirely possible for the feeder pool to have been emptied before your scheduler has found enough compatible tasks to add up to your request quantum. You'll get a reply comprising less work than you asked for, and you ask again a couple of minutes later. By which time, the feeder pool will have been emptied and refilled many times over (I think the usual cycle time is 2 seconds).

That may even account for the missing intel_gpu tasks on its own. As we saw, your request is processed device by device - NV, AND, Intel (in that order). The intel_gpu is most likely to hit 'empty feeder syndrome'. Look back at my summary log (message 104231) - does that explanation fit?

It's not necessarily as bad as that. I found a problem with another project earlier this year, and their response when I pointed it out to them was along the lines of "we maintain our own separate version of the server code, except for the scheduler, where we've brought in the latest version as a complete replacement" (or words to that effect - I'm relying on memory).

All the issues we've discussed have been related to the scheduler, and It's entirely possible that WCG will have done something similar. It's unlikely that they will have written an entirely separate replacement: as you and I have found, it's fiendishly complicated, and I think even experienced coders like Bruce have refrained from meddling as much as possible. The BOINC code is the best guide we've got, and it's likely to be pretty close - it has to be compatible with the requests from the newest clients, at the very least.

Thereby hangs a very nasty tale. This machine actually reports

<coproc_cuda>
   <count>2</count>
   <name>GeForce GTX 1660 SUPER</name>

but in practice it's got one 1660 and one 1650. Left to its own devices, the 1650 would be idle, but you can wake it up with <use_all_gpus>1</use_all_gpus> in cc_config.xml

Having two different cards, but only reporting the characteristics of the 'better' one, causes real problems. If the 'smaller' one has less memory, or can't do double precision, or is pre-OpenCL v1.2, the server will still send impossible tasks, and the client will still try to run them (unless you exclude that project on that GPU, also in cc_config.xml).

I went to the 2014 BOINC workshop in Budapest (my local low-cost airline happened to fly direct to Hungary from the airport just up the road, on the right day of the week both out and back. I could hardly pass that one up!), and I heard the 'historical summary' talk David Anderson gave that year. I heard him say that the decision not to separately identify GPUs was a mistake, in hindsight. I don't think either the text or a recording of that talk is available in the public domain, but the slides can be downloaded from https://boinc.berkeley.edu/trac/attachment/wiki/WorkShop14/workshop_14.pdf. Slide 52 was on screen as he said that: "Reflections on software: things we need to change - Coprocessor model". But nobody has.

In pulling out that summary, I deleted all the references to "Server version 701". In theory, that dates back to 2013 or even earlier - almost as old as Einstein's! - but I suspect that in reality, they've forked off the mainline and done their own tweaks.

<coproc_cuda>
   <count>1</count>

<coproc_cuda>
   <count>2</count>

Ah ok. Yes, I misread that. It would be interesting to know if the client is only requesting nvidia since presumably they have their cache set high enough for 150 NV or if the client is requesting all of them and the WCG server sends for the NV GPU until: the 50 per GPU or the req_secs is reached.

Mine is behaving a little differently. I'm working at a Windows machine, with WCG enabled for NV (x2) and Intel_gpu, but not for CPU. It's also got work for other projects, but I'm not fetching from them at the moment. I'm stable at 120 tasks: 8 WCG for intel, 12 from other projects, and thus 100 for NV - and it says "This computer has reached a limit on tasks in progress". Which is what the book says, but not what's been reported.

Meanwhile, see what you think of https://boinc.berkeley.edu/trac/wiki/ProjectOptions#Joblimits. That's the instruction manual for server operators. It's not bang up to date, but four of the last five changes have been made by Kevin Reed - another of the WCG admins - so I guess they're paying attention. Neither that section ('Job limits'), not the next section down ('Job limits (advanced)') seems to allow them to set different limits for different classes of GPU - so, "All GPUs are created equal". ???

<app>
		<app_name>opng</app_name>
		<total_limit>
			<jobs>200</jobs>
		</total_limit>
		<gpu_limit>
			<jobs>50</jobs>
			<per_proc/>
		<zgpu_limit>
</app>

IIRC, the limit is 50 per GPU with a max of 200 per machine. People with multiple GPUs have pointed that out before.

Yes, we've got that one sorted. But I was drawing attention to "gets up to 150 NV wu's" in the post I quoted. He shouldn't have space for 150 tasks for a single NV GPU if they were allocated strictly '50 for the NV, 50 for the ATI, 50 for the APU'. It seems to '150 in total - first come, first served' - that's the effect we're chasing ("Why are there none left for tail-end Charlie?").

If I am interpreting the code properly, the settings are sent with every sched_reply. The account_[WCG] file is overwritten if "project->gui_urls != old_gui_urls || update_project_prefs". update_projects_prefs is set if the venue has changed or if the sent project settings are different from the current project settings. See cs_scheduler.cpp

The alternative way of reading that is "don't send the settings if nothing's changed". I must have changed mine on Wednesday, but not since. They probably haven't considered the case of "user modified client record, so now it's different from what the server remembered" - maybe that's the mis-match that prompts your send.

The interesting question is "why do we have separate lines for <work_req_seconds> and <cpu_req_seconds>?".

The 'historical legacy' answer seems to hold up.

David Jan 10 2009
- client: work_req_seconds is CPU req, not max(CPU req, CUDA req).

My 'account_[WCG]' file still has a datestamp of 28 April - so it's not re-written every time. Probably only when the setting on the server has changed.

if project is anonymous platform, set the overall work req to the max of the requests of resource types for which we have versions. Otherwise projects with old schedulers won't send us work.

I am guessing that WCG falls under the projects with old schedulers even though it is not anonymous_platform.

The interesting question is "why do we have separate lines for <work_req_seconds> and <cpu_req_seconds>?". For the project where I get my CPU tasks from, both are populated, and identical to the microsecond.

I suspect the answer may be historical: when BOINC was first written, there was only one possible sort of work (GPUs couldn't compute), and maybe the wording was changed in newer servers? The alternative may (and probably should) have been retained for backward compatibility.

I'll add that to my code reading list ('blame' is useful in these situations). The other thing I'll look at is the timing evidence from the modification datestamp on the various files - notably, request/reply and the account file.

if project is anonymous platform, set the overall work req to the max of the requests of resource types for which we have versions. Otherwise projects with old schedulers won't send us work.

Edit - interesting post at https://www.worldcommunitygrid.org/forums/wcg/viewthread_thread,43386_offset,570#657527:

Just mentioned that my GPU cruncher with 1660 Super, 280X and A12-9800 APU gets up to 150 NV wu's and almost non for the ATI/AMD. That's no real problem because I can handle this with a second Boinc instance.

With a 50 per GPU limit, is the limit being applied globally? Does the NVidia card get all 150, and leave no quota for the other two? That makes some sense, because NVidia was the first GPU type to be handled by BOINC, followed by AMD and finally Intel. The requests are almost certainly handled in that order in the server code, but I'll check.

Would it be correct to say that if I get it to use the locally modified account_ file and it starts getting the intel gpu tasks that it may be an issue with the client?

I think it would be fair to say that if you got it to work - we don't know what the heck will happen! Your mission, should you choose to accept it, is to ...

I'm not getting the feeling that this is a client problem. You're getting the right sort of figures for 'work request' in the event log, and I think we'd have noticed by now if those figures didn't also appear in sched_request (though that's something you could check). The problem seems to happen after you send the request, and before you get the reply - and the only thing between those two events is the server.

Yes, that should be possible - for testing, at least. But I think it might only work once: it can only have reached your machine via a scheduler request/reply, and it might be repeated in all subsequent replies. Have a look in sched_reply_[WCG].xml (root of BOINC data folder), and see what you can find in there.

It might be safest to stop the client before making that change, and see what happens on restart.

The value to use for 'no restriction' in BOINC is undefined. Sometimes, David has used 0, and sometimes he's used -1. There's even one file where he's used both within 15 lines of code. [https://github.com/BOINC/boinc/blob/master/lib/common_defs.h#L86]. I think he flipped somewhere around 2015, so 'recent' changes like this one (2016) probably use -1, if it's defined at all. I'll look.

If it was not clear earlier in the thread. There is only one thing I change in the project preferences to switch between getting the intel GPU tasks and not getting the intel GPU tasks:
"Allow research to run on my CPU?"
When set to No -> No gpu tasks are received
When set to Yes-> intel gpu tasks are received

I think that is possibly a specific WCG problem with the implementation of those controls: their way of transferring those choices from the web site to the scheduler will be unique to them (because their website is unique). But I still intend to look through the generic BOINC server code when I get a chance, but it'll be slow - I have open diagnosis calls to both Einstein and CPDN on the go at the moment.

Edit - ah, I'd forgotten that WCG also has a user selection for 'max tasks in progress'. That's another complication: I was going to say 'unique to WCG', but I've recently seen it on 'traditional' BOINC projects too. It'll be a thread on this board - can anyone remind me while I finish breakfast?

OK, you can stop looking: message 103817

  <limited_app>
    <limited_app_id>92</limited_app_id>
    <limited_app_max_in_progress>25</limited_app_max_in_progress>
  </limited_app>
  <limited_app>
    <limited_app_id>94</limited_app_id>
    <limited_app_max_in_progress>25</limited_app_max_in_progress>
  </limited_app>

    <apps_selected>
      <app_id>92</app_id>
      <app_id>94</app_id>
      <app_id>82</app_id>
    </apps_selected>

OK, got it.

Then I'm at a loss, and I do not have any more ideas to come up with. :-(

Yes, I have all 4 settings under Graphics Card Usage to Yes. Meaning:
Graphics Card Usage
Do work on my graphics card while computer is in use? Yes
Use my AMD/ATI graphics card if possible: Yes
Use my Intel graphics card if possible: Yes
Use my NVIDIA graphics card if possible: Yes

If it was not clear earlier in the thread. There is only one thing I change in the project preferences to switch between getting the intel GPU tasks and not getting the intel GPU tasks:
"Allow research to run on my CPU?"
When set to No -> No gpu tasks are received
When set to Yes-> intel gpu tasks are received

OK!
Also early on in the BETA, both Richard and I found it very hard to start getting any iGPU tasks at one point. What we did then, was to set the
settings in BOINC for "Store at least" and "Store up to an additional" to very low numbers. "Store at least" to 0.05 or 0.10 days, and "Store up to an additional" to 0.01 days.
Then tasks for the iGPU started to flow. After it started working then it was possible to increase those numbers.

Edit, added: But then of course iGPU tasks is not something you get at every request, since WCG has the strange idea that iGPU must be matched with another iGPU.
There's a lot fewer iGPU's than NVIDIA and AMD by the looks of things.
Same for NVIDIA and AMD. They must be matched with the same device type,

I have absolutely no problems getting iGPU tasks. I have always had CPU tasks set to NO.

I have one iGPU, and one discrete GTX980 in the same computer, and I have tested to set even the GTX980 (NVIDIA) to NO, and keep the iGPU to YES. No problems getting tasks for the iGPU then either.
I tried the other way around too, and no problems getting tasks for NVIDIA.

So, in my case, CPU NO, and NVIDIA and iGPU to YES, delivers what both GPU's want. (even if I have no AMD GPU, I have AMD set to YES also)

Hi Grumpy_Swede, good to see you over here too! :)
I am only able to reproduce this issue when the iGPU is the only GPU that the machine has.

The same goben. I've been here on the BOINC site for a long time.

Have you set all GPU settings to YES, even if you don't have any NVIDIA or AMD GPU's in that computer?
Early in BETA Uplinger mentioned something about that.

I have absolutely no problems getting iGPU tasks. I have always had CPU tasks set to NO.

I have one iGPU, and one discrete GTX980 in the same computer, and I have tested to set even the GTX980 (NVIDIA) to NO, and keep the iGPU to YES. No problems getting tasks for the iGPU then either.
I tried the other way around too, and no problems getting tasks for NVIDIA.

So, in my case, CPU NO, and NVIDIA and iGPU to YES, delivers what both GPU's want. (even if I have no AMD GPU, I have AMD set to YES also)

Hi Grumpy_Swede, good to see you over here too! :)
I am only able to reproduce this issue when the iGPU is the only GPU that the machine has.

I have absolutely no problems getting iGPU tasks. I have always had CPU tasks set to NO.

I have one iGPU, and one discrete GTX980 in the same computer, and I have tested to set even the GTX980 (NVIDIA) to NO, and keep the iGPU to YES. No problems getting tasks for the iGPU then either.
I tried the other way around too, and no problems getting tasks for NVIDIA.

So, in my case, CPU NO, and NVIDIA and iGPU to YES, delivers what both GPU's want. (even if I have no AMD GPU, I have AMD set to YES also)