Big ask - auto throttling of CPU tasks when using GPUs?

Message boards : GPUs : Big ask - auto throttling of CPU tasks when using GPUs?
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Peter Hucker
Avatar

Send message
Joined: 6 Oct 06
Posts: 609
United Kingdom
Message 96234 - Posted: 1 Mar 2020, 12:53:43 UTC

Would it be at all possible for the Boinc client to automatically detect that the CPU usage is too high and lower the number of CPU tasks running so the GPU tasks aren't throttled? At the moment I have to do this manually with app config files, which is quite a hassle when I've got several different projects that could be running in different combinations. Also if the computer is used for other things, the CPU might get more taxed, causing the GPU work to slow down.
ID: 96234 · Report as offensive
Richard Haselgrove
Volunteer tester
Help desk expert

Send message
Joined: 5 Oct 06
Posts: 4094
United Kingdom
Message 96235 - Posted: 1 Mar 2020, 13:01:11 UTC - in response to Message 96234.  

Yes. Use an app_config.xml file and 'declare' that the GPU tasks require 100% of a CPU (core) in support. Doesn't change the actual GPU application behaviour, but does change BOINC's scheduling.
ID: 96235 · Report as offensive
Peter Hucker
Avatar

Send message
Joined: 6 Oct 06
Posts: 609
United Kingdom
Message 96244 - Posted: 1 Mar 2020, 20:15:44 UTC - in response to Message 96235.  

Yes. Use an app_config.xml file and 'declare' that the GPU tasks require 100% of a CPU (core) in support. Doesn't change the actual GPU application behaviour, but does change BOINC's scheduling.


Yes I'm already doing that. But the problem is different tasks require different amounts of CPU. If you're running several projects it can become quite complicated. And other things use the CPU, like the user using the computer. It's very variable and would be better monitored 24/7 by the computer itself. It must be fairly easy for Boinc to monitor the CPU usage, and if it's hitting 100%, lower the number of CPU tasks running. Then no GPU tasks will ever get throttled, and the user will never experience a slowdown of the interface.
ID: 96244 · Report as offensive
Nick Name

Send message
Joined: 14 Aug 19
Posts: 40
United States
Message 96246 - Posted: 1 Mar 2020, 20:30:11 UTC - in response to Message 96244.  

There's no way to "automatically" do what you want or do any dynamic load balancing, at least within BOINC itself. The easiest thing to do is lower your CPU usage limit. BOINC will typically over-commit the CPU when running GPU work, e.g. if BOINC is set to use four CPU threads and you have two GPUs BOINC will run six jobs in total. I take this into account and set CPU use so the number of jobs doesn't exceed the CPU thread count.
Team USA forum
Follow us on Twitter
Help us #crunchforcures!
ID: 96246 · Report as offensive
Peter Hucker
Avatar

Send message
Joined: 6 Oct 06
Posts: 609
United Kingdom
Message 96248 - Posted: 1 Mar 2020, 20:35:46 UTC - in response to Message 96246.  

There's no way to "automatically" do what you want or do any dynamic load balancing, at least within BOINC itself. The easiest thing to do is lower your CPU usage limit. BOINC will typically over-commit the CPU when running GPU work, e.g. if BOINC is set to use four CPU threads and you have two GPUs BOINC will run six jobs in total. I take this into account and set CPU use so the number of jobs doesn't exceed the CPU thread count.


Yes I do that too, but since I run a variety of projects on CPU and GPU, I'd prefer it was done automatically so everything was run as fast as possible. I can lower the CPU usage so the GPU is never throttled, but under some circumstances I end up with unused CPU cycles. I want everything working flat out!
ID: 96248 · Report as offensive
Richard Haselgrove
Volunteer tester
Help desk expert

Send message
Joined: 5 Oct 06
Posts: 4094
United Kingdom
Message 96249 - Posted: 1 Mar 2020, 20:58:21 UTC - in response to Message 96244.  

Yes I'm already doing that. But the problem is different tasks require different amounts of CPU. If you're running several projects it can become quite complicated. And other things use the CPU, like the user using the computer. It's very variable and would be better monitored 24/7 by the computer itself. It must be fairly easy for Boinc to monitor the CPU usage, and if it's hitting 100%, lower the number of CPU tasks running. Then no GPU tasks will ever get throttled, and the user will never experience a slowdown of the interface.
I think you're asking for something like #2949
ID: 96249 · Report as offensive
Peter Hucker
Avatar

Send message
Joined: 6 Oct 06
Posts: 609
United Kingdom
Message 96251 - Posted: 1 Mar 2020, 21:39:03 UTC - in response to Message 96249.  

I think you're asking for something like #2949


Agreed. Now get to work on it :-)
ID: 96251 · Report as offensive
ProDigit

Send message
Joined: 8 Nov 19
Posts: 546
United States
Message 96256 - Posted: 2 Mar 2020, 3:48:56 UTC

Indeed. Gpugrid uses significantly higher CPU usage than the collatz, even though the numbers aren't fully updated to reflect this.
ID: 96256 · Report as offensive
Peter Hucker
Avatar

Send message
Joined: 6 Oct 06
Posts: 609
United Kingdom
Message 96271 - Posted: 2 Mar 2020, 18:13:31 UTC - in response to Message 96256.  
Last modified: 2 Mar 2020, 18:14:11 UTC

Indeed. Gpugrid uses significantly higher CPU usage than the collatz, even though the numbers aren't fully updated to reflect this.


The problem is Boinc is relying on the project estimating how much CPU will be needed. This is a different amount depending on every single combination of GPU and CPU people have, and also it depends if the user is playing a game or using the CPU in another way. It would be far better if Boinc could simply monitor the total CPU load (in the same way Windows task manager does), and add or remove CPU tasks to keep it near but not hitting 100%.
ID: 96271 · Report as offensive
Peter Hucker
Avatar

Send message
Joined: 6 Oct 06
Posts: 609
United Kingdom
Message 96507 - Posted: 8 Mar 2020, 16:52:41 UTC

I was wondering, why doesn't Boinc just set CPU tasks as lowest priority in Windows, and GPU tasks as a higher priority, then the first should never get in the way of the second.

So I checked. It already does this, I guess Windows is rubbish. How can an OS get such a basic thing wrong?
ID: 96507 · Report as offensive
Profile Jord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 14157
Netherlands
Message 96509 - Posted: 8 Mar 2020, 17:36:53 UTC - in response to Message 96507.  

But BOINC doesn't set run priority of applications, as it doesn't run any work. BOINC is just the managing software. It's the project's science applications that do the work and it's the project that sets the run priority of the applications when they compile said applications. BOINC documentation does advise on what to set it to but it's still up to the project to follow that advice.
ID: 96509 · Report as offensive
ProDigit

Send message
Joined: 8 Nov 19
Posts: 546
United States
Message 96510 - Posted: 8 Mar 2020, 17:45:25 UTC - in response to Message 96507.  

I was wondering, why doesn't Boinc just set CPU tasks as lowest priority in Windows, and GPU tasks as a higher priority, then the first should never get in the way of the second.

So I checked. It already does this, I guess Windows is rubbish. How can an OS get such a basic thing wrong?


I don't know about Windows, but I do know in Linux this is the case.
I'm running 5 tasks on a dual core + HT CPU (Pentium Gold G6500). 2 of them are CPU, 2 are Nvidia GPU, and 1 is the Intel IGP.
Boinc seems to focus on getting all GPU (including the weak Intel IGP) active.
That being said, the IGP is about as powerful as the CPU (200Gflops), so losing 1 thread to the IGP is a win anyway (lose <50Gflops on CPU, gain 200GFlops).
ID: 96510 · Report as offensive
Richard Haselgrove
Volunteer tester
Help desk expert

Send message
Joined: 5 Oct 06
Posts: 4094
United Kingdom
Message 96511 - Posted: 8 Mar 2020, 18:35:27 UTC - in response to Message 96509.  

But BOINC doesn't set run priority of applications, as it doesn't run any work. BOINC is just the managing software. It's the project's science applications that do the work and it's the project that sets the run priority of the applications when they compile said applications. BOINC documentation does advise on what to set it to but it's still up to the project to follow that advice.
'Priority' is a difficult word, because it's used in so many settings to mean so many things. But if you mean the thread priority of the CPU (part of a) project science app - yes, it does.

It's always set full-CPU apps to run at idle priority, so as not to get in the way of the foreground apps the user is using. And when GPUs came along, their apps were set to 'below normal'. It's done when the BOINC client is starting the science app, as part of its management role.
ID: 96511 · Report as offensive
Peter Hucker
Avatar

Send message
Joined: 6 Oct 06
Posts: 609
United Kingdom
Message 96514 - Posted: 8 Mar 2020, 18:56:08 UTC - in response to Message 96511.  

But BOINC doesn't set run priority of applications, as it doesn't run any work. BOINC is just the managing software. It's the project's science applications that do the work and it's the project that sets the run priority of the applications when they compile said applications. BOINC documentation does advise on what to set it to but it's still up to the project to follow that advice.
'Priority' is a difficult word, because it's used in so many settings to mean so many things. But if you mean the thread priority of the CPU (part of a) project science app - yes, it does.

It's always set full-CPU apps to run at idle priority, so as not to get in the way of the foreground apps the user is using. And when GPUs came along, their apps were set to 'below normal'. It's done when the BOINC client is starting the science app, as part of its management role.


So why does it fail to work properly in Windows? If I have say a quad core CPU and a GPU connected to it, then attempt to run 4 CPU tasks and 1 GPU task, the GPU is only 50% utilised. If I restrict Boinc to run only 3 CPU tasks, so the CPU never hits 100%, then the GPU works much faster. Clearly Windows task priorities are broken, which doesn't surprise me.
ID: 96514 · Report as offensive
Richard Haselgrove
Volunteer tester
Help desk expert

Send message
Joined: 5 Oct 06
Posts: 4094
United Kingdom
Message 96516 - Posted: 8 Mar 2020, 19:45:00 UTC - in response to Message 96514.  

'Utilisation' is not the same thing as 'priority'. A GPU, in particular, will only be 100% utilised if the problem is amenable to parallelisation, and the programmer is clever enough to program it that way.
ID: 96516 · Report as offensive
Peter Hucker
Avatar

Send message
Joined: 6 Oct 06
Posts: 609
United Kingdom
Message 96517 - Posted: 8 Mar 2020, 19:52:36 UTC - in response to Message 96516.  

'Utilisation' is not the same thing as 'priority'. A GPU, in particular, will only be 100% utilised if the problem is amenable to parallelisation, and the programmer is clever enough to program it that way.


I'd need to look most of those words up :-)

But what I was observing was if the CPU is too busy with unrelated tasks (eg CPU WUs), then the GPU slowed down. This suggests to me that the CPU part of that GPU WU was not being given enough CPU time. But it should have been, as its priority in the OS was higher than the CPU WUs.
ID: 96517 · Report as offensive
Profile Jord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 14157
Netherlands
Message 96520 - Posted: 8 Mar 2020, 20:53:24 UTC - in response to Message 96511.  

It's done when the BOINC client is starting the science app, as part of its management role.
It's done when you compile the program using the BOINC API:
https://boinc.berkeley.edu/trac/wiki/BasicApi, here you set if the app is single- or multi-threaded, which priority it runs at, whether it checkpoints, whether it uses a wrapper, etc.
Please do not private message me for tech support, these will be ignored!
ID: 96520 · Report as offensive
Richard Haselgrove
Volunteer tester
Help desk expert

Send message
Joined: 5 Oct 06
Posts: 4094
United Kingdom
Message 96523 - Posted: 8 Mar 2020, 21:11:48 UTC - in response to Message 96520.  

It's done when the BOINC client is starting the science app, as part of its management role.
It's done when you compile the program using the BOINC API:
https://boinc.berkeley.edu/trac/wiki/BasicApi, here you set if the app is single- or multi-threaded, which priority it runs at, whether it checkpoints, whether it uses a wrapper, etc.
I prefer to look at the actual code:

https://github.com/BOINC/boinc/blob/master/client/app_start.cpp#L570

// Start a task in a slot directory.
// This includes setting up soft links,
// passing preferences, and starting the process
...
// run it at above idle priority if it
// - uses coprocs
// - uses less than one CPU
// - is a wrapper
Not the first time the code has diverged from the documentation, over time.
ID: 96523 · Report as offensive
Richard Haselgrove
Volunteer tester
Help desk expert

Send message
Joined: 5 Oct 06
Posts: 4094
United Kingdom
Message 96524 - Posted: 8 Mar 2020, 21:16:55 UTC - in response to Message 96517.  
Last modified: 8 Mar 2020, 21:30:18 UTC

But what I was observing was if the CPU is too busy with unrelated tasks (eg CPU WUs), then the GPU slowed down. This suggests to me that the CPU part of that GPU WU was not being given enough CPU time. But it should have been, as its priority in the OS was higher than the CPU WUs.
Not necessarily. It might be that the CPU had to refetch a lot of data that wasn't in local cache memory, at every 'context switch' (look up that one up too).
ID: 96524 · Report as offensive
Profile Jord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 14157
Netherlands
Message 96526 - Posted: 8 Mar 2020, 21:36:47 UTC - in response to Message 96523.  

Not the first time the code has diverged from the documentation, over time.
Gotta wonder though which one it is now. I'm not going to ask.
ID: 96526 · Report as offensive
1 · 2 · Next

Message boards : GPUs : Big ask - auto throttling of CPU tasks when using GPUs?

Copyright © 2020 University of California. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.