Different OpenCL apps runs at the same time

Message boards : Questions and problems : Different OpenCL apps runs at the same time
Message board moderation

To post messages, you must log in.

AuthorMessage
boroda3

Send message
Joined: 24 Jul 12
Posts: 7
Russia
Message 45048 - Posted: 24 Jul 2012, 20:30:56 UTC
Last modified: 24 Jul 2012, 21:19:09 UTC

BOINC manager run OpenCL-based tasks from different projects on the same GPU at the same time, and this dramatically reduces performance.

There are two projects with OpenCL applications: POEM@Home and Einstein@Home (resource share 10000 and 200).
To best performance POEM runs 3 WU simultaneously on the one GPU (0.66CPU + 0.33GPU), or Einstein runs 2 WU on GPU (0.5CPU + 0.5GPU).
In addition to them runs 2 CPU-based tasks on free CPU cores (SETI, MilkyWay, LHC, SAT - don't care).

When applications run separatelly, all is OK:

3 x poem:
Time to run ~66 minutes, CPU total load ~95%, GPU load ~70%, about 8000 credits per hour.

2 x einstein:
Time to run ~125 minutes, CPU total load ~65%, GPU load ~94%, about 500 credits per hour.

But BOINC manager often runs POEM and Einstein application together (1+1). Result: CPU total load 65-80%, GPU total load <=80% (~65-70% by Einstein and less than 6% by POEM), Einstein app runs the same ~120 minutes, but POEM app runs ~180..270 minutes or more. Average 'cost' <1000 cr/h.

Q: How to prevent such mixing but save high performance for each application? Tired of manually split tasks.
(I'm understand that if one app will be set to 1CPU + 1 GPU then problem will not occur, but it is not comfortable.)

BOINC 7.0.28/31, Win7, AMD Phenom X4, AMD HD6850

-------------
Sorry for my bad English, my native language is Russian.
ID: 45048 · Report as offensive
Profile Jord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 15483
Netherlands
Message 45056 - Posted: 25 Jul 2012, 8:25:39 UTC - in response to Message 45048.  

This sounds more like one of the applications doesn't release from memory after its run and went rogue. So you'll have to check which one that does this and complain at their forums that their application should listen to boinc_exit() commands.
ID: 45056 · Report as offensive
boroda3

Send message
Joined: 24 Jul 12
Posts: 7
Russia
Message 45059 - Posted: 25 Jul 2012, 9:23:56 UTC - in response to Message 45056.  
Last modified: 25 Jul 2012, 9:42:22 UTC

This sounds more like one of the applications doesn't release from memory after its run and went rogue.

Not, they runs correctly, but slow. I watched them by Process Explorer.

But I've probably found the cause today but I am not yet sure.
In app_info.xml for POEM were max_ncpus = 1 after my last experiments. But POEM app occupies 0.66 CPU per WU -> ~2 CPU total for 3 x poem.
I tryed to set max_ncpus to 2 and manually tried to mix applications - it not occurs.
I'll keep watching from a few more days.
ID: 45059 · Report as offensive
Richard Haselgrove
Volunteer tester
Help desk expert

Send message
Joined: 5 Oct 06
Posts: 5082
United Kingdom
Message 45060 - Posted: 25 Jul 2012, 10:03:50 UTC

There are many reports, on many message boards, that AMD OpenCL applications are currently running slowly if there isn't sufficient CPU time available.

So far as I can tell, it appears to be a deliberate decision by AMD to enforce 'busy wait' synchronisation in the runtime support bundled with recent catalyst suites. I'm guessing that they found this was the most efficient mode for a single OpenCL program running on an otherwise idle computer: but they didn't consider the typical BOINC user, who tends to load the CPU with other tasks at the same time as OpenCL - many BOINC users describe the high CPU requirement of OpenCL applications as a driver bug.

The Einstein case is different: their application requires CPU support because somes parts of the algorithm aren't suitable for parallel processing on the GPU, and are computed on CPU instead. During that phase of processing, the CPU will be busy with much more real work than the simple busy-wait of OpenCL runtime support - maybe that is what is holding up POEM, when they run together.
ID: 45060 · Report as offensive
boroda3

Send message
Joined: 24 Jul 12
Posts: 7
Russia
Message 45061 - Posted: 25 Jul 2012, 11:09:20 UTC - in response to Message 45059.  

[quote]But I've probably found the cause today but I am not yet sure.

I tryed to set max_ncpus to 2 and manually tried to mix applications - it not occurs.

I'm wrong, max_ncpus = 2 not help. :(
Just now left only one running POEM app and immediately was started Einstein app together with POEM app again.
ID: 45061 · Report as offensive
boroda3

Send message
Joined: 24 Jul 12
Posts: 7
Russia
Message 45062 - Posted: 25 Jul 2012, 13:10:47 UTC - in response to Message 45060.  

So far as I can tell, it appears to be a deliberate decision by AMD to enforce 'busy wait' synchronisation in the runtime support bundled with recent catalyst suites. I'm guessing that they found this was the most efficient mode for a single OpenCL program running on an otherwise idle computer: but they didn't consider the typical BOINC user, who tends to load the CPU with other tasks at the same time as OpenCL - many BOINC users describe the high CPU requirement of OpenCL applications as a driver bug.

The Einstein case is different: their application requires CPU support because somes parts of the algorithm aren't suitable for parallel processing on the GPU, and are computed on CPU instead. During that phase of processing, the CPU will be busy with much more real work than the simple busy-wait of OpenCL runtime support - maybe that is what is holding up POEM, when they run together.

Yes, I understand it. But I can do nothing with AMD drivers.
But I can (or not?) do anything to prevent starting another apps until running app will be fully completed regardless of the proportion of used GPU resources. I just don't know how to do it right. I haven't found any method in BOINC to disable simultaneous running different apps on the one GPU.
I see one bad way now: manually pause one project and resume other, and after a while again switch them on. Is there any other way, not so dummy?
ID: 45062 · Report as offensive
Profile Beyond
Avatar

Send message
Joined: 16 Aug 12
Posts: 39
United States
Message 45313 - Posted: 16 Aug 2012, 21:12:17 UTC - in response to Message 45060.  

So far as I can tell, it appears to be a deliberate decision by AMD to enforce 'busy wait' synchronisation in the runtime support bundled with recent catalyst suites.

Richard, do you know what driver version that this was first instituted?
ID: 45313 · Report as offensive

Message boards : Questions and problems : Different OpenCL apps runs at the same time

Copyright © 2024 University of California.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.