Rosetta hogging resources

Message boards : Questions and problems : Rosetta hogging resources
Message board moderation

To post messages, you must log in.

AuthorMessage
Robster

Send message
Joined: 4 Jul 11
Posts: 21
United Kingdom
Message 88870 - Posted: 16 Nov 2018, 17:31:22 UTC

Hi

This is an issue I have seen for a while!

I am a member of Seti, climate change and Rosetta projects.

In Boinc manager I can see 10 separate running projects at any one time, 1 is Seti, 1 is climate change and 8 are Rosetta!

This seems grossly unfair, how do I change this?

Robin
ID: 88870 · Report as offensive
Jim1348

Send message
Joined: 8 Nov 10
Posts: 191
United States
Message 88871 - Posted: 16 Nov 2018, 18:23:27 UTC - in response to Message 88870.  

How recently did you add Rosetta? It takes almost a month to stabilize, based on the resource share you have set for each project.

If you want to speed that up to a couple of days, place a cc_config.xml file in your BOINC Data folder.

<cc_config>        
 <options>                
  <rec_half_life_days>1.000000</rec_half_life_days>
  </options>
</cc_config>
ID: 88871 · Report as offensive
Robster

Send message
Joined: 4 Jul 11
Posts: 21
United Kingdom
Message 88872 - Posted: 16 Nov 2018, 19:41:54 UTC - in response to Message 88871.  

HI

I added it ages ago, maybe a couple of years!

Robin
ID: 88872 · Report as offensive
Robster

Send message
Joined: 4 Jul 11
Posts: 21
United Kingdom
Message 88873 - Posted: 16 Nov 2018, 19:49:16 UTC

I cannot find a setting anywhere that would allow me to even out the work load across all three projects!

This is running on a Mac mini, hexacore i7.

Robin
ID: 88873 · Report as offensive
Jim1348

Send message
Joined: 8 Nov 10
Posts: 191
United States
Message 88874 - Posted: 16 Nov 2018, 20:58:50 UTC - in response to Message 88873.  
Last modified: 16 Nov 2018, 21:03:11 UTC

You set the Resource Share in your account settings.

For Rosetta: https://boinc.bakerlab.org/rosetta/prefs.php?subset=project

To find the fraction that each project is using, you add them all up and then divide by each project's resource share.
For example, if you have three projects, all set to 100%, then each project gets 33% of the time (or actually cores).

So if you have 8 cores, and have two projects, setting one to 60% and the other to 20% will give one project 6 cores and the other project 2 cores. But that is an average. The BOINC scheduler jumps around. And some projects send out bad estimates; maybe Rosetta has gotten off track? It sometimes happens.
ID: 88874 · Report as offensive
Profile Dave

Send message
Joined: 28 Jun 10
Posts: 384
United Kingdom
Message 88876 - Posted: 17 Nov 2018, 10:14:09 UTC - in response to Message 88874.  

Also if running CPDN which I guess is what you mean by, "climate change" because of the very long work units, it can mess with the resource share on a short term basis even if over the long term it evens out. (Suspect this not happening with your system.)

If I don't want a project to take up so much of the resources, I just set it to no new tasks for a while.
ID: 88876 · Report as offensive
Robster

Send message
Joined: 4 Jul 11
Posts: 21
United Kingdom
Message 88893 - Posted: 19 Nov 2018, 6:50:09 UTC

Many thanks for the helpful answers.

Certainly the there projects seem to be set to 33.3% each.

But cannot edit that setting, at least anywhere I have looked.

Will try again later.

Robin
ID: 88893 · Report as offensive
Profile Ageless-Away
Volunteer moderator
Project administrator
Avatar

Send message
Joined: 29 Aug 05
Posts: 13290
Netherlands
Message 88896 - Posted: 19 Nov 2018, 8:03:34 UTC - in response to Message 88893.  

Project resources are set via the Project preferences website. For Rosetta, that's at https://boinc.bakerlab.org/rosetta/prefs.php?subset=project

If you're using an Account Manager (BAM, GridRepublic, Science United) you'll have to set this through there.
Jord
Please do not private message me for tech support. Use the forums for that. Tech PMs will be ignored!
ID: 88896 · Report as offensive
David Ball

Send message
Joined: 2 Dec 06
Posts: 69
United States
Message 89659 - Posted: 17 Jan 2019, 12:08:30 UTC

When I want to limit how many task Rosetta can run at once, I go to the \projects\boinc.bakerlab.org_rosetta subdirectory of the BOINC program data directory and add an app_config.xml file with the following in it:

<!-- Rosetta -->

<app_config>
<project_max_concurrent>4</project_max_concurrent>
</app_config>


The above will prevent BOINC from running more than 4 Rosetta tasks (aka workunits) at once. It does not effect how many many Rosetta workunits are downloaded from the server - just prevents more than 4 from running at a time. You can change the 4 to any number you want that is 1 or greater up to the number of cores you have BOINC running on.

NOTE: if you already have an app_config.xml, you will need to read the docs to find out how to merge them.

On Windows you can use advanced view on the interface and select from the top menu "Options" and "read config files" to get BOINC to start using the setting in the app_config.xml. BOINC will also place an error message in the log file if it finds an error in the file. I haven't run BOINC on Linux for several years so I can't be sure of the command line to cause it to read the config files, but I think it's

boinccmd --read_cc_config

The --read_cc_config option will Reread the configuration files, to include cc_config.xml and any app_config.xml existing in the project folders.


BTW, my contact info is out of date so posting in the forum is the only way to get a message to me.
ID: 89659 · Report as offensive
Profile marmot
Avatar

Send message
Joined: 16 Sep 13
Posts: 77
United States
Message 90001 - Posted: 11 Feb 2019, 6:09:43 UTC - in response to Message 88874.  


To find the fraction that each project is using, you add them all up and then divide by each project's resource share.
For example, if you have three projects, all set to 100%, then each project gets 33% of the time (or actually cores).


This doesn't actually work.

There are many projects that force down work and flood the cache, ignoring their resource share.

Even have a seen couple of projects that do this when set to 0 (Asteroids and DHEP, both guilty).

Assigning projects to cores and not a general cache would give more definitive control.
ID: 90001 · Report as offensive
Profile Dave

Send message
Joined: 28 Jun 10
Posts: 384
United Kingdom
Message 90005 - Posted: 11 Feb 2019, 8:53:12 UTC - in response to Message 90001.  

To find the fraction that each project is using, you add them all up and then divide by each project's resource share.
For example, if you have three projects, all set to 100%, then each project gets 33% of the time (or actually cores).




This doesn't actually work.

My understanding is that it does work but only if you look at the averages over a long period of time. In the short term many things can mess it up such as the very long tasks of cpdn (Some will be well over 100 days unless you have a fast computer or if I were to use my ageing netbook, over a year!) or if you have downloaded tasks with a very short deadline these will be prioritised.

If keeping a balance in the short or even medium term is what you are after, yes either the changes to BOINC you suggest or micro managing will be needed.
ID: 90005 · Report as offensive
Richard Haselgrove
Volunteer moderator
Volunteer tester
Help desk expert

Send message
Joined: 5 Oct 06
Posts: 3383
United Kingdom
Message 90008 - Posted: 11 Feb 2019, 9:07:22 UTC - in response to Message 90001.  

There are many projects that force down work and flood the cache, ignoring their resource share.
Projects servers can't actually 'force down work' - the firewall and NAT (Network Address Translation) in your router would prevent that.

Project servers can only communicate with your computer in the form of a reply to a request made by your computer. I'd be interested to see the request/reply pair for an event like this - the <sched_op_debug> event log flag is useful for this:

11/02/2019 08:45:02 | SETI@home | Sending scheduler request: To report completed tasks.
11/02/2019 08:45:02 | SETI@home | Reporting 5 completed tasks
11/02/2019 08:45:02 | SETI@home | Requesting new tasks for NVIDIA GPU
11/02/2019 08:45:02 | SETI@home | [sched_op] NVIDIA GPU work request: 7683.10 seconds; 0.00 devices
11/02/2019 08:45:06 | SETI@home | Scheduler request completed: got 8 new tasks
11/02/2019 08:45:06 | SETI@home | [sched_op] estimated total NVIDIA GPU task duration: 7907 seconds
Note that the amounts requested and received are expressed in seconds - duration requested and runtime estimated. They should match.
ID: 90008 · Report as offensive
mmonnin

Send message
Joined: 1 Jul 16
Posts: 141
United States
Message 90016 - Posted: 11 Feb 2019, 16:26:01 UTC

DHEP is an oddball project that sends users a long task that is basically just a placeholder task to take a CPU slot in BOINC while all the work segments are done in the background. BM says it needs a task and it'll get one regardless of the queue buffer.
ID: 90016 · Report as offensive
Richard Haselgrove
Volunteer moderator
Volunteer tester
Help desk expert

Send message
Joined: 5 Oct 06
Posts: 3383
United Kingdom
Message 90018 - Posted: 11 Feb 2019, 16:30:59 UTC - in response to Message 90016.  

Isn't that the mechanism designed for 'non compute intensive' tasks? (Like the old Quake Catcher application, which would simply sit in the background doing very little, waiting for the earth to move)
ID: 90018 · Report as offensive
mmonnin

Send message
Joined: 1 Jul 16
Posts: 141
United States
Message 90020 - Posted: 11 Feb 2019, 16:36:46 UTC

Maybe, I don't know how the QCN tasks worked but DHEP is CPU intensive. DHEP tasks communicate with a separate DHEP server which then sends task/credit info to a BOINC server vs the client communicating directly with a BOINC server.
ID: 90020 · Report as offensive
Profile marmot
Avatar

Send message
Joined: 16 Sep 13
Posts: 77
United States
Message 90031 - Posted: 12 Feb 2019, 3:37:48 UTC - in response to Message 90008.  
Last modified: 12 Feb 2019, 4:16:51 UTC

There are many projects that force down work and flood the cache, ignoring their resource share.
Projects servers can't actually 'force down work' - the firewall and NAT (Network Address Translation) in your router would prevent that.

Project servers can only communicate with your computer in the form of a reply to a request made by your computer.


When I said "force down work" I mean they send more work than their 0 or 1 resource share choice would allow, not that they send work without request (you've seen me on here for years... I know better than this).
DHEP (set to 0 resource share), once the high priority projects are out of work and it alone has WU's, fills the cache by sending down 6x ~19 day WU "place holders" so that no other project will see available work cache.


I'd be interested to see the request/reply pair for an event like this - the <sched_op_debug> event log flag is useful for this:

11/02/2019 08:45:02 | SETI@home | Sending scheduler request: To report completed tasks.
11/02/2019 08:45:02 | SETI@home | Reporting 5 completed tasks
11/02/2019 08:45:02 | SETI@home | Requesting new tasks for NVIDIA GPU
11/02/2019 08:45:02 | SETI@home | [sched_op] NVIDIA GPU work request: 7683.10 seconds; 0.00 devices
11/02/2019 08:45:06 | SETI@home | Scheduler request completed: got 8 new tasks
11/02/2019 08:45:06 | SETI@home | [sched_op] estimated total NVIDIA GPU task duration: 7907 seconds
Note that the amounts requested and received are expressed in seconds - duration requested and runtime estimated. They should match.


Will look for an opportunity to catch DHEP with <sched_op_debug> and also interested in Asteroid's results.

The <max_concurrent> tag in app_config was the best answer for the OP's question. If that's not enough (because the project is dominating the work cache) then suspending a single task will lessen project dominance of the cache as they decay off until you manually allow them to grab more work.
ID: 90031 · Report as offensive
jglrogujgv

Send message
Joined: 6 Jul 18
Posts: 49
Barbados
Message 90131 - Posted: 14 Feb 2019, 16:38:53 UTC - in response to Message 90031.  

If what you're saying is true then it sounds like DHEP is a sneaky tail that has found a way to wag the dog. It sounds like it's a rogue project that is purposefully and knowingly committing fraud and theft of computing resources. If so then people need to file a complaint with the authorities in whatever country DHEP originates from instead of asking BOINC devs to thwart the crook, not that I blame you for asking here first.

Just curious... do any of the DHEP project staff go by names similar to Donald, Junior, Ivanka or Jared?
ID: 90131 · Report as offensive

Message boards : Questions and problems : Rosetta hogging resources

Copyright © 2019 University of California. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.