Running too many tasks the require lots of RAM.

Message boards : Questions and problems : Running too many tasks the require lots of RAM.
Message board moderation

To post messages, you must log in.

AuthorMessage
Sandman192

Send message
Joined: 28 Aug 19
Posts: 49
United States
Message 98842 - Posted: 27 May 2020, 4:25:26 UTC

I have a quad-core with no hyper thread and MAX of 2 Gigs of RAM.

I just installed the Linux OS and BOINC.

I hand no problems when I ran 4 low RAM WUs. But after thos WUs where done BOINC started running 1 WU that requires 1 gig to run. What you would think BOINC would do is run 1 1 gig WU and the rest running that needs less RAM to run. NO. Instead, it started running a second 1 gig of RAM WU then another. All of a sudden I'm running 4 1 gig of WUs at the same time. Not to motion the OS requires 800 megs to run.

Just to let you know it does not matter where the WU came from. This one is coming from WCG.
I can not stress this enough that changing the BOINC settings does NOT fix the issue because it's all irrelevant. And I've tried. This is a bug true and true.

Running Linux Mint 8.3, 2 gigs of RAM MAX, 4 cores w/o threads, and BOINC v7.16.5.

P.S. I had a similar issue with a Windows machine that had 16 gigs and 12 cors and it ran WUs that requires 2 gigs each and ran 16 2 gigs of WU and if you do that math 12 WU (1 per core) x 2 is 24 gigs of work, which I do not have 24 gigs of RAM and not to mention the OS of Windows 10. And the hard drive was just thrashing (you could hear it click away) and my computer became unresponsive.
The problem was fixed (well sort of). The site that I got the WUs went down. I mean their site doesn't even exist anymore.

I use to have a 4 gig machine and there were WUs that require 3 gigs to run and only ran 1 3 gigs and ran whatever it cold that didn't go over the 4 gigs on my machine. But now it's not working right.
ID: 98842 · Report as offensive
Profile Dave
Help desk expert

Send message
Joined: 28 Jun 10
Posts: 2538
United Kingdom
Message 98846 - Posted: 27 May 2020, 7:23:18 UTC - in response to Message 98842.  
Last modified: 27 May 2020, 7:24:33 UTC

Worth checking the recommended minimum ram/core for the different projects/application types.
This is not a bug in BOINC but rather reflects that many projects have larger RAM requirements than they used to.

CPDN for instance now suggest a minimum of 2GB/core. There is a model type that has been through the testing programme that requires 5GB/core peak. Now despite that, I can run 4 of them on my laptop that has four cores but only 8GB RAM. This slows them down a lot. Running three isn't too bad as they rarely peak at the same time and there is virtually no performance hit running only two at a time. Those tasks will not run on my ageing desktop which has only 4GB of RAM which is the maximum the MB can take.

You could ask the BOINC developers to add a feature that would stop more than one task that required that much RAM from running or being downloaded at once but I am pretty certain it would also require the projects to write the applications to work with that.

Edit:The setting you need to change is the % of cores that BOINC can use.
ID: 98846 · Report as offensive
Profile Jord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 15480
Netherlands
Message 98858 - Posted: 27 May 2020, 14:33:46 UTC
Last modified: 27 May 2020, 14:52:04 UTC

The project sends an estimated memory usage value to BOINC along with the task. Basically it's determined from this value if the task can run. If the project grossly underestimates the value, where it's way lower than what it actually uses, you get the trouble you describe. But that's hardly BOINC its fault as it can't know what it didn't get provided. That's wholly the project's fault and you should report this to them, that the value they send in rsc_memory_bound is too low.

The values set in BOINC for memory use when idle and in use are used when tasks run, and if tasks run out of the memory restrictions set here BOINC should suspend those tasks. The restrictions only work when Activity is set to "Run based on preferences".

I've tested this with Rosetta. Set memory restrictions to 1% (of 32GB). Two Rosetta Mini tasks immediately suspend with "Waiting for memory":
27/05/2020 16:31:30 | Rosetta@home | [cpu_sched_debug] enforce: task hp100_run0985_fold_SAVE_ALL_OUT_904300_920_0 can't run, too big 1907.35MB > 326.93MB
27/05/2020 16:31:30 | Rosetta@home | [cpu_sched_debug] enforce: task hp100_run1579_relax_SAVE_ALL_OUT_904300_947_0 can't run, too big 1907.35MB > 326.93MB


LOL, now the other tasks are downloaded, all are suspended:
27/05/2020 16:40:44 | Rosetta@home | [cpu_sched_debug] enforce: task fdcc2f249c67f4ad4e008c0ad6d83fac_relax_SAVE_ALL_OUT_939475_17_0 can't run, too big 1716.61MB > 89.04MB
27/05/2020 16:40:44 | Rosetta@home | [cpu_sched_debug] enforce: task Deep_learning_folding_energy_landscape_wt_frags_mutationsG.6MRS._1_V70_fragments_abinitio_SAVE_ALL_OUT_940449_172_0 can't run, too big 1716.61MB > 89.04MB
27/05/2020 16:40:44 | Rosetta@home | [cpu_sched_debug] enforce: task 29d59ee9cec6df38d1c55e76a824d510_relax_SAVE_ALL_OUT_939475_17_0 can't run, too big 1716.61MB > 89.04MB
27/05/2020 16:40:44 | Rosetta@home | [cpu_sched_debug] enforce: task rb_05_27_26972_26532__t000__3_C1_SAVE_ALL_OUT_IGNORE_THE_REST_940541_28_0 can't run, too big 858.31MB > 89.04MB
27/05/2020 16:40:44 | Rosetta@home | [cpu_sched_debug] enforce: task Junior_HalfRoid_design6_cart_COVID-19_SAVE_ALL_OUT_IGNORE_THE_REST_2ba7kz4j_929576_29_0 can't run, too big 1716.61MB > 89.04MB
27/05/2020 16:40:44 | Rosetta@home | [cpu_sched_debug] enforce: task Junior_HalfRoid_design6_cart_COVID-19_SAVE_ALL_OUT_IGNORE_THE_REST_6ct3sa2l_929381_25_0 can't run, too big 1716.61MB > 89.04MB
27/05/2020 16:40:44 | Rosetta@home | [cpu_sched_debug] enforce: task ff_0_721D94cE4e7c7EEA_fragments_abinitio_SAVE_ALL_OUT_940238_436_0 can't run, too big 1716.61MB > 89.04MB
27/05/2020 16:40:44 | Rosetta@home | [cpu_sched_debug] enforce: task ff_0_75F42Ac1B949395A_fragments_abinitio_SAVE_ALL_OUT_940240_436_0 can't run, too big 1716.61MB > 89.04MB
27/05/2020 16:40:44 | Rosetta@home | [cpu_sched_debug] enforce: task c1f8a8e23ad09d1d28ee5590916ff32e_fold_SAVE_ALL_OUT_939475_17_0 can't run, too big 1716.61MB > 89.04MB
27/05/2020 16:40:44 | Rosetta@home | [cpu_sched_debug] enforce: task hp100_run1579_relax_SAVE_ALL_OUT_904300_947_0 can't run, too big 1907.35MB > 89.04MB
27/05/2020 16:40:44 | Einstein@Home | [cpu_sched_debug] enforce: task h1_1589.75_O2C02Cl4In0__O2MDFV2h_VelaJr1_1590.70Hz_444_0 can't run, too big 237.01MB > 89.04MB
These now show they're waiting until memory clears up, as the maximum amount it can still use is 89MB
ID: 98858 · Report as offensive
ProDigit

Send message
Joined: 8 Nov 19
Posts: 718
United States
Message 98860 - Posted: 27 May 2020, 15:22:48 UTC

There are plenty of projects using very little RAM.
Rosetta, LHC, and a few others use quite a bit of RAM. Also QuChemPedIA long uses more RAM than regular.
Some projects allow you to disable large memory WUs (by de-selecting them on your account page on their website).
ID: 98860 · Report as offensive

Message boards : Questions and problems : Running too many tasks the require lots of RAM.

Copyright © 2024 University of California.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.