Multiple BOINC projects; individual daily schedules??

Message boards : Questions and problems : Multiple BOINC projects; individual daily schedules??
Message board moderation

To post messages, you must log in.

AuthorMessage
GeneAZ

Send message
Joined: 28 Jun 14
Posts: 27
United States
Message 86784 - Posted: 30 Jun 2018, 16:23:47 UTC

Linux system, boinc 7.6.33
I have several boinc projects active with their respective resource shares. What I would like to do is restrict one of those projects to run only at night. There are "daily schedule" options in the computing preferences tab but they seem to apply to all boinc projects on my host and I don't see a way to use different schedules for different projects.

At the moment my host is a "home" location for all projects. If I change one of the projects to a "work" location (for example) would that allow different "daily schedule" options? I don't see that option in boinc manager's computing preferences but maybe that's because I don't have anything besides a "home" location. I can try changing one project to a "work" location and see if that opens additional daily schedule options. Is this a good idea?

What about running a second boinc client on this host? (<allow_multiple_clients> = 1 in cc_config.xml) Could I then attach only the "night" project and set the daily schedule parameters as desired? Is this the best (or only?) way to do what I want? I am aware of the requirement to run the boinc clients in unique data directories.

Thanks for any suggestions that are offered.
ID: 86784 · Report as offensive
Richard Haselgrove
Volunteer tester
Help desk expert

Send message
Joined: 5 Oct 06
Posts: 5077
United Kingdom
Message 86788 - Posted: 30 Jun 2018, 18:19:46 UTC - in response to Message 86784.  

Setting separate locations won't help you with this one, I'm afraid.

Using two different clients would do it, though. You'd need to make sure that they didn't both try to use the same resource at the same time (which would work, but slow everything down). It's a little tricky to set up (especially setting up two different Managers to control the two clients), but it can all be done - I think I have a crib sheet somewhere.
ID: 86788 · Report as offensive
Profile Dave
Help desk expert

Send message
Joined: 28 Jun 10
Posts: 2515
United Kingdom
Message 86793 - Posted: 30 Jun 2018, 20:04:26 UTC - in response to Message 86788.  

That would be of interest to me too. I haven't even been able to get a native linux BOINC to run alongside a windows version running under WINE.
ID: 86793 · Report as offensive
Richard Haselgrove
Volunteer tester
Help desk expert

Send message
Joined: 5 Oct 06
Posts: 5077
United Kingdom
Message 86796 - Posted: 30 Jun 2018, 20:44:13 UTC

OK, this is what I set up some time ago. I'm exclusively Windows, so both my clients are Windows: I don't know anything about how WINE works, so you're on your own - but this may help get you started.

The basic BOINC installation is entirely as normal. I choose to put the programs in D:\BOINC, and the data in D:\BOINCdata, but that's entirely arbitrary.

The only change in this primary BOINC installation is that I have the line

        <allow_multiple_clients>1</allow_multiple_clients>
in the options section of cc_config.xml in that primary data folder. The line "Config: allow multiple clients" appears in the event log at startup.

I have two batch files on my desktop. They are

Start client #2.bat
D:\BOINC\boinc.exe --allow_multiple_clients --redirectio --detach_console --gui_rpc_port 31418 --dir D:\BOINCdata2

Start Manager #2.bat
start D:\BOINC\boincmgr.exe /m /n 127.0.0.1 /g 31418 /p password

Now to see if they both still work :-)))

Yay!

30/06/2018 21:28:36 | | Starting BOINC client version 7.10.2 for windows_x86_64
30/06/2018 21:28:36 | | log flags: file_xfer, sched_ops, task, cpu_sched, sched_op_debug
30/06/2018 21:28:36 | | Libraries: libcurl/7.47.1 OpenSSL/1.0.2g zlib/1.2.8
30/06/2018 21:28:36 | | Data directory: D:\BOINCdata2
30/06/2018 21:28:36 | | Running under account Richard Haselgrove
30/06/2018 21:28:36 | | No usable GPUs found
30/06/2018 21:28:36 | | Version change (7.7.0 -> 7.10.2)
That looks like a data directory I haven't used in a while! Note that the --dir specified for starting the client has to be separate from your primary data folder.

I'm trying to find where I set "don't use GPUs" in this client - it's not obvious. The primary client is happily using all three GPUs in the machine, for three different projects.
ID: 86796 · Report as offensive
GeneAZ

Send message
Joined: 28 Jun 14
Posts: 27
United States
Message 86799 - Posted: 1 Jul 2018, 6:17:57 UTC

@Richard
O.K., thanks. I have all projects in /home/gene/BOINC/projects/*** directories so it looks like the straightforward approach is to move the ~BOINC/projects/asteroidsathome directory (the project I want to run only at night) over to a new /home/gene/BOINC2/** directory. There are several .xml files in the main BOINC directory (like sched_request_asteroids...) that look like they should be removed, or copied to the BOINC2 directory. Here's my thinking on that: do a "projects -> remove" in the current BOINC directory; then switch to the second data directory and do a "tools -> add project". I would expect that sequence to clean out anything from the current data directory and install needed files in the second data directory. What about "cc_config.xml" ? Duplicate that file in both primary and secondary boinc data directories? It has no project specific tags in it. If other files are needed I hope that boincmgr will give useful/informative error messages to point me toward a fix.
...Give me a day or two to step through this process. And hope not to crash the boinc client(s) along the way...
ID: 86799 · Report as offensive
Profile Jord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 15477
Netherlands
Message 86800 - Posted: 1 Jul 2018, 6:50:21 UTC - in response to Message 86799.  
Last modified: 1 Jul 2018, 8:07:11 UTC

Your BOINC installer installs to home/gene/BOINC and sets up all the correct paths for the client to follow afterwards. You cannot indiscriminately move things from the data directory to other places, that will cause errors.
And it's not necessary. If you want to go the multiple clients route, they can all use the same data directory.

Even if you had managed to run a hypothetical 18 clients, they can all run from the same data directory. It's not the data directory that's the problem here, it's the setting up of the multiple clients and their correct ports, in a timely fashion.

ID: 86800 · Report as offensive
Richard Haselgrove
Volunteer tester
Help desk expert

Send message
Joined: 5 Oct 06
Posts: 5077
United Kingdom
Message 86803 - Posted: 1 Jul 2018, 7:07:31 UTC - in response to Message 86800.  

If you want to go the multiple clients route, they can all use the same data directory.
Are you really, really sure about that? Each client will be working with its own client_state.xml, and they'll all have the same name.
ID: 86803 · Report as offensive
Profile Jord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 15477
Netherlands
Message 86804 - Posted: 1 Jul 2018, 7:35:32 UTC - in response to Message 86803.  

Hmmm, thinking about it, perhaps not. But at least, be careful when moving parts of the data directory to elsewhere, do run that by a Linux guru as to me it sounds like a sure way to break things and (lots of) lost work.
ID: 86804 · Report as offensive
Profile Dave
Help desk expert

Send message
Joined: 28 Jun 10
Posts: 2515
United Kingdom
Message 86806 - Posted: 1 Jul 2018, 13:00:00 UTC - in response to Message 86796.  

Thanks Richard,
I will have a play when I get enough time later in the week.
Dave
ID: 86806 · Report as offensive
GeneAZ

Send message
Joined: 28 Jun 14
Posts: 27
United States
Message 86821 - Posted: 2 Jul 2018, 6:57:04 UTC

Good progress, so far, and no disasters. I was able to get a second boinc client and boinc manager up and running. I made a second boinc data directory and copied the account*, job_log*, master*, sched_request*, sched_reply*, and statistics* files for the "night" project from the primary boinc data directory. And copied the full /boinc/projects/asteroids*/ directory to the second directory. Then did a Project->Remove in the first directory. Then tried starting up the second boinc/boincmgr instance more or less along the lines that Richard had posted. The "boinc" client seemed to start up but the "boincmgr" failed to connect. Twiddled with the port number and got past that roadblock on the third try. The only glitch in that process was that the boinc client (apparently) tried to resume suspended tasks and created some slot directory lock discrepancies - two applications contending for the same slot and one going into "postponed" status. Used a bit of "root" magic to sort things out and kill one of the contending applications. (I have the work unit ID's of those tasks and I'll want to check their validation status just to confirm they were not affected.)
I got a "new" CPID for the project that was moved. Not a big surprise and I hope a "merge computers" will resolve that. I have not yet tried a daily schedule setup. The "proof-test" will be whether I can shut down both boinc instances and restart them with no ill effects. That's the "to do" list for tomorrow.
Even though this project has a resource share of 1 it is the only project in its boinc manager instance so it looks like it is entitled to 100% resource share. I'll use settings in app_config.xml and (eventually) daily schedule to provide the resource limits I want.

>>For the boinc "wish list:" enable daily schedule settings for each project instead of just a global host setting...
ID: 86821 · Report as offensive
GeneAZ

Send message
Joined: 28 Jun 14
Posts: 27
United States
Message 86900 - Posted: 5 Jul 2018, 18:23:39 UTC

Here are the "scripts" I'm using for the two boinc clients (and respective boinc managers):
...for the "primary" client, i.e. default gui/rpc port access.
#!/bin/bash
exec /usr/bin/boinc --dir /home/gene/BOINC --redirectio --allow_multiple_clients &
sleep 3
exec /usr/bin/boincmgr -d /home/gene/BOINC -e /home/gene/BOINC -n localhost -m &


And, for the "extra" client, i.e. custom gui/rpc port access.
#! /bin/bash
exec /usr/bin/boinc --dir /home/gene/BOINC2 --redirectio --gui_rpc_port 31420 --allow_multiple_clients &
sleep 3
exec /usr/bin/boincmgr -d /home/gene/BOINC2 -e /home/gene/BOINC2 -n localhost -m -g 31420 &


These are Linux scripts, of course, and for Windows systems some different procedure is used.
Obviously I have two boinc data directories, with really imaginative names BOINC and BOINC2. The only glitch I ran into is the initiation order. If I start the "primary" boinc client first (and its matching manager) then the "extra" client will start but a boincmgr CANNOT connect to it! BUT... if I start the "extra" client first then the "primary" client will start and the boincmg WILL connect to it.

The "multiple_clients" parameter options are included in both scripts (above). The respective cc_config.xml files in both directories have the <allow_multiple_clients>1</allow_multiple_clients> so it's possible the script options are not necessary. I haven't tested that, and probably never will since working scripts are in hand.

I noticed that the extra client suspends work when the primary client is running. Reason = CPU busy. I had a "suspend when non-BOINC usage is above 50%" setting in computing preferences. It seems as if the primary boinc client usage is considered as "non-BOINC" in this context. Raising that level to 90% removes that constraint. I guess it makes sense that each boinc client looks at kernel values for CPU utilization and has no way of knowing that it is another boinc instance running apps that should not really count as non-BOINC use.

So, dual clients running and daily schedule values in use for the low-priority project. A more thoughtful approach to the changes in the data directories might have avoided the creation of a new computer ID (since merged with the original one). I suspect that if I searched these message boards I would find the right method.

...and no work units lost or mangled along the way... :)
ID: 86900 · Report as offensive
jglrogujgv

Send message
Joined: 6 Jul 18
Posts: 49
Barbados
Message 86909 - Posted: 6 Jul 2018, 16:30:11 UTC

Two simultaneously running clients and managers is one solution but it lacks simplicity and elegance. Also, to change a project from day processing to night processing involves creating a new host (or adding a project) and deleting an old host (or removing a project) which isn't hard but it creates ripples down stream. If you look at it as a timing problem rather than a distributed computing problem the solution becomes more clear.

I would have used a couple of cron jobs and the GUI RPC interface available via boinccmd to suspend/resume projects at specified times. It requires only 1 client and 1 manager.

Suppose you have projects A, B, C, D and E and you want to run only A and B between 7 AM and 6 PM and only C, D and E between 6 PM and 7 AM. Accomplish that with 2 cron jobs and 2 bash scripts: one named morn.sh and another named nite.sh. Let's look at the bash scripts first, the cron jobs further down.

The morn cron job runs at 7 AM and it suspends projects C, D, and E and resumes projects A and B.
Nite cron job runs at 6 PM and it suspends A and B and resumes C, D and E.

A cron job is essentially a bash script that you tell the cron utility, via a crontab, to run at a specified time or times. The script(s) we need merely calls boinccmd 5 times, once for each of the 5 hypothetical projects. Morn.sh looks something like the code below where password is the contents of your gui_rpc_auth.cfg file:

#!/usr/local/bin/bash

# resume A, B
boinccmd --passwd <password>  --project A <project A's project URL>  resume
boinccmd --passwd <password>  --project A <project A's project URL>  resume
# suspend C, D, E
boinccmd --passwd <password>  --project C <project C's project URL>  suspend
boinccmd --passwd <password>  --project D <project D's project URL>  suspend
boinccmd --passwd <password>  --project E <project E's project URL>  suspend

Nite script is similar but opposite. Note the simplicity.... if you want to change project A from day to night processing then just edit the scripts so that it resumes in nite.sh and suspends in morn.sh. I dunno, seems less cumbersome than adding and deleting projects and dealing with the downstream ripples but tastes vary.

WARNING: You might have to adjust execute permissions on boinccmd to allow you to run it. No, you cannot simply preface each of the commands in the script with sudo for security reasons. Use:

sudo chmod a+x /usr/bin/boinccmd


That covers the scripts. Now to setup the cron jobs. I am a little rusty on cron jobs so forgive me if I leave out some details but there are numerous excellent tutorials available on the 'net so why re-invent the wheel. The trick is in the crontabs. You would need 2 crontabs: one to run morn.sh at 6 AM and one to run nite.sh at 7 PM. Crontabs use the 24 hour clock so 7 AM is 7, 6 PM is 18.

* 6 * * *  /usr/bin/bash  /home/<username>/bin/morn.sh
* 18 * * *  /usr/bin/bash  /home/<username>/bin/nite.sh

Note the above code snippet assumes you have the morn.sh and nite.sh saved in /home/<username>/bin which in your case might be /home/gene/bin.

See this tutorial for a good explanation on how the asterisks in the crontabs work. Note that it's not necessary to use the vi editor as suggested in that tutorial.

With appropriate crontabs and scripts you can suspend/resume projects on a very intricate schedule. For example, you can schedule as discussed in the above example on week days but on weekends do something entirely different.
ID: 86909 · Report as offensive
jglrogujgv

Send message
Joined: 6 Jul 18
Posts: 49
Barbados
Message 86911 - Posted: 7 Jul 2018, 3:37:30 UTC

Additional info and a correction to my previous post:

I used the term crontab incorrectly to refer to what should be referred to as "entries in a crontab". A crontab is a plain text file that contains a table (a list) of cron jobs. Use the crontab -e command to create/edit the crontab. Note the command does not include a filename because there is only 1 crontab per user. A crontab can have as many entries as you wish.

The crontab entries I gave in my previous post would work but they would do a lot of unnecessary work as they would run every minute of hour 6 and hour 18 due to the first asterisk in each entry. The first asterisk in each entry should be an integer between 0 and 59 inclusive. For example:
0 6 * * *  /usr/bin/bash  /home/<username>/bin/morn.sh
0 18 * * *  /usr/bin/bash  /home/<username>/bin/nite.sh
ID: 86911 · Report as offensive
GeneAZ

Send message
Joined: 28 Jun 14
Posts: 27
United States
Message 86924 - Posted: 7 Jul 2018, 23:48:30 UTC

@jgl...
I like your idea of cron scheduling. It is a lot easier to experiment with times and projects without the worry of losing work units or, as I ran into, inadvertently creating a duplicate computer id. I'll try it with the two boinc clients currently running. There are "root" cron jobs active, for logrotate and such, that should be useful as examples. When in doubt read the man!
And thanks for joining this boinc message board, perhaps just to comment on this thread.
ID: 86924 · Report as offensive
jglrogujgv

Send message
Joined: 6 Jul 18
Posts: 49
Barbados
Message 86926 - Posted: 8 Jul 2018, 0:52:52 UTC - in response to Message 86924.  

Thanks but I've begun to wonder what havoc my approach might create for BOINC's scheduler. It might eventually put BOINC into a state where it won't fetch work for some of your projects. There might be an easy fix for that or maybe not, I don't know. The two-client approach avoids that entire issue.
ID: 86926 · Report as offensive

Message boards : Questions and problems : Multiple BOINC projects; individual daily schedules??

Copyright © 2024 University of California.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.