I agree, you need to sort out your Internet first before tackling other issues.

I don't see any output from the work_fetch_debug logging option. But that is because you haven't successfully contacted any scheduler yet because of the connection issues.

Your BOINC or OS also has some issues obviously as your screenshot of the logging options is truncated.

I would look into your KDE Window Manager and its environment because it is not drawing all the elements of the Manager.

I agree, you need to sort out your Internet first before tackling other issues.

I don't see any output from the work_fetch_debug logging option. But that is because you haven't successfully contacted any scheduler yet because of the connection issues.

Your BOINC or OS also has some issues obviously as your screenshot of the logging options is truncated.

I would look into your KDE Window Manager and its environment because it is not drawing all the elements of the Manager.

That’s not necessarily a problem. My Ubuntu system also truncates the menu and has a hidden scroll bar to take you down to the rest of the options.

I agree, you need to sort out your Internet first before tackling other issues.

I don't see any output from the work_fetch_debug logging option. But that is because you haven't successfully contacted any scheduler yet because of the connection issues.

Your BOINC or OS also has some issues obviously as your screenshot of the logging options is truncated.

I would look into your KDE Window Manager and its environment because it is not drawing all the elements of the Manager.

I have successfully contacted the scheduler. It has decided no project is worthy enough.

Wed 08 Jul 2020 05:35:00 PM CDT | Cosmology@Home | Sending scheduler request: To report completed tasks.
Wed 08 Jul 2020 05:35:00 PM CDT | Cosmology@Home | Reporting 1 completed tasks
Wed 08 Jul 2020 05:35:00 PM CDT | Cosmology@Home | Not requesting tasks: don't need (not highest priority project)
Wed 08 Jul 2020 05:35:03 PM CDT | Cosmology@Home | Scheduler request completed
Wed 08 Jul 2020 05:45:07 PM CDT | LHC@home | Sending scheduler request: Requested by project.
Wed 08 Jul 2020 05:45:07 PM CDT | LHC@home | Not requesting tasks: don't need (not highest priority project)
Wed 08 Jul 2020 05:45:11 PM CDT | LHC@home | Scheduler request completed
Wed 08 Jul 2020 05:59:49 PM CDT | Cosmology@Home | Computation for task wu_070720_223748_1_0_0 finished
Wed 08 Jul 2020 05:59:49 PM CDT | Cosmology@Home | Starting task wu_070720_223608_0_0_0
Wed 08 Jul 2020 05:59:51 PM CDT | Cosmology@Home | Started upload of wu_070720_223748_1_0_0_r1481457144_0
Wed 08 Jul 2020 05:59:51 PM CDT | Cosmology@Home | Started upload of wu_070720_223748_1_0_0_r1481457144_1
Wed 08 Jul 2020 05:59:55 PM CDT | Cosmology@Home | Finished upload of wu_070720_223748_1_0_0_r1481457144_1
Wed 08 Jul 2020 05:59:55 PM CDT | Cosmology@Home | Finished upload of wu_070720_223748_1_0_0_r1481457144_0
Wed 08 Jul 2020 05:59:55 PM CDT | Cosmology@Home | Started upload of wu_070720_223748_1_0_0_r1481457144_2
Wed 08 Jul 2020 05:59:55 PM CDT | Cosmology@Home | Started upload of wu_070720_223748_1_0_0_r1481457144_3
Wed 08 Jul 2020 05:59:59 PM CDT | Cosmology@Home | Finished upload of wu_070720_223748_1_0_0_r1481457144_2
Wed 08 Jul 2020 05:59:59 PM CDT | Cosmology@Home | Finished upload of wu_070720_223748_1_0_0_r1481457144_3
Wed 08 Jul 2020 05:59:59 PM CDT | Cosmology@Home | Started upload of wu_070720_223748_1_0_0_r1481457144_4
Wed 08 Jul 2020 05:59:59 PM CDT | Cosmology@Home | Started upload of wu_070720_223748_1_0_0_r1481457144_5
Wed 08 Jul 2020 06:00:01 PM CDT | Cosmology@Home | Finished upload of wu_070720_223748_1_0_0_r1481457144_4
Wed 08 Jul 2020 06:00:01 PM CDT | Cosmology@Home | Finished upload of wu_070720_223748_1_0_0_r1481457144_5
Wed 08 Jul 2020 06:25:00 PM CDT | | Suspending computation - computer is in use
Wed 08 Jul 2020 06:28:45 PM CDT | | Re-reading cc_config.xml
Wed 08 Jul 2020 06:28:45 PM CDT | | Config: GUI RPCs allowed from:
Wed 08 Jul 2020 06:28:45 PM CDT | | log flags: file_xfer, sched_ops, task, work_fetch_debug
Wed 08 Jul 2020 06:28:45 PM CDT | | [work_fetch] Request work fetch: Core client configuration
Wed 08 Jul 2020 06:29:11 PM CDT | LHC@home | update requested by user
Wed 08 Jul 2020 06:29:11 PM CDT | | [work_fetch] Request work fetch: project updated by user
Wed 08 Jul 2020 06:29:11 PM CDT | | [work_fetch] ------- start work fetch state -------
Wed 08 Jul 2020 06:29:11 PM CDT | | [work_fetch] target work buffer: 8640.00 + 604800.00 sec
Wed 08 Jul 2020 06:29:11 PM CDT | | [work_fetch] --- project states ---
Wed 08 Jul 2020 06:29:11 PM CDT | Citizen Science Grid | [work_fetch] REC 0.000 prio -0.000 can request work
Wed 08 Jul 2020 06:29:11 PM CDT | Cosmology@Home | [work_fetch] REC 544.946 prio -1.299 can request work
Wed 08 Jul 2020 06:29:11 PM CDT | Einstein@Home | [work_fetch] REC 657.591 prio -1.538 can request work
Wed 08 Jul 2020 06:29:11 PM CDT | LHC@home | [work_fetch] REC 530.829 prio -1.242 can request work
Wed 08 Jul 2020 06:29:11 PM CDT | Rosetta@home | [work_fetch] REC 393.460 prio -0.920 can request work
Wed 08 Jul 2020 06:29:11 PM CDT | World Community Grid | [work_fetch] REC 438.214 prio -1.025 can request work
Wed 08 Jul 2020 06:29:11 PM CDT | | [work_fetch] --- state for CPU ---
Wed 08 Jul 2020 06:29:11 PM CDT | | [work_fetch] shortfall 4562751.58 nidle 0.00 saturated 22834.00 busy 0.00
Wed 08 Jul 2020 06:29:11 PM CDT | Citizen Science Grid | [work_fetch] share 0.167
Wed 08 Jul 2020 06:29:11 PM CDT | Cosmology@Home | [work_fetch] share 0.167
Wed 08 Jul 2020 06:29:11 PM CDT | Einstein@Home | [work_fetch] share 0.167
Wed 08 Jul 2020 06:29:11 PM CDT | LHC@home | [work_fetch] share 0.167
Wed 08 Jul 2020 06:29:11 PM CDT | Rosetta@home | [work_fetch] share 0.167
Wed 08 Jul 2020 06:29:11 PM CDT | World Community Grid | [work_fetch] share 0.167
Wed 08 Jul 2020 06:29:11 PM CDT | | [work_fetch] ------- end work fetch state -------
Wed 08 Jul 2020 06:29:11 PM CDT | LHC@home | [work_fetch] request: CPU (0.00 sec, 0.00 inst)
Wed 08 Jul 2020 06:29:12 PM CDT | LHC@home | Sending scheduler request: Requested by user.
Wed 08 Jul 2020 06:29:12 PM CDT | LHC@home | Not requesting tasks: don't need (not highest priority project)
Wed 08 Jul 2020 06:29:17 PM CDT | LHC@home | Scheduler request completed
Wed 08 Jul 2020 06:29:17 PM CDT | | [work_fetch] Request work fetch: RPC complete
Wed 08 Jul 2020 06:29:23 PM CDT | | [work_fetch] Request work fetch: Backoff ended for LHC@home

---

According to this line, BOINC thinks that it has quite enough work already:

Wed 08 Jul 2020 06:29:11 PM CDT | | [work_fetch] shortfall 4562751.58 nidle 0.00 saturated 22834.00 busy 0.00

If I am reading this wrong, hopefully someone will correct me. The following line looks to me like you have your buffers set to 0.1 for Store at least __ days of work and 7.0 for Store up to an additional __ days of work.

Wed 08 Jul 2020 06:29:11 PM CDT | | [work_fetch] target work buffer: 8640.00 + 604800.00 sec

Now, the following line shows that you have 0 idle instances (nidle 0.00). That should be correct as Cosmology's website lists 21 tasks in progress for your machine (14 single thread tasks and 7 multi-thread tasks) at the time of my writing this message.

Wed 08 Jul 2020 06:29:11 PM CDT | | [work_fetch] shortfall 4562751.58 nidle 0.00 saturated 22834.00 busy 0.00

As long as the client considers you having that 0.1 minimum days of work and 0 idle instances, it will not request work from any project. I would suggest if you want the client to get more tasks, then you should change the 0.1 to 0.5 and the 7.0 to 1.0 (or something smaller than 7.0). Right now with your current settings, once the buffer drops low enough to request work, it will get 7.1 days worth of work from the first project it contacts so long as that project allows that much work and has a deadline longer than 7 days.

I agree with BobCat: these two lines together explain the 'problem'.

Wed 08 Jul 2020 06:29:11 PM CDT | | [work_fetch] target work buffer: 8640.00 + 604800.00 sec
Wed 08 Jul 2020 06:29:11 PM CDT | | [work_fetch] shortfall 4562751.58 nidle 0.00 saturated 22834.00 busy 0.00

Any time the first entry for 'target work buffer' is less than 'saturated', BOINC will say that no work is needed, and won't fetch any by itself.

Set the first figure for 'store at least' to a large enough figure to tide you over the gaps in your internet connection: add a small 'additional' figure if you wish, to tide you over any unexpected problems. But keep the two figures together well below the minimum deadline for any of your projects. BOINC will run much more smoothly then.

I agree with BobCat: these two lines together explain the 'problem'.

Wed 08 Jul 2020 06:29:11 PM CDT | | [work_fetch] target work buffer: 8640.00 + 604800.00 sec
Wed 08 Jul 2020 06:29:11 PM CDT | | [work_fetch] shortfall 4562751.58 nidle 0.00 saturated 22834.00 busy 0.00

Any time the first entry for 'target work buffer' is less than 'saturated', BOINC will say that no work is needed, and won't fetch any by itself.

Set the first figure for 'store at least' to a large enough figure to tide you over the gaps in your internet connection: add a small 'additional' figure if you wish, to tide you over any unexpected problems. But keep the two figures together well below the minimum deadline for any of your projects. BOINC will run much more smoothly then.

It was empty for 5 days. Absolute nothing in the work queue.

Every attempt at manually pulling down work via Update claimed every project was "not the highest priority"

I appreciate everyone trying to help, but this isn't a 1&Done tweak something situation. This is a scheduler bug. It's actually multiple bugs because the manual Update button should be forcing a retrieval if there is any space in the queue. Instead, for 5 days, with absolutely nothing in the queue, it did this.



Those were me pressing the Update button for each project with absolutely nothing in the queue. "Not highest priority project" is returned for each one I connected with and BOINC refused to pull anything down even though the queue was completely empty. The queue had been empty for days. Not one task to work on.

I've not written one line of code for BOINC. I've written an awful lot of code over my 30+ years and even written a number of books about writing code:
http://theminimumyouneedtoknow.com/
a few of them won awards.

I'm not saying that to sound uppity, I'm saying it so people understand, I know this is a bug. The scheduler is not handling zero properly. It ass-u-me s an always up Internet connection _and_ that whatever it deems the highest priority project will always have work units to dole out. It doesn't temporarily change project priority when:

A) the Update button is pressed
B) "The Most Holy" project, in its eyes, hands out nadda

There was not a single entry in the work queue for what I believe to be 5 days (I didn't keep exact track. More than 3 less than 7, could have been 6 or 4 as well). Adding insult to injury, the message in the event log doesn't bother to tell you what it thinks the highest priority project is. That's a bug all by itself.

I understand that everyone wants it to be "just a hiccup" and "just a settings tweak," but this is an algorithm failure. There are apparently multiple assumptions baked into the scheduler and when they don't hold true, this happens.

Before anyone can perform the root cause analysis, the event log message has to be fixed so it states what the scheduler believes the highest priority project is. Without knowing that one cannot consistently replicate the problem.

The queue went to zero and the scheduler got fixated on a project that wasn't handing out tasks. There appears to be no forehead slap routine, sometimes called a watchdog, that kicks in when the queue goes to zero.

At any rate, until the message gets fixed, nothing else can get fixed. Too many unknowns.

---

It's actually multiple bugs because the manual Update button should be forcing a retrieval if there is any space in the queue.

You have a project with 500,000 computers doing your work. Your server manages to adequately give work to 10,000 computers at the same time, maybe 15,000. One day 100,000 of your users work out a prank and they massively at the same time start hitting their Update buttons, every second. You know what that is? That's a DDOS attack. That will bring your server down. And that's exactly the reason why the Update button does not have precedence over the back-off procedure that runs in the background. As that will try to spread the load of the amount of computers and certainly not allow for several presses of the Update button to all go through.

This is a scheduler bug.

And your only proof is a picture. Not even a log (look through stdoutdae.txt and stdoutdae.old in your data directory for older messages). Not even a log with a work_fetch_debug flag on at the time (And I did ask for it before you started changing your preferences! You quoted my post but did absolutely nothing with its contents)

We have a client simulator: https://boinc.berkeley.edu/trac/wiki/ClientSim. Go test your scenarios in that. Or go back to the previous situation if you can (may take a while) and then test correctly.

I've not written one line of code for BOINC

And not looked up any code to substantiate your claims either: https://boinc.berkeley.edu/trac/wiki/SourceCodeGit

I've written an awful lot of code

Now you are advertising your book site and so a moderator can take your whole post down just based on that. Can even kick you for however long they feel is necessary. Might be useful so you take the time to check how you properly report bugs, instead of sprouting your amount of years as code writer as irrefutable proof that you know what you're talking about while you haven't read one line of BOINC code or produced a log with the correct debug flags set.

It's actually multiple bugs because the manual Update button should be forcing a retrieval if there is any space in the queue.

You have a project with 500,000 computers doing your work. Your server manages to adequately give work to 10,000 computers at the same time, maybe 15,000. One day 100,000 of your users work out a prank and they massively at the same time start hitting their Update buttons, every second. You know what that is? That's a DDOS attack. That will bring your server down. And that's exactly the reason why the Update button does not have precedence over the back-off procedure that runs in the background. As that will try to spread the load of the amount of computers and certainly not allow for several presses of the Update button to all go through.

This is a scheduler bug.

And your only proof is a picture. Not even a log (look through stdoutdae.txt and stdoutdae.old in your data directory for older messages). Not even a log with a work_fetch_debug flag on at the time (And I did ask for it before you started changing your preferences! You quoted my post but did absolutely nothing with its contents)

We have a client simulator: https://boinc.berkeley.edu/trac/wiki/ClientSim. Go test your scenarios in that. Or go back to the previous situation if you can (may take a while) and then test correctly.

I've not written one line of code for BOINC

And not looked up any code to substantiate your claims either: https://boinc.berkeley.edu/trac/wiki/SourceCodeGit

I've written an awful lot of code

Now you are advertising your book site and so a moderator can take your whole post down just based on that. Can even kick you for however long they feel is necessary. Might be useful so you take the time to check how you properly report bugs, instead of sprouting your amount of years as code writer as irrefutable proof that you know what you're talking about while you haven't read one line of BOINC code or produced a log with the correct debug flags set.

roland@roland-HP-EliteDesk-800-G2-SFF:~$ cd /
roland@roland-HP-EliteDesk-800-G2-SFF:/$ sudo find -iname stdoutdae
[sudo] password for roland:
find: ‘./run/user/1000/doc’: Permission denied
find: ‘./run/user/1000/gvfs’: Permission denied
roland@roland-HP-EliteDesk-800-G2-SFF:/$ sudo find -iname stdoutdae*
find: ‘./run/user/1000/doc’: Permission denied
find: ‘./run/user/1000/gvfs’: Permission denied
roland@roland-HP-EliteDesk-800-G2-SFF:/$


I did turn it on manually per the on-line instructions and posted a screen shot. I even rebooted afterwards. It didn't take effect until I found the hidden scroll bar.

One cannot begin to do root cause analysis until the event log identifies which project the scheduler considers highest priority. One has to work backwards from that point identifying the "why" behind the fixation.

---

Before going much further, can I ask you to do one more thing?

Yes, there are bugs in BOINC. We do our best to resolve them, when we are presented with sufficient and relevant diagnostic data. Keith Myers and I successfully argued for two successive improvements to the scheduling logic last time round.

So, please update your copy of BOINC from the rather old v7.9.3, to one of the development versions v7.16.6 or v7.17.0 - who knows - we may even have solved your problem even before you posted it ;-)

And with the new version, use the 'priority_debug' event log setting to show us a little more of what's going on with the work fetch logic.

As a developer yourself, you will be aware that legacy versions of software eventually drop out of long term support.

One cannot begin to do root cause analysis until the event log identifies which project the scheduler considers highest priority. One has to work backwards from that point identifying the "why" behind the fixation.

Yes, you can quote my whole post and answer on absolutely nothing again, it doesn't matter. You had the original problem while having LHC, Cosmology, WCG, Rosetta, Citizen Science Grid, Einstein and GPUGRID added to your BOINC.

Your original screen shot only showed part of the log for LHC, Cosmology and GPUGRID. All projects in your original screen shot showed they were allowed to fetch work, none was set to No New Tasks or anything. There was quite a bit of information missing. Is when we started asking for logs with debug flags on. For which you thought you had to 'hack' cc_config.xml, while you can just open that with an ASCII text editor (had you but asked...) or via the BOINC Manager Event Log Options menu item.

And before you even gave a full log, you detached from GPUGRID. By doing so you destroyed your chances of replicating anything, or showing a correct log.
There is no working back to anything because you changed the outcome by detaching from GPUGRID. You say your computer had been without work for anything between 3 and 7 days already, so why the hurry to get work instead of trying to squash what you think is a bug? Because now those chances are slim to none.

But of course, you can now claim conspiracy: "whoever created this bug is intent on hiding it".

I haven't seen a log with work_fetch_debug for LHC, Cosmology, WCG, Rosetta, Citizen Science Grid, Einstein and GPUGRID. (with the priority messages, which if you hurry you can still fetch from stdoutdae (and if you want to know where that can be found under Linux: ask the people here as they'll be able to point you to it))
I haven't seen a log with sched_op_debug for LHC, Cosmology, WCG, Rosetta, Citizen Science Grid, Einstein and GPUGRID.

Before going much further, can I ask you to do one more thing?

Yes, there are bugs in BOINC. We do our best to resolve them, when we are presented with sufficient and relevant diagnostic data. Keith Myers and I successfully argued for two successive improvements to the scheduling logic last time round.

So, please update your copy of BOINC from the rather old v7.9.3, to one of the development versions v7.16.6 or v7.17.0 - who knows - we may even have solved your problem even before you posted it ;-)

And with the new version, use the 'priority_debug' event log setting to show us a little more of what's going on with the work fetch logic.

As a developer yourself, you will be aware that legacy versions of software eventually drop out of long term support.

Yes, the problem with YABU distros is the fact Ubuntu rarely has anything "current" in the repos.

Do you have a link to download a Deb for one of those versions? This link https://boinc.berkeley.edu/download.php has only 7.4.22.

Do you recommend a complete un-install of the distro installed version or is the Deb designed to in-place update?

Now that I've had an evening to digest a bit more, I'm "thinking" what happened is GPUGrid hadn't sent anything to this machine in a long time. Yes, it is supposed to be NVidia only, but once in a blue moon I do see a work unit from it on this machine. The load leveling logic became fixated on GPUGrid to the exclusion of all other projects. Removal and later re-adding "reset the clock" so to speak. It's never been a problem before. I was always able to sign up for all of the projects that all of the other machines are running and just check on the contributions here.

---

LOL,

Because early on another in here suggested GPUGrid was the problem. They may well have been correct. Simply removing the project wasn't enough. Had to remove and reboot to make things better.

Chances of replication are not slim to none, but as Richard has pointed out, they would be misguided with this current version. Once he answers back with the link to pull down a KDE Neon compatible Deb for the development version he wants me to try I will install it and set up all of the same projects. It will take a few months for all of the other projects to rack up work while GPUGrid gets little to none. Then I can disconnect this machine from the Internet, let it run dry, and attempt replicating.

If the development version, with the logging switch he ask for checked, indicates when GPUGrid is considered "The Most Holy" it will make identifying "when" to drain the swamp much cleaner.

As he stated, there have been scheduler changes in the newer versions. They "might" have fixed this. It has the feel of load balancing gone wrong. A project that only once in a blue moon gets work manages to fall far enough behind in the balancing schedule that it becomes the focus of the schedule to the exclusion of all other things. GPUGrid should never issue a packet to a machine without NVidia card if all the work is purely GPU based. It should perpetually fall behind in the load balancing calculations. Those balancing calculations should also take into account the project gets asked for work and sends none.

When I get through with the modifications to the Diamond editor from the CopperSpice project https://github.com/copperspice/diamond I should have some time to look at the scheduler code. The soon to be test scenario will take several months anyway.

---

I did find this site
https://boinc.berkeley.edu/download_all.php

and downloaded the 7.16.6 sh file. Waiting to hear back from you before proceeding.

---

Can't help with that particular variant of Linux, but I have Linux Mint (derivative of Ubuntu) running v7.17.0 - is that close enough?

I get it from https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/locutusofborg-ppa: it's designed to 'upgrade in place' without losing work.

Crossed in the post. The difference between the repo versions and the BOINC version is that the repo ones are designed to run as a service, and BOINC's sh script is designed to run in user mode. The permissions are different, and - more importantly - the file locations are different.

Since you have a repo distribution already, I wouldn't mix'n'match - use Gianfranco's PPA for consistency.

Crossed in the post. The difference between the repo versions and the BOINC version is that the repo ones are designed to run as a service, and BOINC's sh script is designed to run in user mode. The permissions are different, and - more importantly - the file locations are different.

Since you have a repo distribution already, I wouldn't mix'n'match - use Gianfranco's PPA for consistency.

I've got other PPAs added and they work, but there seems to be something KDE Neon simply does not like about that one. No issues adding the PPA, but it seems Neon doesn't recognize any of the package information. A PPA would be the correct solution because "if" the problem is replicated changes to test could be pushed there and tested on a machine that is already in the "correct" state.

---

Sorry, I'll have to pass on that one - I'm still a novice when it comes to the difference between various flavours of Linux.

Sorry, I'll have to pass on that one - I'm still a novice when it comes to the difference between various flavours of Linux.

Not a problem. Someone will eventually chime in with a Neon compatible PPA.

---

I haven't heard of Neon before, so I went looking for info, and found this: Neon: A Wannabe Linux Distro For KDE Lovers.
The 2nd paragraph says:

Neon looks and feels much like a Linux distribution, but its developers assert quite openly on their website that Neon is not a real Linux distro. It just installs and functions like one -- sort of.

So it may be a case of "Roll your own BOINC" for this OS.