Message boards : Number crunching : Why does this project waste peoples time and money ?
Message board moderation
| Author | Message |
|---|---|
|
Send message Joined: 12 Dec 25 Posts: 6 Credit: 9,093,041 RAC: 0 |
You send workunits out with a configuration of: minimum quorum 1 initial replication 3 max # of error/total/success tasks 1, 3, 1 Here is an example workunit: https://boinc.berkeley.edu/central/workunit.php?wuid=1707012 It shows - "errors Too many total results" In case it gets purged here are the results for it: 1879998 41780 14 Jan 2026, 3:07:05 UTC 15 Jan 2026, 16:51:50 UTC Completed and validated 135,885.00 1,630,620.00 860.00 Universal Docker app v1.28 (docker) windows_x86_64 1906513 45449 15 Jan 2026, 2:31:10 UTC 15 Jan 2026, 17:54:42 UTC Completed, can't validate 55,412.00 1,773,184.00 0.00 Universal Docker app v1.25 (docker)x86_64-pc-linux-gnu 1916485 43574 15 Jan 2026, 14:01:40 UTC 15 Jan 2026, 16:52:15 UTC Cancelled by server 0.00 0.00 --- Universal Docker app v1.25 (docker)x86_64-pc-linux-gnu The second task was completed but received no credit because the settings only allow for 1 successful task. It was fortunate that the third task had not started before it got cancelled otherwise that would also have been a waste of time and money. I have also seen you sending workunits out with different settings: minimum quorum 1 initial replication 1 max # of error/total/success tasks 1, 3, 1 This can still result in people wasting their time and money if they are sent a resend, start work on it but fail to return it before an earlier host host returns their task late to take the credit and invalidate any later returns. Why are you doing this, how can you justify it ? |
|
Send message Joined: 26 May 23 Posts: 7 Credit: 10,032,854 RAC: 0 |
I'm seeing the same thing too. I'm speculating that if during the period of no work available, the server may resend these "aged" (~1 day old or longer or follow some algorithm to determine when to resend "aged" tasks) tasks to other hosts even though the tasks have not yet expired. Again, I'm speculating based on my anecdotal experience over the past few days of running when the server occasionally ran dry for hours. Perhaps a way to speed up the tasks completion rather than let the tasks expire in the next 7 days during work shortage only. Since yesterday evening, I'm seeing new tasks are readily available for download and let see if some of my "aged" tasks will be resent to other hosts. Certainly it is a waste of resources to recycle the tasks before they expire when tasks are readily available. I can understand why they do this when the server is dry. My one cent. |
|
Send message Joined: 26 May 23 Posts: 7 Credit: 10,032,854 RAC: 0 |
It seems like my speculation is inaccurate. The server keeps sending out tasks to other hosts when the tasks is ~ 1 day old give or take in my work cache even though there are new tasks available to download from the server during the past 24 hours. Seems like I need to keep my work cache to about 1 day of tasks. |
|
Send message Joined: 12 Dec 25 Posts: 6 Credit: 9,093,041 RAC: 0 |
I have never looked at the code related to sending work out so no idea how it processes sending the replicants out. It's clear they aren't all generated and distributed at the same time. Yes, reducing your cache size helps. You could also manually, or write a script to automatically, delete any tasks sent that aren't '_0' . Of course it would be simpler, and better for the project if they set the max # of successful tasks to the intitial replication value plus 1. |
|
Send message Joined: 26 May 23 Posts: 7 Credit: 10,032,854 RAC: 0 |
I posted our "minor" concern to Oliver here: https://boinc.berkeley.edu/central/forum_thread.php?id=26&postid=327#327 Hopefully he will be able to attend and resolve this. |
|
Send message Joined: 22 Mar 25 Posts: 8 Credit: 10,445,418 RAC: 2 |
Hey, thanks for the ping in the other thread, pututu. @PDW: The linked workunit is gone now, unfortunately, I probably retired it to save space before seeing this thread. I looked at the app's configuration, and there's "Run at most this many total instances of each job", which is set to 3. It was my understanding that this means it'll try at most 3 times to get results, if there are errors or timeouts. I don't think it's supposed to send out three jobs in parallel. But based on pututu's posts: if it starts happening after 1 day, then that may be because the max job turnaround time is set to 1 day. He said he set his work cache to 1 day and that fixed it. I'd have thought the server tells the client what the turnaround time is, and the client wouldn't start work on jobs it has had for longer than that. I won't retire batches for now, so we can look into this. I understand that you saw some batches where an "initial replication" setting is set to 1 and also some where it is set to 3. I don't think I've changed anything like that, and if it's the setting above, then that has been the same for months. I definitely agree that the work shouldn't be replicated. And if it is, then those results also would still be valuable for verification. If you (PDW, pututu, or anyone else) see another two examples where the setting differs or where the same job was clearly sent to multiple hosts without waiting for the first result, then please post them. In the meantime I'll link this thread on the Discord and ask people with a better understanding of BUDA/BOINC Central. |
|
Send message Joined: 12 Dec 25 Posts: 6 Credit: 9,093,041 RAC: 0 |
Here is one of your tasks that failed https://boinc.berkeley.edu/central/workunit.php?wuid=1827564 It has these settings: minimum quorum 1 initial replication 2 max # of error/total/success tasks 1, 3, 1 In fact I have been seeing tasks with replication 1, 2 and 3 quite often for weeks. The task I list failed for you, but has been sent to 2 other hosts that are processing it now... 2022378 35549 23 Jan 2026, 0:21:40 UTC 23 Jan 2026, 4:38:49 UTC Error while computing 0.00 0.00 --- Universal Docker app v1.28 (docker) windows_x86_64 2032303 35550 23 Jan 2026, 5:01:19 UTC 30 Jan 2026, 5:01:19 UTC In progress --- --- --- Universal Docker app v1.28 (docker) windows_x86_64 2032822 41674 23 Jan 2026, 14:32:23 UTC 30 Jan 2026, 14:32:23 UTC In progress --- --- --- Universal Docker app v1.28 (docker) windows_x86_64 The fact remains that even without the replication problem if the first host takes too long to complete that, another task is sent out and then the first task does complete (before the second one does) the second task does not get cancelled if it has already started. It will run to completion, assuming no errors, and then get marked as invalid. Having the max # number of success tasks to 1 is the cause of that. Either abort the overdue task, abort the now started task so it won't waste even more time, or preferably increase the max # of success tasks so they both can be valid and given credit. |
|
Send message Joined: 12 Dec 25 Posts: 6 Credit: 9,093,041 RAC: 0 |
Another of your tasks, again replication of 2: https://boinc.berkeley.edu/central/workunit.php?wuid=1834084 You got the second task, less than 24 hours since the first one, you have completed it and got credit. The first task is still being processed and will get no credit if it completes successfully (it hasn't been marked as cancelled, so I assume is running). 2028898 36870 23 Jan 2026, 16:07:46 UTC 30 Jan 2026, 16:07:46 UTC In progress --- --- --- Universal Docker app v1.28 (docker) windows_x86_64 2044624 41084 24 Jan 2026, 14:25:09 UTC 24 Jan 2026, 19:36:27 UTC Completed and validated 18,678.00 224,136.00 236.57 Universal Docker app v1.25 (docker) x86_64-pc-linux-gnu |
|
Send message Joined: 12 Dec 25 Posts: 6 Credit: 9,093,041 RAC: 0 |
In reply to or's message of 24 Jan 2026: .... then those results also would still be valuable for verification.One last thing regarding the above. How are you verifying that a returned result is valid ? If you think having a second result is "valuable for verification" how certain can you be that a single returned result is correct ? |
|
Send message Joined: 12 Dec 25 Posts: 6 Credit: 9,093,041 RAC: 0 |
In reply to or's message of 24 Jan 2026: I'd have thought the server tells the client what the turnaround time is, and the client wouldn't start work on jobs it has had for longer than that.The task has a deadline of 7 days, the client will not abort it if unstarted until that deadline is reached. |
|
Send message Joined: 26 May 23 Posts: 7 Credit: 10,032,854 RAC: 0 |
@or, looking at your host, specifically this one https://boinc.berkeley.edu/central/results.php?hostid=41084&offset=300&show_names=1&state=0&appid= and specifically this job batch_1871, there are some tasks within that batch_1871 that is close to 1 day of aging having a resend while the task has not clearly expired yet. There are two tasks out of nine where you are the wing-person for this batch (ending with task number xxxx_1, i.e. resend, instead of xxxx_0) are about a day old already. Those other seven tasks that you returned much earlier than a day did not have any resend. That affected user will likely finish these two tasks and likely not to get any credit (i.e. waste of electricity) since you returned the result earlier as a wing-person than the original user. batch_1871__job_job-000077_0 1833753 23 Jan 2026, 15:50:47 UTC 23 Jan 2026, 21:28:59 UTC Completed and validated 20,292.00 243,504.00 257.01 Universal Docker app v1.25 (docker) x86_64-pc-linux-gnu batch_1871__job_job-000188_1 1833864 24 Jan 2026, 14:25:09 UTC 24 Jan 2026, 20:54:40 UTC Completed and validated 23,371.00 280,452.00 296.01 Universal Docker app v1.25 (docker) x86_64-pc-linux-gnu batch_1871__job_job-000198_0 1833874 23 Jan 2026, 15:50:47 UTC 23 Jan 2026, 21:28:59 UTC Completed and validated 20,292.00 243,504.00 257.01 Universal Docker app v1.25 (docker) x86_64-pc-linux-gnu batch_1871__job_job-000247_0 1833923 23 Jan 2026, 15:50:47 UTC 23 Jan 2026, 22:51:06 UTC Completed and validated 25,219.00 302,628.00 319.42 Universal Docker app v1.25 (docker) x86_64-pc-linux-gnu batch_1871__job_job-000328_0 1834004 23 Jan 2026, 15:50:47 UTC 23 Jan 2026, 21:28:59 UTC Completed and validated 20,292.00 243,504.00 257.01 Universal Docker app v1.25 (docker) x86_64-pc-linux-gnu batch_1871__job_job-000331_0 1834007 23 Jan 2026, 15:50:47 UTC 23 Jan 2026, 22:51:06 UTC Completed and validated 25,219.00 302,628.00 319.42 Universal Docker app v1.25 (docker) x86_64-pc-linux-gnu batch_1871__job_job-000333_0 1834009 23 Jan 2026, 15:50:47 UTC 23 Jan 2026, 22:51:06 UTC Completed and validated 25,219.00 302,628.00 319.42 Universal Docker app v1.25 (docker) x86_64-pc-linux-gnu batch_1871__job_job-000344_0 1834020 23 Jan 2026, 15:50:47 UTC 23 Jan 2026, 22:51:06 UTC Completed and validated 25,219.00 302,628.00 319.42 Universal Docker app v1.25 (docker) x86_64-pc-linux-gnu batch_1871__job_job-000408_1 1834084 24 Jan 2026, 14:25:09 UTC 24 Jan 2026, 19:36:27 UTC Completed and validated 18,678.00 224,136.00 236.57 Universal Docker app v1.25 (docker) x86_64-pc-linux-gnu |
|
Send message Joined: 26 Nov 21 Posts: 52 Credit: 1,827 RAC: 0 |
This behavior is controled by BOINC Central admins not "or". And it was enabled intentionally to not wait for a couple of jobs in a batch that got to the host that has never returned back to the server. As you may notice, very host contains a field called `Low turnaround time?`. This means that if the host returns results quite fast and doesn't have tasks that were faled dur to missing deadlines, it will be marked as 'low turnaround time', and its tasks will not be sent to the other host until the deadline happens. Also this means that such hosts will likely get a second copy of the task if the first copy was sent to the host that has not replied back for several days. This is a new mechanism that is designed to increase the comletion of batches. Since it's new - there might be some unintantional side effects. Thank you all for reporting this to us. We will investigate the situation and will do required actions to make this new functionality more stable and reliable. BOINC maintainer. For any insight, check my BOINC Development Blog. |
|
Send message Joined: 26 May 23 Posts: 7 Credit: 10,032,854 RAC: 0 |
@Vitalli, thanks for pointing out the new feature "Low turnaround time" (LTT). I didn't pay attention to it when I saw it for the first time. From what I read here https://github.com/BOINC/boinc/wiki/Batch-acceleration, the BOINC Central admins should be able to set the "max delay" period by modifying the "--batch_accel" option for the server feeder if this option is already enabled. If that's the case, it should be straightforward to modify the "max delay" parameter here? Back to the original OP post, if the other host that got a second copy of the task and finishes later than the first copy, credit should also be granted to the second copy of the task as long as the task is still within the deadline and is valid. Is this related to the LTT feature or something else? Perhaps there are two issues here. |
|
Send message Joined: 26 Nov 21 Posts: 52 Credit: 1,827 RAC: 0 |
In reply to pututu's message of 25 Jan 2026: Back to the original OP post, if the other host that got a second copy of the task and finishes later than the first copy, credit should also be granted to the second copy of the task as long as the task is still within the deadline and is valid. Is this related to the LTT feature or something else? Perhaps there are two issues here. Yes, it looks like an issue. We will take a look at it and fix if required. BOINC maintainer. For any insight, check my BOINC Development Blog. |
|
Send message Joined: 22 Mar 25 Posts: 8 Credit: 10,445,418 RAC: 2 |
In reply to PDW's message of 24 Jan 2026: In reply to or's message of 24 Jan 2026: Apologies for the late response, I don't seem to be getting notifications for posts, I'll look into that. My program is looking for a minimal solution in a search space, I already know a solution of length 19, this search is for length 18. So if we find a solution of length 18, then it can be verified easily and it's done, no need to continue. Alas, it looks like there is no such solution, as almost the entire search space has been completed now. It's still a success, because that proves 19 is best, but in this case it's not so easy to prove correctness: the program could be buggy or something could have gone wrong on some hosts. My assumptions are that the program is correct (I've tested it a lot; it found many minimal solutions for other inputs) and that faulty runs are rare. I've seen a few outputs that appear to be corrupted by bit flips due to faulty memory or some other issue on the host. I've discarded the results of those hosts. However, I can't be 100% certain this hasn't happened elsewhere in a less obvious way. Ideally each chunk would be computed twice and if the outputs match, then it's near impossible that the same bit flips occurred twice, but that's a lot of extra computation, I don't think that's necessary. To increase confidence in the results I've added a bunch of verification batches for a decently sized random sample of the whole space, and once I have those outputs I can check how many mismatches there were and calculate an estimate of how many mismatches there are in total. I haven't aggregated all of that yet, but I think it'll show that the vast majority of results are sound. As long as the estimated number of corrupted results is low, the probability that a missed solution happened to be in those chunks is very low. |
David AndersonSend message Joined: 26 Nov 21 Posts: 6 Credit: 46,164 RAC: 0 |
I investigated this and found the possible source of the problem. We added a 'batch acceleration' mechanism that tries to speed up the completion of batches by creating new instances for the last 10% of their jobs. However, this could cause the jobs to reach their max # instances limit and get marked as errors. I just deployed a change that should fix this (by incrementing the max # instances when we create the new instances). We'll keep an eye on things. BTW, none of this is Oliver's fault or responsibility. |
