PROBLEM: I am really a big fan of all projects of BOINC,so i would like to add all projects to my list of BOINC Manager. However,there's a big problem: the deadline for many tasks are so tight that:
1) tasks are downloaded but may be postponed to the deadline due to the tight processing power,
2) each task may be inadequately run and not finish in time while already-spent computing power and time is wasted.

WISH: So i'd like to see a project cycling mechanism in BOINC(i don't know if there's already one in it) which could cycle all projects in my BOINC Manager list (better user configurable)as for all projects to share possibility to be run in a reasonable period(may be a week or several days,or one or more task cycle.)

Does your host run 24/7 or do you turn it off or hibernate it for part of the day?

Which version of BOINC do you use?

You have asked for BOINC to cycle all the tasks in its cache. You don't explain exactly what that means so nobody can say whether its a good idea or a bad idea. If it means simply aborting all tasks that have not started yet then I don't think the BOINC developers would agree to that because it could cause problems for project servers (increased overhead and network load).

The problem is probably that you are caching too many tasks. If you're crunching many different projects and BOINC can connect to the network anytime it needs to then you are better off turning BOINC's cache off so that your host has only 1 task per core.

To turn caching off create a cc_config.xml file in your BOINC data directory and put <fetch_minimal_work>1</fetch_minimal_work> in it. If you already have a cc_config.xml file simply add the line. Otherwise create the cc_config.xml with a text editor. It should look like this:

<cc_config>
   <log_flags>
   </log_flags>
   <options>
       <fetch_minimal_work>1</fetch_minimal_work>
   </options>
</cc_config>

See here for more details on cc_config.xml.

---

"Windows" -- an American English word, meaning "A real operating system is too hard for me."

BOINC Version:7.028(x86)
I am not asking to cycle tasks,but to cylce projects.

I've noticed that there seems to be a queuing mechanism for Tasks as i've seen some tasks may wait to be exectuted,but there seems to be no such a mechanism for Projects.
If a user added all BOINC projects to his or her project list(may becos he or she wants all projects to equally share the computing power of his/her computor,like i am,or just for fun ),what would happen?
Would all these projects have to compete randomly for computing power? thus some of them may possibly fail to complete their tasks by the deadline.
So far i don't know if BOINC Manager already has a certain mechanism to negotiate among projects(if they are too many) as to ensure all running projects can have adequate computing time to finish their tasks before deadline(supposed that those task results not submitted in time would be abandoned somehow?).

A Project Cycling Mechanism may allow the BOINC Manager to queue all projects up and make specified numbers of projects in queued order to run concurrently so that each project may have a relatively relaxed period to carry out its work(with less computing power competitive stress).Now BOINC seems to just share the computing capacity of the CPU equally among projects,but if there were too many projects running,the shared percentage of each project may be very low.
e.g. on my computor,i have added 7 projects, each of them only shares 14.29% computing capacity of the CPU. I think some of them may fail to submit in time,cos the deadline was assigned by the project designer and is not predicatable.

I had thought about two mechanisms to solve this problem:
1)BOINC Manager calculates the remaining Lifetime for a project's current task and allocate a proper priority for it,when remaining Lifetime gets tight,the manager may hang up execution of less urgent tasks to guarantee execution of those urgent tasks.(of course the number of concurrent projects will also be limited)
2)Introduce a project queuing and cycling mechnism,ensure only a limited number of projects can use the CPU,and also allow all projects be run in order so that their task packages have an equal possibility to be executed. This solution may also favour Pan-Projects lovers like me,as i can then add all projects and don't leave any in cold,but just to set a upper-limit for concurrent projects,and even possibly the project priority, task cycles per project period and conditions for automatical promotion of priority,etc.

BOINC Version:7.028(x86)
I am not asking to cycle tasks,but to cylce projects.

Ah! Yes, my mistake. Thanks for correcting.

The quick and dirty answer to your question is that BOINC has a mechanism for cycling tasks and projects. The mechanism is called "the scheduler" because it schedules tasks and projects. Well, it tries very hard to schedule them in a sane way but there is a lot of entropy and chaos in the BOINC universe and it frequently overwhelms the scheduler. The amazing thing about the scheduler is not that it works well; it is that it works at all.

I've noticed that there seems to be a queuing mechanism for Tasks as i've seen some tasks may wait to be exectuted,but there seems to be no such a mechanism for Projects.

The mechanism is there but sometimes it's difficult to see that it is working. And sometimes it breaks down and does not work the way it should. There are things you can do to help it work better (more on that topic later).

If a user added all BOINC projects to his or her project list(may becos he or she wants all projects to equally share the computing power of his/her computor,like i am,or just for fun ),what would happen?
Would all these projects have to compete randomly for computing power?

I know users who have done exactly that... all projects in their list. They say it works very well (not perfect) if you configure certain options wisely. The projects do not compete randomly for CPU time. The scheduler attempts to make the process sane instead of random. I will attempt an explanation but ity's complicated, if I miss something I invite someone to point out my omission.

The scheduler is governed by the project shares you specify. At each project website you can go into your account, click the link beside the "Preferences for this project" item and set the "Resource share" preference. The default is 100. If you have 5 projects and they all have a resource share of 100 then the total is 500 and each project will receive 20% of your resources. If you add a 6th project and give it a share of 200, for example, then the total is 700. The first 5 projects will have a 100/700 share (14.3%) and the 6th will have a 200/700 share (28.6%). When you start BOINC client it obtains your resource share settings from all the projects you are attached to and calculates the percentages. The percentages are shown in BOINC manager in the Advanced View on the Projects tab. I don't think you can see the percentages in the Simple View which is BOINC's default view. Use the Advanced View if you want to see more details and get the big picture. Simple View hides many details.

The scheduler's 3 principle goals are:

1) schedule tasks in a way that respects the resource shares specified by the user
2) ensure that every task is completed before the deadline
3) keep the host working, i.e. don't allow the task cache to run dry

Those are the goals and they are good. How does the scheduler attempt to achieve the goals?

1) machine ops estimates + benchmarks + duration estimate correction ==> duration estimate

BOINC client runs periodic benchmarks in an attempt to measure the performance of the host in terms of floating point operations and integer operations (FLOPS and IOPS). Projects tag each task with an estimate of how many FLOPS and IOPS the task will require to complete the computation. The scheduler does some simple arithmetic using the benchmarks and the FLOPS and IOPS estimates to derive a task duration estimate which is an estimate of how much CPU time the task will require. The benchmarks and the ops estimates are both inaccurate, sometimes VERY inaccurate, so after a task completes the client compares the duration estimate with the actual task duration. If the actual duration was higher than the estimate then the estimate for tasks from that project is increased. If the actual duration is less than the estimate the estimate is decreased. The scheduler uses the duration estimates to decide how much work to download and hold in the tasks cache on the host. The mechanism works pretty good but obviously if the benchmarks are not accurate and/or the ops estimates provided by projects are inaccurate the mechanism becomes less reliable and the scheduler's ability to make rational scheduling decisions decreases.

2) usage tracking

The client tracks how many hours per day the host is up and running and how many of those hours BOINC is allowed to run and do work. Those figures are also used in scheduler decisions regarding how much work to download and cache. Again, the mechanism works well but it has potential flaws. For example, if you are in the habit of allowing your host to run 24/7/365 then suddenly change your habits and allow BOINC to run only 12 hours per day then you will probably find some of the tasks in your cache will miss their deadline. The reason is that BOINC was operating on the assumption that you would allow 24/7 usage and it cached tasks appropriate for that assumption not a 12/7 assumption. BOINC client will adjust if your usage pattern changes but that adjustment takes time.

3) task cache

The scheduler attempts to keep a number of tasks in a cache where they are ready to crunch immediately upon completion of running tasks. The cache helps mitigate situations where the host loses connection to the 'net for a period of time as well as situations where projects go offline or have no tasks available to fulfill host requests for work

4) the 1 day buffer and high priority mode

The scheduler attempts to schedule work such that tasks will complete at least 1 day before their deadline. Under normal circumstances tasks will run for the time specified by the "switch tasks" setting in your preferences. That setting is 60 minutes by default which would mean tasks willl run for 60 minutes then the scheduler will give tasks from other projects 60 minutes. If it appears that a task will not complete 1 day before it deadline the scheduler will raise that task to "high priority" status and allow that task to crunch continuously, exempt from the task switch rule, until it appears it will meet the "1 day prior to deadline" objective.

So that's how it's supposed to work and I've attempted to explain how and why it sometimes fails. Now the question is... What can you do to help the scheduler?

1) Obviously you should not change your usage patterns abruptly and drastically but sometimes that is unavoidable. It's your computer; you run it to suit your needs not BOINC's needs. However if you decrease the time alloted to BOINC then be prepared to manually abort some tasks while BOINC adjusts to the decreased on time.

2) Set a safe and prudent cache size. As I said, the scheduler works very well but the BOINC universe is full of chaos and entropy. Project admins make mistakes and issue tasks with FLOPS and IOPS estimates that are 10% of what they actually should be. At other projects the algorithm in use is totally nondeterministic and there is no way to predict in advance how long a given task will take. It can happen that you receive 1,000 tasks from a project and they all need about 5 hours to run then suddenly you receive a task from that project that requires 100 hours to complete. here is no way the scheduler can predict that. Therefore the smart thing for you/us to do is specify a very small cache. That way if something goes wrong we have a bigger time buffer between now and the approaching deadline in which the client can complete the tasks in the cache. In your website and local preferences you can set 2 cache related preferences. The "connect about every _ days" setting should be no higher than 0.1 days and the "additional buffer" setting should be 0. If BOINC has unrestricted connection to the 'net, even if it is just dial-up and not high speed, and if you are attached to several projects then the 3 goals I mentioned above will be met 99% of the time. I guarantee you that less prudent people will arrive in this thread in a few hours/days and argue that the 0.1 and 0 settings are ridiculous and that they use 1 and 2 instead of .1 and 0. In some cases those same people will complain their tasks constantly miss deadline and the only thing one can do is wonder why they won't decrease their cache settings and avoid missing deadlines. Their responses will range from the ridiculous to the bizarre. Be prepared for some giggles. Others will claim 1 and 4 works great for them and they almost never have a task miss deadline. To them my response is that they seem to have found a quadrant in the BOINC universe where entropy and chaos are lower than in the quadrant I live in. If 1 and 4 works then feel free to use it. if 10 and 10 works and you don't miss deadlines then go for it. However, if you need to decrease cache down to ridiculously low levels like 0.1 and 0 then that's what you need to do or be prepared to miss deadlines.

The safety net(s)

What happens when all attempts to avoid entropy and chaos fail and you have an expired (past deadline) task or one that is about to expire?

The answer to that question depends on which version of BOINC client you run, which version of BOINC server software the project runs and how the server software is configured. Older clients and older servers have less functionality and cannot deal with expired tasks the way newer client-sever combinations are able to. Also, I may not be completely up to date on this but I can give you the basics.

1) If a task is expired and if it has not started (i.e. it has status "ready to start" in BOINC manager and if the server is sufficiently recent and configured to do so and if the client is sufficiently recent to participate in the transaction, the server will send a cancel/abort message to the client to cancel/abort the task. The catch is that the sever *never* initiates communications with the client. Comms are initiated *only* by the client though the server can tell the client it wants a communication every x minutes and the client will fulfill that request. The point is taht you can have a task that is about to expire in say 10 minutes. It is still waiting to start. It could be canceled/aborted at that time without incurring lost crunch time but unfortunately the client might not contact the server prior to statring the task. If that happens then the task will not be canceled/aborted next time the client contacts the server. The user might receivbe a warning about the task prior to its expiration if the client is a newer version but if the user does not see the message then it does little good.

Many servers allow a period of grace for expired tasks. The period of grace varies between projects. If you return an expired task within the period of grace and it verifies you will receive credit for it. When a task expires some projects issue a resend which is another iteration of the same task sent o another host to make up for the expired task. If you return an expired task after the period of grace but before the resend task is returned you will receive credit for it id it verifies. Remember, not all projects operate that way and the period of grace can vary. If in doubt about any project's policy ask for details in the project's forum.

...allow all projects be run in order so that their task packages have an equal possibility to be executed.

I think you are saying tasks should be crunched in the order in which they are received. Unfortunately that can lead to missed deadlines in some cases. There are often situations where the scheduler can avoid missing a deadline if it crunches a task not in the order received. Humans tend to think the "first come first served" rule should prevail in all circumstances because it is appealing on some quasi moral-ethical-religio-economic level but it leads to tasks missing deadline when they need not miss deadline.

The scheduler is not perfect. It attempts to satisfy the needs of the volunteers as well as the needs of the projects. I think the scheduler could do things better if it did some things differently but there are arguments that suggest such practices would cause additional and unnecessary load on project servers or are problematic for other reasons.

I applaud all the developers who have contributed to the scheduler code. It's not perfect but so far nobody has come up with something that works better. Ofd course part of the rason for that is because the devs won't implement certain suggestions to see whether they work or not but that will change in the future when I own BOINC ;-) <=== (that's a winky and it means I'm being sarcastic and is different from a smiley, it's sarcasm because I will never own BOINC and everybody knows it, just to prevent the paranoid from thinking I'm thinking I am going to take over BOINC development, you can 'ave 'er I don't want 'er she's too fat for me)

---

"Windows" -- an American English word, meaning "A real operating system is too hard for me."

Thank you very much for your generous fingerwork :) <-- this is real sincere smile.

I know users who have done exactly that... all projects in their list. They say it works very well (not perfect) if you configure certain options wisely.

TO be wise requires a mind space with less entropy,lol.
One case is that if volunteers mistakingly think that a long-time-100% processor and processing time occupation of their CPU may do harm to their computor or increase chances of the OS deadlock,they may possibly consider reduce the percentage of relevant parameters,such psychologically occurs frequently ,especially among beginners,and may possibly fail the scheduler. Even i've noticed if i would set 100% percentage of the processor occupation,some slight pauses between operations in BOINC and other application may be felt frequently,i don't know why or do others experienced this as well.

I think a wizard or one-click Smart Configurer may help some volunteers to set those parameters WISELY,and tell them it's the best settings so far,even though may be totally psychological. LOL. I know BOINC is not designed for volunteers to entertain themselves,but a banlanced mutual benefit may help BOINC grow bigger and better, like what those screensavers do.I am always wondering why project designers won't make more screensavers and make'em better and more spetacular as to attract more people to contribute... as well as, to amuse themselves meanwhile.Sci-tech research may also require a psychological co-solution,doesn't it?

Thank you very much for your generous fingerwork :) <-- this is real sincere smile.

[quote]I know users who have done exactly that... all projects in their list. They say it works very well (not perfect) if you configure certain options wisely.

TO be wise requires a mind space with less entropy,lol.

I think you have found a mind space with less entropy. It is more typical of Eastern philosophy and practice than Western philosophy and practice :)

The symptoms are everywhere: decline in Western art and science, the rise of religious dogma and the desire to be stupid (U.S. Republicanism), rampant crime, people electing monkeys (Bush, not just once, twice) as leaders. The West is slipping back into the Dark Ages. The East is emerging and it will be interesting to see whether it recovers its position as the centre of learning and culture or whether it follows the West into the abyss.

One case is that if volunteers mistakingly think that a long-time-100% processor and processing time occupation of their CPU may do harm to their computor or increase chances of the OS deadlock,they may possibly consider reduce the percentage of relevant parameters,such psychologically occurs frequently ,especially among beginners,and may possibly fail the scheduler.

Precisely! The scheduler will recover in time but it helps if you have a smaller cache as opposed to a larger cache.

Even i've noticed if i would set 100% percentage of the processor occupation,some slight pauses between operations in BOINC and other application may be felt frequently,i don't know why or do others experienced this as well.

Some users experience pauses, others do not. There are many possible causes for the pausing. One cause is when you set the option to leave applications in memory when suspended (aka LAIM option). If the host does not have enough RAM to hold the application(s) when it suspends then it can take several seconds to move the applications to the swap file. If several applications are forced to the swap file simultaneously the pause can be quite long. There can be another pause when the application moves from swap back to RAM. Sometimes it happens that the applications are continuously moving back and forthe between RAM and swap and in that case the host can freeze. If you turn LAIM off you reduce the pausing but unfortunately LAIM off causes some tasks to lose progress which requires them to recompute all or part of the data they have already computed.

I think a wizard or one-click Smart Configurer may help some volunteers to set those parameters WISELY,and tell them it's the best settings so far,even though may be totally psychological. LOL.

That is a very good idea. I have given a lot of thought to how such a wizard would work. BOINC is very complicated and its performance depends on many factors and a complex inter-play between those factors. There is much entropy in the system. Such a wizard could work but it would be difficult to program, I think. An optimist would say "It would not be perfect but it would help many users."

I know BOINC is not designed for volunteers to entertain themselves,but a banlanced mutual benefit may help BOINC grow bigger and better, like what those screensavers do.I am always wondering why project designers won't make more screensavers and make'em better and more spetacular as to attract more people to contribute... as well as, to amuse themselves meanwhile.Sci-tech research may also require a psychological co-solution,doesn't it?

Screensavers suck. They are a waste of CPU cycles and I won't waste another second on that topic.

---

"Windows" -- an American English word, meaning "A real operating system is too hard for me."