Why does it seem that 5.8.x fetches less work than previous versions of BOINC?

Read on for the explanation from John McLeod VII...

source


OK, the primary goal is to not report work late. The goal of keeping the CPU busy is of much lower priority. therefore, the CPU scheduler is working correctly as designed. Work has to be finished before the connection before it is actually due. This might be as long as connect every X before the report deadline. The scheduler also to ensure that the task is not actually processing at the time of the connection - hence the slop factors of one day and project switch interval. the rr simulator also takes into account the estimated processing time remaining on all tasks.

It is only projects that have a task that is in deadline trouble that are barred from downloading work. If you are attached to several projects and there are some projects that are not in deadline trouble, those projects will fetch work until the queue is filled or they have a task in deadline trouble.

As was stated earlier, the computation deadline for all work is: Report deadline - (connect every + project switch time + 1 day). If you are really disconnected most of the time, this is required in order to ensure that the report is made on time. If you are really disconnected for most of the time, projects with short deadlines may not be appropriate. If you are really connected all of the time, then a short queue is probably sufficient.

3 days of work + 3 days of disconnected + 1 day + 1 hour > 7 days. You ought to be able to use a 3 day queue for a project with an 8 day deadline. But not a 4 day queue (4 days of work + 4 days of disconnected + 1 day + 1 hour > 9 days). Work fetch and the CPU scheduler have to assume that the worst case can happen - i.e. there will be a connect every X days gap in the connection for the entire period specified by X prior to the report deadline so the work has to be completed before that.

The notice that a project has N deadline misses is an indication that the host is entering EDF mode for N CPUs for that project. If you get the same notice for more than one project, then sum all of the numbers to get the number of CPUs that are being dedicated to EDF.

---

Kathryn :o)

Why does it seem that 5.8.x fetches less work than previous versions of BOINC?


It is only projects that have a task that is in deadline trouble that are barred from downloading work.

NOT SO. I am currently running only one project. EVERY workunit must finish before downloading more workunits. One computer is currently running the last remaining worldcommunitygrid workunit, which isn't due until 3/23. It's further set to run only one WCG project, Fight Aids At Home. Today's only the 19th, but it refuses to download more work until this one finishes. I must make sure to babysit and connect at that point in time about 3 hours from now just before the queue becomes empty. My queue is normally 5.5 days, which remained full with 5.4.11. After I downloaded 5.8.8 and watched this behavio(u)r for awhile, I reinstalled 5.4.11, and the queue filled up. All computers are now at 5.8.15.

Why does it seem that 5.8.x fetches less work than previous versions of BOINC?


It is only projects that have a task that is in deadline trouble that are barred from downloading work.

NOT SO. I am currently running only one project. EVERY workunit must finish before downloading more workunits. One computer is currently running the last remaining worldcommunitygrid workunit, which isn't due until 3/23. It's further set to run only one WCG project, Fight Aids At Home. Today's only the 19th, but it refuses to download more work until this one finishes. I must make sure to babysit and connect at that point in time about 3 hours from now just before the queue becomes empty. My queue is normally 5.5 days, which remained full with 5.4.11. After I downloaded 5.8.8 and watched this behavio(u)r for awhile, I reinstalled 5.4.11, and the queue filled up. All computers are now at 5.8.15.

If the deadline is 4 days and the next connection is 5.5 days then that computer will be constantly in EDF. i.e. only one unit/cpu at a time.

Andy

restof is running WCG which has an 8 day deadline. If that is the only project he plans to run, then the Switch application setting can be changed to 1 minute to minimize it's effect on the scheduler. Even with that set to 1 minute, if I understand JM7's post about the scheduler correctly, with an 8 day deadline, the maximum setting for Connect would be 3.49 days.

3.49 + 3.49 + 1 = 7.98

That is pushing it right to the 8 day deadline, so it's probably too high. I would suggest a setting of 3.00 days as an initial test to see if it creates a steady flow of work.

Actually to loosen the noose, WCG extended the deadline to 9 days!

I had not even noticed that. In that case, the Connect setting can be closer to 4 days. I would still start with 3.5 days and see how that goes, making any adjustments necessary.

restof is running WCG which has an 8 day deadline. If that is the only project he plans to run, then the Switch application setting can be changed to 1 minute to minimize it's effect on the scheduler. Even with that set to 1 minute, if I understand JM7's post about the scheduler correctly, with an 8 day deadline, the maximum setting for Connect would be 3.49 days.

3.49 + 3.49 + 1 = 7.98

That is pushing it right to the 8 day deadline, so it's probably too high. I would suggest a setting of 3.00 days as an initial test to see if it creates a steady flow of work.

3.29 is the max for a 8 day deadline with 1 hour switch time, 3.76 for 9 day deadline. From http://boinc-wiki.ath.cx/index.php?title=Work_Buffer.

---

BOINC WIKI

BOINCing since 2002/12/8

OK, the primary goal is to not report work late. The goal of keeping the CPU busy is of much lower priority. therefore, the CPU scheduler is working correctly as designed. Work has to be finished before the connection before it is actually due. This might be as long as connect every X before the report deadline. The scheduler also to ensure that the task is not actually processing at the time of the connection - hence the slop factors of one day and project switch interval. the rr simulator also takes into account the estimated processing time remaining on all tasks.

[etc...]

I think my problem is related. My BOINC core won't keep my CPUs loaded; it will sometimes run two projects and put 100% of each on each CPU, and then switch to 50% of each both on the same CPU. (And it always chooses the CPU that gets hotter; this seems to be from an assumption that the second CPU will stay cooler, but on this SuperMicro H8DCE that is wrong.)

I'm not aware of having set any limits other than disk space - everything else I can find is set to no limits.

Also, I don't understand the reasoning I've quoted - if the goal is to avoid reporting work late, surely any threat of doing so must be countered by doing more work, not less.

How can I set my BOINC client so that it uses the full capacity of the system? It's a dual Opteron 246 running Gentoo Linux with 4 GB and a permanent ADSL link via a Web proxy. I want the system to run flat out all the time it's powered up, which is from breakfast time to bed time.

There are still issues with the scheduler on multiple CPU systems where it doesn't get work for the other cores when a project is having deadline trouble. This is a known problem and hopefully JM7 changes to the algorithm that will be implemented in the next release will address some of this. Boinc continues to evolve and address issues as they become more bothersome and rise to the top of the todo list.

There are still issues with the scheduler on multiple CPU systems where it doesn't get work for the other cores when a project is having deadline trouble. This is a known problem and hopefully JM7 changes to the algorithm that will be implemented in the next release will address some of this. Boinc continues to evolve and address issues as they become more bothersome and rise to the top of the todo list.

I didn't know I was having deadline trouble. How do I get out of it? How do I even diagnose it? I'd still like to know what parameters to set to enable the core to run at full speed.

Thanks for your reply.

I think my problem is related. My BOINC core won't keep my CPUs loaded; it will sometimes run two projects and put 100% of each on each CPU, and then switch to 50% of each both on the same CPU. (And it always chooses the CPU that gets hotter; this seems to be from an assumption that the second CPU will stay cooler, but on this SuperMicro H8DCE that is wrong.)

You said you're using Linux. You can check which processor runs which process real-time. Open terminal window and then run command top. Processor being used byy particular process is shown in column named 'P'. By default it won't show processor being used, so you have to switch on the column:

• press key 'f', it'll show the list of available columns
  
• press key 'j', it'll enable display of last used CPU
  
• press enter to return to main screen
  

Now you should be able to see processor number just in front of command column.

Difference in temperature of the two CPUs might come from non-even air flow through your computer even if the load on processors is equal. On the other hand it might well be that even if you see both processors 100% in use, processes being run use different parts of processors and thusly cause different power consumption and heat dissipation. My gut feeling is that intensive usage of SSE/SSE2 instructions activate much larger part of CPU than intensive usage of simple integer instructions.

How can I set my BOINC client so that it uses the full capacity of the system? It's a dual Opteron 246 running Gentoo Linux with 4 GB and a permanent ADSL link via a Web proxy. I want the system to run flat out all the time it's powered up, which is from breakfast time to bed time.

It's up to BOINC to start as many processes as there are CPUs available in the system. Seems that this task is performed as expected as you have two project apps running in parallel.

It's up to operating system to schedule execution of processes that need CPU time effectively. In multiprocessor system this means also spreading running processes between processors. My experience is that Linux does this job decently. I'm not saying that it couldn't be done better but there are worse OSes in this regard.

What is special with your machine is that you're running dual processor Opteron system. Opterons are special in the way they are handling system memory (RAM). They have memory controller on chip as opposed to slightly older architecture (as used by Intel) which features memory controller on north bridge. As you're running two processors this means your system has two memory controllers. This means your 4GB of RAM is actually split to two parts, most probably in halves. One half is controlled by first processor and second half by second processor. This kind of memory arrangement is called NUMA (Non-Uniform Memory Architecture) and is quite well known from massive parallel super-computers while AMD was first to introduce it to consumer market. BTW, connectivity between processors and between core of processor to its memory controller is implemented using Hyper-transport technology. The two sub-families of Opterons (2xxx and 8xxx) mostly difer by the number of Hper-transport links present: sub-family 2 has one and sub-family 8 has three.

Now, when second processor runs process that has code or data located in memory region controlled by first processor, access to memory has to go through first processor and its memory controller. Which causes slight penalty. Newer Linux kernels have process scheduler that tries to keep processes running on the same physical processor as it might benefit from keeping code and data in CPU's cache. When running on NUMA architecture, this keeps inter-processor memory accesses to minimum. Side effect is what you're seeing. Even newer Linux kernels have another feature: it can move process' memory from one physical RAM region to another in case process runs on different processor than the one controlling the RAM in use. So to say: memory follows process.

Now to your problem: I kind of doubt that both project applications get migrated to one processor just like this. Most probably they get there due to some other process running in your system. After other processes stop using CPUs it might happen that OS scheduler doesn't redistribute processes between CPUs immediately. It'd help if you could state version of Linux kernel you're running.

[edit] spelling

---

Metod ...

Thanks for your detailed reply.

You said you're using Linux. You can check which processor runs which process real-time. Open terminal window and then run command top...

Yes, I know, but I have an easier way - gkrellm runs continuously on this box and draws graphs of, among others, the load on each processor: one colour each for system, user and nice tasks, 90 seconds wide with one-second intervals. It sits at the right edge of my screen so that trends and changes are noticeable. It also shows CPU temperatures as reported by lm_sensors.

Difference in temperature of the two CPUs might come from non-even air flow through your computer even if the load on processors is equal. On the other hand it might well be that even if you see both processors 100% in use, processes being run use different parts of processors and thusly cause different power consumption and heat dissipation. My gut feeling is that intensive usage of SSE/SSE2 instructions activate much larger part of CPU than intensive usage of simple integer instructions.

On my previous motherboard (an MSI K8T Master 2 FAR), CPU0 was always a few degrees hotter than CPU1, even when running apparently similar processes. I put that down to the first CPU's circuits having the memory controller. That board went faulty just before Christmas and was replaced with this SuperMicro H8DCE, which has its memory split into two banks, one per CPU, as you say. I've put 2 GB in each bank, thus splitting them equally between CPUs. Now, even when the system is idling, CPU1 runs about 5 degrees C hotter than CPU0. It's possible that I've crossed the sensor allocations, but I don't think so.

It's up to BOINC to start as many processes as there are CPUs available in the system. Seems that this task is performed as expected as you have two project apps running in parallel.

Yes, it is.

It's up to operating system to schedule execution of processes that need CPU time effectively. In multiprocessor system this means also spreading running processes between processors. My experience is that Linux does this job decently. I'm not saying that it couldn't be done better but there are worse OSes in this regard.

Indeed, and millions suffer them every day. :-(

Newer Linux kernels have process scheduler that tries to keep processes running on the same physical processor as it might benefit from keeping code and data in CPU's cache. When running on NUMA architecture, this keeps inter-processor memory accesses to minimum. Side effect is what you're seeing. Even newer Linux kernels have another feature: it can move process' memory from one physical RAM region to another in case process runs on different processor than the one controlling the RAM in use.

Yes, that last is true of this kernel. But it seems unlikely that kernel scheduler decisions are responsible for chopping CPU time in half. I assumed that must be due to the BOINC scheduler, no?

Now to your problem: I kind of doubt that both project applications get migrated to one processor just like this. Most probably they get there due to some other process running in your system.

On my old motherboard the CPUs were loaded properly; I'm not aware of any major change in system use since then, though I haven't conducted an audit. Of course, since the system is updated daily, there must be thousands of opportunities for programs to change in subtle ways, and I do know that udev has changed the boot process substantially, for instance.

It'd help if you could state version of Linux kernel you're running.

This is kernel 2.6.21-gentoo. I have the following processor options selected in the kernel:
Subarchitecture Type (PC-compatible) --->
Processor family (AMD-Opteron/Athlon64) --->
< > /dev/cpu/microcode - Intel CPU microcode support
<*> /dev/cpu/*/msr - Model-specific register support
<*> /dev/cpu/*/cpuid - CPU information support
[*] MTRR (Memory Type Range Register) support
[*] Symmetric multi-processing support
[ ] SMT (Hyperthreading) scheduler support
[ ] Multi-core scheduler support
Preemption Model (No Forced Preemption (Server))--->
[ ] Preempt The Big Kernel Lock
[*] Non Uniform Memory Access (NUMA) Support
[*] Old style AMD Opteron NUMA detection
[*] ACPI NUMA detection
[ ] NUMA emulation
Memory model (Discontiguous Memory) --->
[ ] Allow for memory hot-add
[*] Page migration
(2) Maximum number of CPUs (2-256)
[ ] Support for hot-pluggable CPUs (EXPERIMENTAL)
[*] Provide RTC interrupt
[ ] IBM Calgary IOMMU support
--- Machine check support
[*] Intel MCE features
[*] AMD MCE features
[ ] kexec system call
[ ] kernel crash dumps (EXPERIMENTAL)
[*] Enable seccomp to safely compute untrusted bytecode
[ ] Enable -fstack-protector buffer overflow detection (EXPERIMENTAL)
Timer frequency (300 HZ) --->
[*] Function reordering

I could revert to an older version, or switch to vanilla sources (i.e. one not tweaked by the Gentoo project), if you think that might help this diagnosis, but if you do think the problem is in the Linux kernel, I'd better try some different optimisations. Perhaps the compiler, too, because, as I'm sure you know, virtually everything on a Gentoo box has been compiled in situ.

Thanks again for your help.

It's up to BOINC to start as many processes as there are CPUs available in the system. Seems that this task is performed as expected as you have two project apps running in parallel.

Yes, it is.

Newer Linux kernels have process scheduler that tries to keep processes running on the same physical processor as it might benefit from keeping code and data in CPU's cache. When running on NUMA architecture, this keeps inter-processor memory accesses to minimum. Side effect is what you're seeing. Even newer Linux kernels have another feature: it can move process' memory from one physical RAM region to another in case process runs on different processor than the one controlling the RAM in use.

Yes, that last is true of this kernel. But I can't believe that kernel scheduler decisions are responsible for chopping CPU time in half. I assumed that must be due to the BOINC scheduler, no?

Your assumptions were wrong. It's up to OS' scheduller to assign process to a certain CPU to execute as it's the only piece of software in your system that knows everything about all processes.

There were customized versions of BOINC CC (mainly targeted at Windows) that enabled so called CPU affinity. This meant that BOINC CC explicitly instructed OSes scheduller to try to keep process on a particular CPU and not to migrate it around if not needed.

Now to your problem: I kind of doubt that both project applications get migrated to one processor just like this. Most probably they get there due to some other process running in your system.

On my old motherboard the CPUs were loaded properly; I'm not aware of any other processes that have changed much since then, though I haven't conducted an audit. Of course, since the system is updated daily, there must be thousands of opportunities for programs to change in subtle ways, and I do know that udev has changed the boot process substantially, for instance.

My guess is that you're also running rather newer version of Linux kernel than you used to on the old MB. And those changes in scheduler are rather recent. Also presence of two memory controllers makes huge difference as you actually didn't have NUMA architecture with the old MB.

I can not see anything in your kernel configuration that is much different from setup as I have on my Opteron systems. And I have yet to see a CPU being idle while another CPU executes two project applications ... I do, however, run kernels 2.6.15-gentoo-r7 on Gentoo, 2.6.20 and Debian Etch and 2.6.21.1 on Debian Etch. Neither of them is exactly the same as yours so it just might make a difference.
But: I'd be much surprised if any distribution packagers would go and change such a feature as scheduler. They mostly tweak parts that interface kernel with users or hardware (device drivers or such) and mostly stay away from core functionality.

Basically I don't have any further idea :-(

---

Metod ...

I can't believe that kernel scheduler decisions are responsible for chopping CPU time in half. I assumed that must be due to the BOINC scheduler, no?

Your assumptions were wrong. It's up to OS' scheduller to assign process to a certain CPU to execute as it's the only piece of software in your system that knows everything about all processes.

No, I didn't mean assigning processes to CPUs; I meant requiring a certain percent loading of a CPU. I don't know how that might work though.

My guess is that you're also running rather newer version of Linux kernel than you used to on the old MB. And those changes in scheduler are rather recent. Also presence of two memory controllers makes huge difference as you actually didn't have NUMA architecture with the old MB.

Ok, it looks as though I'd better start experimenting with kernel versions and configurations.

I have yet to see a CPU being idle while another CPU executes two project applications

Yet that is what I see virtually all the time.

I do, however, run kernels 2.6.15-gentoo-r7 on Gentoo, 2.6.20 and Debian Etch and 2.6.21.1 on Debian Etch. Neither of them is exactly the same as yours so it just might make a difference.

I already have two versions of 2.6.20 to boot from; I'll try those and perhaps some older versions. I can only go back as far as 2.6.16-r13 though.

[quote]I'd be much surprised if any distribution packagers would go and change such a feature as scheduler. They mostly tweak parts that interface kernel with users or hardware (device drivers or such) and mostly stay away from core functionality.

Indeed; that's my understanding too.

Basically I don't have any further idea :-(

Well thanks for your help so far, anyway.

---

Rgds
Peter.

No, I didn't mean assigning processes to CPUs; I meant requiring a certain percent loading of a CPU. I don't know how that might work though.

I've never experimented with this BOINC CC feature. My understanding, though, is that BOINC enforces CPU loading rather coarsely: eg. if user limits BOINC to 33% of CPU usage, it'll run a process for a second full speed and then suspend it for two seconds. It's a matter of probability and statistics but if you'd have setting to limit CPU usage to 50% then you'd probably end up seeing CPU usage of both project applications vary wildly from one refresh of top screen to another, mostly summing up as 100% (of a single CPU). In this case linux kernel scheduler could actually end up running both processes on the same physical CPU, but probably not all the time.
The easiest way of checking if you do have CPU usage limited is to inpect contents of file global_prefs.xml. There's a line which looks something like this:

<cpu_usage_limit>100</cpu_usage_limit>

The line above says use all available CPU, it can be anything between 0 and 100 (%).

---

Metod ...

The easiest way of checking if you do have CPU usage limited is to inpect contents of file global_prefs.xml. There's a line which looks something like this:

<cpu_usage_limit>100</cpu_usage_limit>

The line above says use all available CPU, it can be anything between 0 and 100 (%).

Yes, I have that line too.

While experimenting with various settings to trace bad scheduling decisions, I found the SETI site apparently down, so to save clogging my log files with transmission failures I decided to detach from that project, while remaining connected to two others. This is what happened:

$ boinc -detach_project http://setiathome.berkeley.edu/
2007-05-20 11:59:42 [---] Starting BOINC client version 5.8.15 for i686-pc-linux-gnu
2007-05-20 11:59:42 [---] log flags: task, file_xfer, sched_ops
2007-05-20 11:59:42 [---] Libraries: libcurl/7.16.0 OpenSSL/0.9.8d zlib/1.2.3
2007-05-20 11:59:42 [---] Data directory: /home/prh/common/boinc
2007-05-20 11:59:42 [---] Processor: 2 AuthenticAMD AMD Opteron(tm) Processor 246 [fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext 3dnow]
2007-05-20 11:59:42 [---] Memory: 3.90 GB physical, 5.85 GB virtual
2007-05-20 11:59:42 [---] Disk: 29.34 GB total, 14.40 GB free
2007-05-20 11:59:42 [SETI@home] Resetting project
2007-05-20 11:59:42 [SETI@home] Detaching from project
SIGSEGV: segmentation violation
*** glibc detected *** boinc: malloc(): memory corruption: 0x08243090 ***

Then, each time I pressed <CTRL>-C I got this:

2007-05-20 11:59:55 [---] Received signal 2

but the program didn't exit. I had to "killall -9 boinc" from another terminal.

I'd seen this before but hadn't got round to reporting it.

---

Rgds
Peter.