Benchmarks. Linux v Windows

Message boards : BOINC client : Benchmarks. Linux v Windows
Message board moderation

To post messages, you must log in.

AuthorMessage
The Gas Giant

Send message
Joined: 30 Aug 05
Posts: 65
Message 7459 - Posted: 13 Jan 2007, 23:19:04 UTC

With a 60% difference in benchmarks between linux and windows and a very limited uptake of FLOP counting based credit across the projects, isn't it about time the BOINC devs fixed this benchmarking problem!

Live long and BOINC.

Paul.
ID: 7459 · Report as offensive
Rene
Avatar

Send message
Joined: 20 Nov 06
Posts: 34
Netherlands
Message 7470 - Posted: 14 Jan 2007, 11:30:55 UTC - in response to Message 7459.  

With a 60% difference in benchmarks between linux and windows and a very limited uptake of FLOP counting based credit across the projects, isn't it about time the BOINC devs fixed this benchmarking problem!

Live long and BOINC.


Things are improving and started of with the 5.7.5 linux build.
Difference has decreased to approx 10-15% on the 5.8.2 build.

See these Docking posts...

;-)
ID: 7470 · Report as offensive
MikeMarsUK

Send message
Joined: 16 Apr 06
Posts: 386
United Kingdom
Message 7486 - Posted: 14 Jan 2007, 20:02:01 UTC


FLOP counting isn't the only solution - there are many ways of granting fair credit. Benchmark-based credits are pretty rubbish due to the ease of cheating, it may be worth lobbying the relevant project(s) to move away from using them.

ID: 7486 · Report as offensive
The Gas Giant

Send message
Joined: 30 Aug 05
Posts: 65
Message 7507 - Posted: 15 Jan 2007, 19:31:40 UTC

It's also a fact that projects that do FLOP counting give more credit than benchmark based projects. Something wrong there as well.

I'm running 5.8.2 on my dual boot box (XP / FC5)and still see quite different bm's.
ID: 7507 · Report as offensive
The Gas Giant

Send message
Joined: 30 Aug 05
Posts: 65
Message 7508 - Posted: 15 Jan 2007, 19:36:06 UTC - in response to Message 7507.  

It's also a fact that projects that do FLOP counting give more credit than benchmark based projects. Something wrong there as well.

I'm running 5.8.2 on my dual boot box (XP / FC5)and still see quite different bm's.

FC5
Tue 16 Jan 2007 06:33:47 AM EST||Benchmark results:
Tue 16 Jan 2007 06:33:47 AM EST|| Number of CPUs: 2
Tue 16 Jan 2007 06:33:47 AM EST|| 929 floating point MIPS (Whetstone) per CPU
Tue 16 Jan 2007 06:33:47 AM EST|| 1469 integer MIPS (Dhrystone) per CPU
Tue 16 Jan 2007 06:33:52 AM EST||Resuming computation
ID: 7508 · Report as offensive
The Gas Giant

Send message
Joined: 30 Aug 05
Posts: 65
Message 7510 - Posted: 15 Jan 2007, 19:44:46 UTC - in response to Message 7508.  
Last modified: 15 Jan 2007, 19:54:26 UTC

It's also a fact that projects that do FLOP counting give more credit than benchmark based projects. Something wrong there as well.

I'm running 5.8.2 on my dual boot box (XP / FC5)and still see quite different bm's.

FC5 (5.8.2)
Tue 16 Jan 2007 06:33:47 AM EST||Benchmark results:
Tue 16 Jan 2007 06:33:47 AM EST|| Number of CPUs: 2
Tue 16 Jan 2007 06:33:47 AM EST|| 929 floating point MIPS (Whetstone) per CPU
Tue 16 Jan 2007 06:33:47 AM EST|| 1469 integer MIPS (Dhrystone) per CPU
Tue 16 Jan 2007 06:33:52 AM EST||Resuming computation

XP (5.7.5)
16/01/2007 6:42:29 AM||Benchmark results:
16/01/2007 6:42:29 AM|| Number of CPUs: 2
16/01/2007 6:42:29 AM|| 1413 floating point MIPS (Whetstone) per CPU
16/01/2007 6:42:29 AM|| 1846 integer MIPS (Dhrystone) per CPU

Same box different OS. Major difference in bm's! The linux client is only 73% of the XP client. There goes the idea that 5.8.2 is getting better at benchmarking! Oh well, I'll still alter my <p_fpops>, <p_iops> and <p_calculated> until this problem is fixed.

How hard can it be devs?

ID: 7510 · Report as offensive
ohiomike
Avatar

Send message
Joined: 26 Dec 06
Posts: 26
United States
Message 7525 - Posted: 16 Jan 2007, 4:24:40 UTC

With out repeating my whole rant-
http://boinc.berkeley.edu/dev/forum_thread.php?id=1382
from the Manager forum. There does need to be some work done on the benchmarking code. The thread/priority handling is much different between windows and Linux. I think if we launched the benchmarks as a free-running task (not a thread) and let them run for a set period and report (ie- start and run for 30 seconds then dump results to a file and suicide) that we might get better results. One of the biggest problems with running a benchmark from the client is that the client wants to be low priority, where as the science app wants to be as high of a priority as can be run without affecting user interaction. Windows and Linux once again handle this differently. What priority do we want to give to the benchmark- full speed (no other tasks) or something approximating "real-world" usage. And what is "real-world" usage? It is much difference between this machine I use for email, etc and my Linux development machine, which generally just crunches 20 hours a day or so.

ID: 7525 · Report as offensive
ohiomike
Avatar

Send message
Joined: 26 Dec 06
Posts: 26
United States
Message 7528 - Posted: 16 Jan 2007, 8:06:40 UTC

Just as an added note. I just took the Whetstone code from Boinc V5.9.0 and converted it to stand-alone code. Test machine is a Linux machine with an AMD x2 4200+ 2.2 gHz, running at 2.75gHz. Compiled using gcc (standard Linux) and icc (Intel compiler with math pack).
Stock Boinc v5.9.0 compiled with gcc benchmarks at 3.521 mFP.
Stock Boinc v5.9.0 compiled with icc benchmarks at 4.069 mFP.
Stand-alone Whetstone compiled with gcc benchmarked at 4.499.
Stand-alone Whetstone compiled with icc benchmarked at 7.298.
So which one do we use? How accurate of a predictor it is depends on the app, the optimized SETI app I run is probably close to the 7.298. Einstein maybe also. QMC and others I don't know about.
???
ID: 7528 · Report as offensive
ohiomike
Avatar

Send message
Joined: 26 Dec 06
Posts: 26
United States
Message 7547 - Posted: 17 Jan 2007, 1:38:38 UTC

If anyone is interested, I have posted the code used on the web. I have a stand-alone version of the Boinc benchmark code that can be compiled under Linux or Windows. Suprising enough, the Windows version is real close to the Boinc benchmarks, the Linux is way off.
http://members.cox.net/castleengineering/
Click on the C Programming/Boinc link to grab the tar containing all files.

ID: 7547 · Report as offensive
The Gas Giant

Send message
Joined: 30 Aug 05
Posts: 65
Message 7689 - Posted: 21 Jan 2007, 21:14:48 UTC - in response to Message 7511.  

Benchmarks are very susceptible to other activity.... moving the mouse is enough at critical moments in the run to throw it. Set BOINC.exe to the second highest priority in both OSses and truly hands off run the BM again. Still, 73% is substantially better as the long established and multiple times confirmed 55% was, what it was before on Windows versus Linux.

Yeah, I've been involved with BOINC now for over 3yrs so I know all the nuances of benchmark system. This is one bug that I keep mentioning every so often to see if the devs care. Looks like I have to e-mail the dev and alpha mail list.
ID: 7689 · Report as offensive
Nicolas

Send message
Joined: 19 Jan 07
Posts: 1179
Argentina
Message 7691 - Posted: 21 Jan 2007, 22:27:46 UTC - in response to Message 7528.  

Stock Boinc v5.9.0 compiled with gcc benchmarks at 3.521 mFP.
Stock Boinc v5.9.0 compiled with icc benchmarks at 4.069 mFP.
Stand-alone Whetstone compiled with gcc benchmarked at 4.499.
Stand-alone Whetstone compiled with icc benchmarked at 7.298.

Stock 5.9.0? No such version existing, not even in CVS.
ID: 7691 · Report as offensive

Message boards : BOINC client : Benchmarks. Linux v Windows

Copyright © 2024 University of California.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.