Well, there are no publishable pictures of the complete beast. Indeed the complete beast is very boring to look at, just a large box with power, data and cooling connections.
Initially a small system was tested using RTX2080 to give an idea of what feeders were going to be needed. Next tests were with earlier Quadro which left the RTX2080 behind, after six months (and some mods to the cooling) the RTX8000 were installed, and they are a step up again. The trouble with bench marks and specs is they don't always reflect what happens in real life under very high stress.

Being a totally air-cooled system the GPUs were obtained without their fans, etc. blast air at ~4C keeps everything in check.

But we digress.

No publishable pictures?! Then I guess it's something dodgy it's doing. The US government hacking into everyone's emails?

Why do they have the 5 second limit? Is it a real limitation of the GPU, or a silly decision made by the CUDA inventors? and why can't everything be stored in GPU memory, with just calls by the CPU for the GPU to access it? I thought that was the whole point of a GPU having memory on board, so it didn't have to access main RAM and a) get in the way of the CPU trying to use it for something else, and b) wait on bus transfers.

Googling around (an exercise I suggest you try sometime), it seems that the execution limit is applied by the operating system, and only applies to GPUs with video output ports. Perhaps it's the working limit that the operating system 'look and feel' developers think that a human being is prepared to watch the eggtimer (or modern equivalent) while her search results load. I don't know.

The 5-second limit doesn't appear to apply to Tesla-class professional GPUs without video outputs, or presumably to coin mining frivolities - though I think you have to use the professional drivers to get that benefit. But since BOINC is designed to be used on volunteers' home computers with - presumably - screens attached, it's a moot point.

let me know what something like that is built by one person and using their hobby money lol. a multi-million <insert currency> supercomputer built by a team of engineers from one or more companies for a very specific use case is a bit outside the scope of this topic i think. not to mention that a system like that couldn't even by recognized by BOINC as a single host anyway lol (hard cap of 64 GPUs per host).

You could always run multiple hosts - people do that on home machines to get over the 900 task per host limit from Milkyway.

"in one rack" or "in one room" might be more appropriate than "in one computer" and the resources would have to be split up virtually to ever run these BOINC projects.

Yes, when does a computer become a farm? Is this beast running one single OS? I mean you could call every Boinc volunteer in total one "computer" as they're working together.

Googling around (an exercise I suggest you try sometime),

I wasn't being lazy, honest! It's just that you might have already known, or know what to look for, how to interpret it, and how to dumb it down for a non-programmer :-)

it seems that the execution limit is applied by the operating system, and only applies to GPUs with video output ports. Perhaps it's the working limit that the operating system 'look and feel' developers think that a human being is prepared to watch the eggtimer (or modern equivalent) while her search results load. I don't know.

The 5-second limit doesn't appear to apply to Tesla-class professional GPUs without video outputs, or presumably to coin mining frivolities - though I think you have to use the professional drivers to get that benefit. But since BOINC is designed to be used on volunteers' home computers with - presumably - screens attached, it's a moot point.

But the rig in question here has no monitor. I can plug one in while setting things up for troubleshooting, but usually it's easier to access it remotely. Boinctasks to control Boinc and Remote Utilities for accessing Windows (the built in Windows remote program shuts off the GPU!) So would the limit still apply?

But if you run two tasks on the GPU, the data requirement of each task is halved, as they are running slower.

the requirement is not halved. the available bandwidth is halved. and the WU runs slower as a consequence. the whole idea is to NOT slow down the WU, to do that you need to provide enough resources so as to not bottleneck it.

Then they really should launch it mainstream.

launch what mainstream? the app is available to anyone with Linux and a capable GPU and running SETI. the source code is also available, but its unlikely to be useful to anyone trying to port it to another project doing totally different science and calculations. This is the square peg in a round hole kind of situation.

AMD GPUs do not have CUDA so they cannot be used.
The app was written for Linux and has not been ported for Windows (lack of Windows CUDA developers)
The app makes use of certain techniques that require CC 5.0 or greater, so you need a Maxwell or better card.

The app only works for seti, and the techniques used are probably only really applicable to other forms of signals analysis.

I don't suppose it's possible to force PCI Express 1.0 if I have a long cable that can't manage 3.0?

you can enforce PCIe 1.0 in the BIOS if the option is available. hard to say on such an old motherboard. just look for settings regarding the PCIe slots. but you shouldnt have any issue on your current hardware since none of the slots are PCIe 3.0 anyway.

I thought it was a whole new app, not just a command or configuration you typed in.

in terms of the app that I am running personally, it is the CUDA specially optimized application combined with a single command line parameter for it.

however seti also has OpenCL stock applications, and you can optimize those further with command line options for them. you may have options available to use for your apps in a similar fashion. you just need to ask the guys over on those projects.

No publishable pictures?! Then I guess it's something dodgy it's doing. The US government hacking into everyone's emails?

LOL, I think if you inspect his profile a little closer you will find he's on the same side of the pond that you are on ;)

my guess is probably some form of AI/machine learning. or simulations like weather or space-related modelling.

No publishable pictures?! Then I guess it's something dodgy it's doing. The US government hacking into everyone's emails?

LOL, I think if you inspect his profile a little closer you will find he's on the same side of the pond that you are on ;)

Oh, I could have just looked at his flag. I know Richard is over here, I didn't know there was another one :-)

my guess is probably some form of AI/machine learning. or simulations like weather or space-related modelling.

Or maybe he's really in the USA and is pretending to live here to hide what he's up to ;-)

How can any of the things you suggested be top secret? Unless it's part of the government plan to prove global warming exists.

I don't see anywhere that said top secret other than your post. but companies certainly have the right to protect their IP from competitors in any way they wish, which is more likely the case. just because you build something, doesn't mean you have to tell everyone how you did it.

companies working on AI/ML are all secretive of their work and technologies. especially since that sector is in a huge boom right now. everyone want's to be the first to have the next great AI technology they can sell or put into a product they can sell. if they told eveyone exactly what they were doing, someone could just steal their work, claim it as their own, and beat them to market.

the requirement is not halved. the available bandwidth is halved. and the WU runs slower as a consequence. the whole idea is to NOT slow down the WU, to do that you need to provide enough resources so as to not bottleneck it.

What I mean is, if you (and you don't with SoG) have the option to run one WU on one GPU, or to run two WUs on one GPU, then when you double them up, each WU has half the requirements external to the graphics card - both CPU time and bus time, since effectively that WU is running on a slower GPU. In my case, my CPU is rather slow, so running one WU at a time, the GPU is constantly stopping to wait for the CPU to do its bit. But running two at a time, although the total CPU requirement is the same overall, the requests for the CPU are at different times. One WU might be waiting for the CPU, but the GPU can then put it's full power into the other WU, meaning the GPU is always fully utilised. The same should be true of the bus. Your GPU would still need the same amount of data transfer overall for all the WUs it's running (it has to, the total work done is limited by the GPU speed), but since it has two or more things to do at once, while one WU is waiting for the bus, the other can be crunching. Anyway, I was only suggesting that when I thought that half the time your WUs were using a lot less bus. It appears to be true for MW and Einstein though, I can see the bus usage in short peaks but zero most of the time (if I'm reading the task manager correctly), so running more tasks per GPU would mean if any task was held up by the overloaded bus, it could run the others.

launch what mainstream? the app is available to anyone with Linux and a capable GPU and running SETI. the source code is also available, but its unlikely to be useful to anyone trying to port it to another project doing totally different science and calculations. This is the square peg in a round hole kind of situation.

I meant run it without you having to choose to do so manually. So anyone who happens to have Linux and the right kind of card gets given that app, without having to know it exists or be capable and willing to play around with it.

The app only works for seti, and the techniques used are probably only really applicable to other forms of signals analysis.

Any other projects doing similar analysis? Or is SETI unique in doing those sort of calculations?

you can enforce PCIe 1.0 in the BIOS if the option is available. hard to say on such an old motherboard. just look for settings regarding the PCIe slots. but you shouldnt have any issue on your current hardware since none of the slots are PCIe 3.0 anyway.

I guess not, I should be able to run the cable a lot longer with 2.0 than 3.0 anyway. And I can always connect the further away GPUs to the x1 slots.

in terms of the app that I am running personally, it is the CUDA specially optimized application combined with a single command line parameter for it.

[quote]however seti also has OpenCL stock applications, and you can optimize those further with command line options for them. you may have options available to use for your apps in a similar fashion. you just need to ask the guys over on those projects.

I'll ask. Although why don't they come preoptimized?

But the rig in question here has no monitor. I can plug one in while setting things up for troubleshooting, but usually it's easier to access it remotely. Boinctasks to control Boinc and Remote Utilities for accessing Windows (the built in Windows remote program shuts off the GPU!) So would the limit still apply?

The general principle of BOINC is that all science apps are generally written to be downloaded and run on general-purpose, consumer grade electronics. Some projects have a particular 'need for speed' and may recommend minimum limits, but SETI - since we've mentioned it already - is designed to be accessible to all-comers and their kit (hence the extremely long deadlines).

If, against the whole spirit of BOINC, you choose to build your own supercomputer for bragging rights - you have to develop your own super-software to run on it. As the GPU User Group have done at SETI. Both BOINC and SETI - and some other projects as well - have published their canonical source code under the GPL, and (provided you comply with the licence terms under GPL) you are free to grab it and use it. Remembering that your scientific results will only be accepted and recognised if they pass the quality-control verification provided by validation.

I don't see anywhere that said top secret other than your post. but companies certainly have the right to protect their IP from competitors in any way they wish, which is more likely the case. just because you build something, doesn't mean you have to tell everyone how you did it.

companies working on AI/ML are all secretive of their work and technologies. especially since that sector is in a huge boom right now. everyone want's to be the first to have the next great AI technology they can sell or put into a product they can sell. if they told eveyone exactly what they were doing, someone could just steal their work, claim it as their own, and beat them to market.

True, but I didn't think a photo of the overall machine would reveal much. Unless they've completely redesigned and built their own motherboards etc, then someone might steal a few ideas from the picture, but not a lot.

Mind you, what could be wrong with just saying what it does?

The general principle of BOINC is that all science apps are generally written to be downloaded and run on general-purpose, consumer grade electronics. Some projects have a particular 'need for speed' and may recommend minimum limits, but SETI - since we've mentioned it already - is designed to be accessible to all-comers and their kit (hence the extremely long deadlines).

If, against the whole spirit of BOINC, you choose to build your own supercomputer for bragging rights - you have to develop your own super-software to run on it. As the GPU User Group have done at SETI. Both BOINC and SETI - and some other projects as well - have published their canonical source code under the GPL, and (provided you comply with the licence terms under GPL) you are free to grab it and use it. Remembering that your scientific results will only be accepted and recognised if they pass the quality-control verification provided by validation.

It's not "bragging rights" I do it for, way too much money goes into the electricity for that, I can't believe anyone would build an expensive power hungry machine just to say "Ha! I got more credits than you this week!". I do it to benefit science. And to have fun with computers along the way.

I meant run it without you having to choose to do so manually. So anyone who happens to have Linux and the right kind of card gets given that app, without having to know it exists or be capable and willing to play around with it.

that's a question for the SETI project scientists *shrug*. they have been aware of the app for a long time, but as of yet, they have decided against providing it on SETI main for distribution for unknown reasons.

Any other projects doing similar analysis? Or is SETI unique in doing those sort of calculations?

don't know

I'll ask. Although quite why don't they come preoptimized?

that's a question for the developers of the apps on those projects. but if they are anything like SETI, its probably because different hardware requires different optimization. so they push out an app that can run OK on almost all hardware, and for the people that want to mess with it, they can apply the specific optimizations needed to take full advantage of their hardware. an RTX 2080ti could use much more aggressive tuning than say a GT 1030, just as an example. it doesnt make sense to try to provision the apps for every possible hardware combination, or even have a ton of different apps for different hardware combinations. much simpler to have a single app that can be tweaked in many different ways.

It's not "bragging rights" I do it for, way too much money goes into the electricity for that, I can't believe anyone would build an expensive power hungry machine just to say "Ha! I got more credits than you this week!". I do it to benefit science. And to have fun with computers along the way.

some people do it for all three of those reasons :) they are not mutually exclusive

that's a question for the SETI project scientists *shrug*. they have been aware of the app for a long time, but as of yet, they have decided against providing it on SETI main for distribution for unknown reasons.

I assume it produces good results or yours wouldn't get validated. Do you have to do a lot of testing and fiddling about to make sure they validate? If so, that would be a reason not to hand it out to people who don't fiddle.

that's a question for the developers of the apps on those projects. but if they are anything like SETI, its probably because different hardware requires different optimization. so they push out an app that can run OK on almost all hardware, and for the people that want to mess with it, they can apply the specific optimizations needed to take full advantage of their hardware. an RTX 2080ti could use much more aggressive tuning than say a GT 1030, just as an example. it doesnt make sense to try to provision the apps for every possible hardware combination, or even have a ton of different apps for different hardware combinations. much simpler to have a single app that can be tweaked in many different ways.

Yes, but wouldn't it be fairly simple to have a list of optimisations, and use the correct one automatically after detecting what card you have etc?

some people do it for all three of those reasons :) they are not mutually exclusive

I can understand bragging being one of them, but not the only one. It's not like owning a Ferrari where you can really show it off. Just showing someone a number on a screen isn't quite so amazing :-)

I assume it produces good results or yours wouldn't get validated. Do you have to do a lot of testing and fiddling about to make sure they validate? If so, that would be a reason not to hand it out to people who don't fiddle.

yes it produces valid results. most of the testing is done by the author of the application before it is released to everyone else. the last few releases have only gotten better and produced less and less invalid results. the app as it stands is just as accurate as any of the other apps in my opinion. its just a whole lot faster if you have the correct GPU and Operating System.

Yes, but wouldn't it be fairly simple to have a list of optimisations, and use the correct one automatically after detecting what card you have etc?

if it were simple, the app developers would have done that. you're welcome to try to program in the thousands of different hardware combinations and the exact right optimizations for each one if you think it's easy. in most cases, the user has to play with the available options to discover the set of optimizations that works best for their system.

also most of the apps are probably being written by project scientists who 1. are probably not software developers as their main subject of expertise, and 2. who care more about the result being RIGHT than being FAST, so once they get an app produced that gives them correct results, they probably don't want to waste too much time further developing the app to be more efficient. this is how the most efficient apps at SETI came from volunteer work.

Well, there are no publishable pictures of the complete beast. Indeed the complete beast is very boring to look at, just a large box with power, data and cooling connections.
Initially a small system was tested using RTX2080 to give an idea of what feeders were going to be needed. Next tests were with earlier Quadro which left the RTX2080 behind, after six months (and some mods to the cooling) the RTX8000 were installed, and they are a step up again. The trouble with bench marks and specs is they don't always reflect what happens in real life under very high stress.

Being a totally air-cooled system the GPUs were obtained without their fans, etc. blast air at ~4C keeps everything in check.

But we digress.

I dare you to run Boinc on it, just for a day.

Well, there are no publishable pictures of the complete beast. Indeed the complete beast is very boring to look at, just a large box with power, data and cooling connections.
Initially a small system was tested using RTX2080 to give an idea of what feeders were going to be needed. Next tests were with earlier Quadro which left the RTX2080 behind, after six months (and some mods to the cooling) the RTX8000 were installed, and they are a step up again. The trouble with bench marks and specs is they don't always reflect what happens in real life under very high stress.

Being a totally air-cooled system the GPUs were obtained without their fans, etc. blast air at ~4C keeps everything in check.

But we digress.

I dare you to run Boinc on it, just for a day.

Some Russian scientist tried something similar on their own "super beast" It did not go well

Some Russian scientist tried something similar on their own "super beast" It did not go well

The only way you can actually make money with Bitcoins - use someone else's stuff.

Hang on, 1 petaflop? That's only 250 of my GPUs. I could build that for 18 grand. Unless it's CPU petaflops, which I guess can do more than RISC petaflops.

I used to run Boinc where I worked. It caused a little bit of debate in rooms with many computers, much heat was generated. Nobody ever considered the electricity cost :-)