Posts by rowet

1) Message boards : Questions and problems : Windows-only air-gapped network grid computing solution? (Message 79021)
Posted 14 Jun 2017 by rowet
Post:
I'm trying to evaluate BOINC as a potential solution for my problem just by perusing the docs with no prior exposure, and I'm not doing so great figuring it out. Hopefully it isn't rude to lay out the situation and ask for advice.

I have several Windows only networks (no chance of a Linux server in the mix, no way to run a virtual machine, no cygwin). These networks are air-gapped.

I have a monte-carlo simulation executable (C++) that takes a seed and an input file ranging in size from kilobytes to hundreds of megabytes. The executable also has about 20 megabytes of DLL dependencies and some other static files. The simulation run times range from seconds to days. The simulation spits out a results file.

So I need a grid computing solution that runs entirely on Windows. It needs to take the specification of the simulation executable, the simulation input file, and a seed list, perform the replications, and collect the output files somewhere. There are about five perfectly adequate and mature solutions for problem this in Linux-land, but this whole thing needs to be Windows only.

An additional problem is I need low latency. For the cases when single runs of the simulation only take a few seconds, dozens of seconds of wait time when distributing the work and starting it up is not acceptable. After hitting run, all the cores should be hot inside ten seconds. Executables and DLLs and input files should be cached on the job nodes to avoid re-copying.

HTCondor in theory works on Windows but it failed me on the latency constraint and I had other problems with it. There was no reliable way to make it snappy for short jobs on Windows. In my experiments I also hit tons of odd behavior and bugs to do with Windows credentials. It would even randomly lock out user accounts.

Is there any chance that BOINC makes more sense for my case than writing something from scratch or paying for Xoreax Grid Engine, or maybe some commercial Hadoop distribution? My assesment is that BOINC is not suitable for my case, but I figured it wouldn't hurt to ask.



Copyright © 2024 University of California.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.