I forwarded your post to the BOINC Alpha email list.
Can you in the mean time leave behind which BOINC version you're using for this?

---

A fix was recently put into the client (7.2.24+ I believe), where "memory used by VM tasks" is now better accounted for by BOINC on Windows. BOINC now correctly "accounts" for a VM task based on how much RAM the VM requests, instead of reading memory usage from the VM processes (which doesn't work for VM tasks/processes). Oracle VM documentation says the host must supply the RAM as free RAM, and never expect any of it back, so now BOINC uses that "VM RAM" value as how much RAM the VM task is always consuming.

So, for instance, I only allow BOINC to use 60% of my 12 GB of RAM, so 7.2 GB. Each RNA World VM requests 4 GB RAM. This means that, as of 7.2.24+, BOINC will never allow 2 RNA World VM tasks to run concurrently on my system.

And thus, VM tasks follow the same "memory rules" that other BOINC tasks follow; BOINC should only allocate up to the user-specified amount of RAM.

Note: I think it's perfectly plausible to have 8 VMs running at once, provided the sum of their RAM values was less than what you specified in your setting. And I don't think Windows would BSOD. But I've yet to encounter the scenario where I could test that theory.

Regards,
Jacob