A bit of background:

A big part of what I do involves E-Mail and mail servers.

SMTP servers accept connections, transfer some number of bits, and then the connection is dropped. SMTP clients try to connect to servers and deliver mail, with a time-tested back-off strategy.

It's all well documented in RFC-2821.

SMTP shares with BOINC clients that decide when they'll connect, and have settings for how often.

Unlike the BOINC client, the sending server can adjust those parameters. The RFC recommends waiting 30 minutes after the first failure, for example.

Mail servers that have been down, even for a few minutes, get bombarded by servers with queued mail, just like SETI@Home after an outage.

The big difference is that the BOINC clients are all developed by the same people who develop the BOINC servers. There are hundreds and thousands of different mail systems all written by different organizations.

(Quick terminology note: a mail server sending mail is a client, GMail delivering mail here is a client.)

... and I started thinking "what would I like as an SMTP server operator to maximize throughput?"

I'd like to be able to tune the clients. Ideally, I'd like to be able to get the clients to connect at exactly the rate that fill whatever resource is the limiting resource.

If the BOINC scheduler (hardware, software and available bandwidth) can handle 1000 simultaneous connections efficiently (and throughput suffers when it is bombarded by 10,000 or more) then being able to tell the BOINC clients to SLOW DOWN! would be incredibly useful to a project.

Slowing down the clients means "satisfying" every client that does connect, and satisfied clients don't come back to hammer the server.

... and while I like the current algorithm (backing off to twice/day) I don't think the algorithm matters that much if a project can tune it.

So my thought is to take every retry timer, and multiply the interval by some constant.

By default, the constant should be 1.0. That's exactly what we have now, and projects that don't want to use this can simply ignore it.

A project that wants lower latency (or simply has fewer crunchers) may want to set the constant to 0.25, meaning a minimum of 2.5 minutes (vs. ten)and a maximum of 3 hours (vs. twelve).

... and a project like SETI@Home, after the weekly outage, may benefit by setting the number to 4.0, which would spread the first retries to 40 minutes, and the maximum to every other day.

Varying the number speeds up or slows down the clients, matching the load to the servers, and failures go down, efficiency goes up.

The biggest problem is being able to adjust the value of the constant without using the same servers that are getting hammered by the BOINC clients.

DNS contains a number of different record types. It's robust, and it's distributed. An unfortunate caching bug in older versions of BOINC is fixed.

The project master file (which connected hosts have) gives the name of the scheduling server. For SETI, it's setiboinc.ssl.berkeley.edu.

To connect, BOINC (or CURL, same thing) queries for an "A" record (or the IPv6 equivalent) to get a list of addresses, and connects.

If there was also a TXT record, we can pass a string. Clients can query for the string and parse it.

For SETI, that record (BIND 4 format) could look like:

setiboinc.ssl.berkeley.edu IN TXT "sf=2.0"

... assuming that BOINC can do a general purpose query, and isn't listed to the typical "gethostbyname()" call.

If we're stuck with gethostbyname() I have an alternative.

gethostbyname() returns an IP address, which is really just a 32 bit number.

Instead of a TXT record, we prepend _sf. to the scheduler name, and my example above becomes:

_sf.setiboinc.ssl.berkeley.edu IN A 0.0.2.0

(The dotted notation is just a convenience, it's a 32 bit value).

(((0*256)+0*256)+2*256)+0*256=512. 512/256 is our scaling factor, 2.

The BOINC client would have to query periodically for changes. This could be twice a day (and on start-up) or every couple of hours, or whenever a retry counter times out.

Using DNS means that a project could publish a new (very large) factor even when everything else was completely broken.

All of the changes would be on the client side, no changes on the server.

I think one factor project wide would be a big improvement. The general idea could be extended to each named server (upload/download, etc.) but that might simply be more complex (it might not be obvious that you'd need to slow down the uploads to free up bandwidth for the scheduler).

David says "interesting idea". ;-)

---

David says "interesting idea". ;-)

It takes advantage of one unique fact.

BOINC controls both the client and the servers.

... and it fits nicely with the whole BOINC concept:

There exists all kinds of "big science" that will never, ever be well funded, that could never afford to do the kind of processing that they need any other way.

It follows that the traditional solutions (buy bigger servers, buy more servers, buy a fatter pipe) just aren't going to happen.

Bandwidth shaping and ACLs and the other tricks they're trying at SETI to deal with more demand than available bandwidth work up to some point. Beyond that the sheer volume of attempted connections (the TCP SYN packets) start swamping out the data.

The answer is to do the shaping at the client: slow down the clients so their load matches the resources at the other end. Match loads are the most efficient, and better efficiency will make everyone happy.

... but there is no one-size-fits-all number, and it's even likely that the right number for SETI on a Friday is not the right number on Tuesday afternoon after an outage, or when some unfortunate glitch happens over the weekend.