BOINC Upload Server on Seperate Host

Message boards : Server programs : BOINC Upload Server on Seperate Host
Message board moderation

To post messages, you must log in.

AuthorMessage
Seth

Send message
Joined: 19 Nov 16
Posts: 50
Australia
Message 75501 - Posted: 26 Jan 2017, 12:33:26 UTC

Hello,
Curious if anyone has set up a BOINC upload server on a separate host to the rest of the BOINC server. How did you go about it etc?

Cheers
Seth
ID: 75501 · Report as offensive
ChristianB
Volunteer moderator
Volunteer developer
Volunteer tester

Send message
Joined: 4 Jul 12
Posts: 305
Germany
Message 75523 - Posted: 28 Jan 2017, 15:25:05 UTC

The answer depends on why you want to move the upload handler to a different server? The usual case is that you are limited by hard disk IO and want to split of IO intensive processes from each other.

If you have different applications in your project you might want to have a separate upload server for each application. Each server will need to have the FUH, validator, assimilator and file deleter for this application. The transition is a bit tricky but not so difficult after all.

If you have a high throughput storage system at your backend and instead the bottleneck seems to be webserver connections you can also migrate to a multi server setup.
ID: 75523 · Report as offensive
Seth

Send message
Joined: 19 Nov 16
Posts: 50
Australia
Message 75539 - Posted: 28 Jan 2017, 22:27:40 UTC - in response to Message 75523.  

Hi Christian,
Thanks for your response.

To be clear my backend is all in AWS so lots of levers to pull in terms of additional capacity.

My DB is running at about 2100 iops and I have provisioned for 3000iops. Database server humms away with plenty of spare CPU and Memory, its a m4.2XL

My WU gzipped files reside in S3 and are fetched by the clients directly from there using a URL.

Results files are pushed to S3 from my BOINC server every 5 minutes.

My remaining server is running on a x1.32xlarge, it seems to be coping well now with the load but going to hit a limit shortly I think. My WU take about 65 seconds to process on my AWS C size cleints. I was running at about 12000 WU per minute. I have 50 each of the transitioners and validators. I'm using a simple bitwise validator and a single instance is working well.

I'm not using FCGI, I have read some mixed thing on it which lead me to focus on some other optimization first.

I have read about moving transitioners/validators to other clients but unsure that will give me a big increase in terms of capacity.

The nature of my project means WU are small in size. It suits me anyway as I'm using an AWS spot fleet for my clients. The result of small files is lots of web server hits thus my interest in splitting the upload server component onto another host. I have an idea in my mind how to do this but if you have any gotchas or any other suggestions for performance tuning I would very much like to hear them.

Cheers
Seth
ID: 75539 · Report as offensive
ChristianB
Volunteer moderator
Volunteer developer
Volunteer tester

Send message
Joined: 4 Jul 12
Posts: 305
Germany
Message 75559 - Posted: 29 Jan 2017, 19:04:34 UTC - in response to Message 75539.  

My remaining server is running on a x1.32xlarge, it seems to be coping well now with the load but going to hit a limit shortly I think. My WU take about 65 seconds to process on my AWS C size cleints. I was running at about 12000 WU per minute. I have 50 each of the transitioners and validators. I'm using a simple bitwise validator and a single instance is working well.

Do you mean that you have a single assimilator or a single validator?

I'm not using FCGI, I have read some mixed thing on it which lead me to focus on some other optimization first.

I have read about moving transitioners/validators to other clients but unsure that will give me a big increase in terms of capacity.

The nature of my project means WU are small in size. It suits me anyway as I'm using an AWS spot fleet for my clients. The result of small files is lots of web server hits thus my interest in splitting the upload server component onto another host. I have an idea in my mind how to do this but if you have any gotchas or any other suggestions for performance tuning I would very much like to hear them.

With this high throughput and short runtimes you should definitly think about FCGI. Just remember that for every connection of a client a new process has to spin up. With FCGI you would keep a number of processes running that would serve the requests without the need of starting new ones. This can be done with the stock CGI and file_upload_handler using multiwatch and spawn_fcgi.

Just keep in mind that the validator and assimilator need to access the data uploaded by the client so they should be on the same machine if the data is held locally. If you transition you need either make sure that all old WUs are processed before you change the upload URL in your template file or restrict the old validator, assimilator chain to only process WUs with an ID lower than what was the highest ID when you changed the upload URL (don't forget to stop the work generator).
ID: 75559 · Report as offensive
Seth

Send message
Joined: 19 Nov 16
Posts: 50
Australia
Message 75572 - Posted: 31 Jan 2017, 0:29:13 UTC - in response to Message 75559.  

Hi Christian,
Thanks for the tips.

I meant I have a single assimilator but I have seen an issue over the last day with assimilation that suggests to me the single instance was not coping with the load. I have work units marked as complete (success) in the boinc DB but the files are still in my upload directory. Also some results seem to have gone missing completely.

I have 50 validators running.

I'll move to FCGI tonight. I understand I need to upgrade my apache first correct?

To be clear are you saying that I don't need to rebuild any of my boinc components to start using fcgi? I've looked at some instructions here that say I need to rebuild some components: https://boinc.berkeley.edu/trac/wiki/MultiHost I'm a bit confused when you say it can be done with "stock CGI and file_upload_handler using multiwatch and spawn_fcgi"

Thanks again for your advice.

Cheers
Seth
ID: 75572 · Report as offensive
Seth

Send message
Joined: 19 Nov 16
Posts: 50
Australia
Message 75610 - Posted: 2 Feb 2017, 1:18:06 UTC - in response to Message 75572.  
Last modified: 2 Feb 2017, 1:18:28 UTC

Hi Christian,
I've followed the steps for FCGI but I keep seeing in my apache2 log the following:

FCGI: FCGI_Accept failed - exiting

Any ideas?

Seems to be triggered by download requests only (cgi?).

Cheers
Seth
ID: 75610 · Report as offensive
Seth

Send message
Joined: 19 Nov 16
Posts: 50
Australia
Message 75611 - Posted: 2 Feb 2017, 2:23:20 UTC - in response to Message 75610.  

BTW I'm using the instructions here, I think they are a little out of date: https://boinc.berkeley.edu/trac/wiki/MultiHost
ID: 75611 · Report as offensive
Seth

Send message
Joined: 19 Nov 16
Posts: 50
Australia
Message 75613 - Posted: 2 Feb 2017, 3:51:12 UTC - in response to Message 75611.  

Yep the instructions are out of date. It's running now.

I have asked before but don't think I got a reply. When I find mistakes or outdated instructions how can I contribute back to the project with updates?

Cheers
Seth
ID: 75613 · Report as offensive
Profile Ageless
Volunteer moderator
Project administrator
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 12223
Netherlands
Message 75619 - Posted: 2 Feb 2017, 12:50:07 UTC - in response to Message 75613.  

Easiest way is to ask David for an account on there to make changes yourself.
But else sign up to the BOINC projects email list and write your changes via there. Someone (Christian or me) will then probably pick it up and put it into place.
Jord
Please do not private message me for tech support. Use the forums for that. Tech PMs will be ignored.
ID: 75619 · Report as offensive
ChristianB
Volunteer moderator
Volunteer developer
Volunteer tester

Send message
Joined: 4 Jul 12
Posts: 305
Germany
Message 75624 - Posted: 2 Feb 2017, 13:51:25 UTC - in response to Message 75572.  

To be clear are you saying that I don't need to rebuild any of my boinc components to start using fcgi? I've looked at some instructions here that say I need to rebuild some components: https://boinc.berkeley.edu/trac/wiki/MultiHost I'm a bit confused when you say it can be done with "stock CGI and file_upload_handler using multiwatch and spawn_fcgi"

There are separate binaries for FCGI built if you have the fcgi development headers installed. It seems you already found them.
ID: 75624 · Report as offensive

Message boards : Server programs : BOINC Upload Server on Seperate Host

Copyright © 2017 University of California. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.