Continuing the news outage thread in a brand new coating.

Last news was posted by mo.v

CPDN main project

One of the upload servers has been down but it's now up and running again.

An extra server has been ordered for CPDN which is good news.

Please for clarity don't add your signature when you post to this thread.

I'm sorry also to be starting the new thread.

CPDN, Beta, BBC and SAP

The forums and websites of all CPDN projects lost communication with their regional network about 30 minutes ago. My Beta models have just been unable to upload trickles so I expect the situation is the same on all projects. All Oxford University servers on the ja.net network are probably affected.

Edit: The network is now fully up again - Oxford Uni including all our projects is no longer incommunicado.

The Einstein database server crashed; David H. is busy bringing the project back online now.

Einstein's servers are back up.

CPDN main project and CPDN Beta

Milo says

We've just received a large server that will give us another 10TB of space, and will try to set it up this week. This means that there may be outages on uploader.oerc and cpdn-upload1.comlab that may prevent uploading.

This is good news for CPDN and the programmers.

It would be a good idea for members to keep BOINC network activity suspended most of the time this week and only enable it when you are present and can see whether your trickles and files do upload. We realise that many multi-project crunchers cannot do this.

You can check CPDN server status here and Beta server status here.

Cosmology's quite happy to start a download, then it gives interminable HTTP errors.

---

CPDN main project

The CPDN database including the CPDN-BOINC forum has been down this morning for maintenance. Trickles and file uploads should not have been affected but the awarding of credits and their export to the stats sites may be delayed.

Edit: The BBC database and forum also appear to be down.

It looks like malariacontrol.net is off line. Anyone in the know who has information?

CPDN main project and BBC

Update to my post two above this:

BBC and CPDN trickles and uploads are failing. Milo has had to turn off all BOINC services for both projects so the database isn't updated while he makes a copy of it. If possible please suspend BOINC network activity until the server status page is available and shows the upload server programs working.

Malariacontrol.net seems to be back on-line again after a couple of days away. Nothing about the outage on their home page so far tho...

From Nick at Malariacontrol.net:

It seems that the web-server was very unreliable over the last few days. As a consequence, the website and also the job scheduler here on the server could be reached only intermittently.

We have restarted the server and currently it seems to work fine. We have increased the grace period for overdue results to 3 three days to minimize the impact of this problem on your credit accounts. In addition, we have set up an additional monitoring service that will help us respond faster in case this should happen again.

We apologize for the inconvenience!
Nick

From Nick at Malariacontrol.net:
It seems that the web-server was very unreliable over the last few days. As a consequence, the website and also the job scheduler here on the server could be reached only intermittently.

We have restarted the server and currently it seems to work fine. We have increased the grace period for overdue results to 3 three days to minimize the impact of this problem on your credit accounts. In addition, we have set up an additional monitoring service that will help us respond faster in case this should happen again.

We apologize for the inconvenience!
Nick

Thanks for the info Mr Ageless, sir.

"grace period"? I hadn't realised there was a grace period! :-)

"grace period"? I hadn't realised there was a grace period! :-)

I like it that it's "3 three days" .. does that mean it's 9 or 33 days? ;-)

CPDN main project and perhaps also BBC

Milo has posted on the independent forum:

The main database has been back on-line since yesterday; thanks for your patience during this outage.
We are still undergoing some database maintenance and cleanup and so there may well be periods in the near future when it goes down again. We will attempt to give as much warning as possible when this is likely to happen.

CPDN main project

A member trying to upload a model's zip file has these BOINC messages:

|climateprediction.net|[error] Error on file upload: can't open file /home/cpdn/boinc/hadsm3fub_k2xu_005968597_1_3.zip: No space left on device
|climateprediction.net|Temporarily failed upload of hadsm3fub_k2xu_005968597_1_3.zip: transient upload error

The 'device' doesn't mean your own computer's disk. A server disk in Oxford has filled up. Members who see this message should if possible please suspend BOINC network activity until Milo can fix the server problem. Trickles may also be affected.

Because there are several CPDN upload servers, only some models will be affected.

That server problem has now been fixed.

But please see the CPDN website index page for details of a planned server maintenance outage on Monday 10 Nov. Members who can do so should suspend BOINC network activity before the outage begins. Credits will probably not be exported to the stats sites until Tuesday or Wednesday.

CPDN main project and BBC

Milo says 'The database and BOINC daemons are now back on line - thanks again for everyone's patience.'.

I'm wondering if the Rosetta servers are down. My CPU is communicating with everyone else but them.

---

It turns out their file servers crashed the night of Nov 13-14. It will still have intermittant problems, but it is up and running right now.

---

One of the CPDN main project upload servers has been intermittently refusing to accept trickle and file uploads and Milo hasn't yet discovered why. Due to this server's location he only has access to restart it during working hours.

If your model's uploads are affected, you can suspend BOINC network activity and from time to time check CPDN server status here.