What is wrong with the SAP server?

If you don't want to know the technical details, you only need to read the green section below.

Milo says

Apologies for the SAP problems.
What is going on is as follows. The server software needs upgrading as well as the whole OS re-installing, new disks adding and the database fixing. Some of this we don't have time to do and others we can't due to the location of the server. So, as a temporary measure I changed the server so that requests would be re-directed to the main site instead, thus avoiding some of the broken bits.
This was supposed to not affect BOINC, but has not worked as planned. I can quite quickly restore the server to its previous state which will fix BOINC, but I can't ssh in to Physics right now. When I do, and revert it to its previous state, it will need some more updating and so it will end up going up and down again as before.
With luck I can get this done tomorrow morning.

The SAP server is located in a different department at Oxford University, which is why Milo hasn't got full and normal access to it. The problem is exacerbated by a recent serious Linux security problem/vulnerability affecting computers with Debian and Ubuntu. Most of the Physics department computers use Ubuntu. As a result of this, the Physics department has further restricted access to its computers and servers, including the SAP server, until they are sure the security problem is fixed.

In addition, the Physics Department only allows Milo and Tolu to access the SAP server remotely via a single CPDN computer, and this computer isn't behaving very well at the moment.

We will try to keep members informed about what's happening, but of course Milo can't constantly keep in touch with us while he's fixing problems.

If you have a SAP HADAM model:

* Don't detach from the CPDN or SAP projects
* Don't reset the ClimatePrediction or SAP projects
* Don't abort any SAP HADAM models
* Do suspend HADAM models in the Tasks tab of BOINC manager
* If you are already receiving unpleasant BOINC manager messages about your HADAM model, suspend BOINC network activity if possible.

SAP

Milo has fixed the immediate problem of the SAP redirect to CPDN that didn't work as planned and produced the dreadful BOINC manager messages quoted two posts above this. If you have messages like that, please ignore them and don't do what they suggest!

The SAP website and forum are now running but the SAP server status page looks as if the SAP server won't accept trickle and zip file uploads at the moment.

The problem of the SAP server redirect to CPDN doesn't appear to have been really solved. Thyme Lawn has posted a fix on the SAP forum here .

Thyme Lawn has now modified those instructions. The link to them in the above post is still correct.

SAP

This afternoon Milo managed to get into the Physics department. He has cancelled the SAP server redirect to the CPDN server and upgraded the server software. The SAP server is now completely up and running, though Milo can't guarantee its future reliability.

It should now be possible to run SAP models again and allow them to send trickles and uploads to the server.

Hydrogen@Home has problems.

Jack Shultz wrote:

So I tried to restore the snapshot. Turns out that snapshot had a corruption. Good news is I got the service provider to recover the project directory and post it on an FTP so I can rebuild and restore everything like new. I have a database backup. I expect to have it back up in the next two days.

Jack

You wonder what happened to Hydrogen@Home?

Jack Shultz wrote:

Hi All,

I just made a snapshot. After which I deleted something unintentionally, I just restored the snapshot, when I boot, none of the services are initiating. No apache, ssh, no ping. I submitted a ticket for this provider to look at it. Hopefully I don't have to rebuild again. If that's the case, I'm going to start looking at new providers. I can't run the project if the snapshots are unreliable.

Jack

cpdn & Seasonal Attribution Project

Some more work has been done on the servers, which should fix the last problems with trickles, on all sites.

CPDN main project

CPDN server status currently appears normal

http://climateapps2.oucs.ox.ac.uk/cpdnboinc/server_status.php

but model results pages are in many cases hanging and refusing to display. Milo will look at the database on the server tomorrow to see whether there's an overload.

CPDN main project

The CPDN server (climateapps2.oucs.ox.ac.uk) has been unstable since about 1000 UTC today. Milo did sort out the initial problem by restarting the HTTP server, but most connections have been failing since about 1200 UTC. This is likely to be the case until at least Monday morning.

The current problems will affect all scheduler requests (i.e. trickles, reporting completed results and work requests). Uploads will continue to work as they are sent to different servers.

The other CPDN projects (BBC Climate Change Experiment, Seasonal Attribution Poject and CPDN Beta) are unaffected.

---

"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer

What happened to The Lattice Project ?

What happened to The Lattice Project ?

June 6, 2008 Unexpected Outage
Some days ago we experienced an HVAC failure which caused our main file server to overheat. We have since been working on fixing the problems that resulted, and are still in the process of doing so, but we hope to be back online and fully functional shortly. As for the outstanding work in the system, we'll just have to see how well the system recovers. Sorry for the extended absence!

The servers are back up, but there is no work available this weekend. Perhaps by mid-day on 9 June.

---

CPDN main project

Tolu worked on the database this morning to try to speed it up. He disabled the database server over lunchtime so the CPDN-BOINC forum was down for a while as well as access to our account and model pages. Everything appears to be running normally now.

CPDN Beta2 project only

Here's the Beta2 server status page showing one of the server programs down:

http://cpdnbeta.oerc.ox.ac.uk/server_status.php

The best idea is if possible to suspend network activity to avoid multiple failed trickle attempts. Zip files will probably upload successfully but not the accompanying trickle.

The CPDN Beta2 server programs have all now been up and running for several hours.

Seeing all the errors at Einstein@Home, I think they're now technically off line. No news from anyone on what is happening though.

Seeing all the errors at Einstein@Home, I think they're now technically off line. No news from anyone on what is happening though.

Einstein is working fine for me, didn't notice any complaints on the msg boards; what are the "errors" to which you refer?

Einstein is working fine for me, didn't notice any complaints on the msg boards; what are the "errors" to which you refer?

It's working for me too (again). Einstein has been offline for some time yesterday. I was wondering about the lack of information about the outage.

Looks like the Lattice website is off-line again since about 19 June. The servers have been out of work all week, but are still responding to BOINC mgr queries -- to say the project is down for maintenance.

---

Milkyway seems to be unaccessible.
I tried this morning and it connected and gave the message 'the servers may be down.'

Now I get a 'connect error'.

My dialup won't go to the website.