It is looking extremely likely that Climateapps2.oucs.ox.ac.uk will require an emergency shutdown this afternoon (11 July 2011) at 15.00 BST.

We do not know how long the server will be down, but it is possible that it will be for several days.
Obviously the BOINC forum will be unavailable during this time.

I will endeavour to keep everyone informed of the status of the repair.

Please accept my apologies for the inconvenience.

Jonathan

Jonathan Miller
CPDN SysAdmin

We have brought the BOINC server climateapps2.oucs back online this afternoon 15 July 2011 at approximately 16.00 BST
We think she is working correctly, but if you experience problems please check the boards for information from project staff and moderators.
Jonathan Miller
CPDN SysAdmin

Climateapps2 has been (mostly) restored, with an upgrade to both the OS and the BOINC server software. It was complicated by the server being located in a room of a different department, and needing the IT person from there to do the work.
Hopefully things will be more stable for a while. :)

Les Bayliss
Forum moderator
ClimatePrediction.Net - (AKA CPDN)

climateapps2.oucs appears to have suffered a serious injury, and is not responding to messages from Jonathan. Unfortunately, it lives in a foreign country, and Jonathan hasn't been able to obtain a visa to visit it, so there's no information on what is wrong, or when it might be OK again.

Les Bayliss
Forum Admin and moderator
ClimatePrediction.Net - (AKA CPDN)

We have moved climateapps2.oucs to a new home in a server room to which CPDN staff do actually have access.Over the coming weeks and months we will be carrying out similar transfers of other project servers, and will try to do this with prior notice and a minimum of disruption.In the mean time, climateapps2.oucs should be back on line as soon as the DNS propagation of her new IP address has taken place, this could be up to 24 hours for some participants.

Jonathan Miller
CPDN SysAdmin
ClimatePrediction.Net - (AKA CPDN)

Hi,

We think that the project is up and running again (with the exception of some credit processing scripts). The problems started with a failure of the hard drive of the main project server. Part way through the repair, the University's Computing Services department decided to enact a policy that meant that CPDN staff were excluded from the machine room in which the server was located. For the previous 7 years CPDN staff have been granted access to the machine room without comment or concern. No explanation has been given by Computing Services for their actions.

I was forced to complete the repair without physical access to the machine, which meant that the repair took hours longer than was required. The server was brought online on the Friday evening.

Unfortunately, after a period of excessive load, the server became unresponsive again on Wednesday evening. Computing Services were unwilling to allow CPDN staff access to the machine, meaning that non-CPDN staff would have been required to repair the server.

It was decided to remove the server from the machine room at Computing Services to rehome her in a server room to which CPDN staff do have access. This move required the reconfiguration of a number of CPDN servers, and because of another hitherto unannounced policy of Computing Services, a change of DNS name for the server.

After formal representations by CPDN managers Computing Services agreed to allow the use of the old DNS name for six months to allow a period of transition. With this agreement in place Computing Services allowed access to the main project server through the University Firewall, granting volunteers access via the web.

I apologise for the delay in returning the project to operational status, and I regret that it may seem that no-one is bothered about the project status. However, CPDN staff do care very strongly about the project, but we sometimes have to deal with bureaucracy that has no clear purpose, but has a detrimental effect on the service we can provide.

CPDN staff and managers are committed to bringing this project back to rude health, and have plans in place that should mean that the service is robust and resilient.

CPDN has come a long way in the years that it has been running, though its budget has always been small. This is thanks entirely to the effort put in by you, the volunteers, without whom the project simply would not exist.

Please bear with us, things are getting better.

Jonathan Miller
CPDN SysAdmin
ClimatePrediction.Net - (AKA CPDN)

The BOINC server IP seems to have changed, if you still experience problems connecting to the scheduler, you might need to restart your BOINC client.
Allem Anschein nach hat sich die IP des BOINC-Servers geaendert, wenn weiterhin Probleme beim Schedulerkontakt auftreten, muesst Ihr evtl. Euren BOINC-Client einmal durchstarten.

edit :

This fixes only the "Couldn't connect to server" error, server side errors cannot be fixed that easily.
Das behebt nur den Fehler "Couldn't connect to server", Fehler auf Serverseite kann man leider nicht so leicht beheben.

The project server is online and running, having been rehoused in a new datacentre. There are a number of issues relating to trickles and credit allocation, but CPDN staff are looking at these as I write. If you are experiencing any problems with the CPDN project please do let us know by posting on the user support boards.

Jonathan Miller
CPDN SysAdmin
ClimatePrediction.Net - (AKA CPDN)

The recent 2 batches of regional models, (pnw and then eu), have a problem with trying to return the final file, zip13.
This is the restart file, and is intended to go to climateapps1.oucs. This is one of the servers that was moved from the oerc department to the oucs department. Redirects were put in place for the present, but the one to climateaps1 isn't working.

This is resulting in HTTP errors when attempting to upload zip13 files.

To fix this, an edit of client_state.xml is necessary.
========================

Suspend BOINC in the manager, and then exit from both the manager and the client parts.
With a plain text editor, e.g. Notepad, open client_state.xml.
Locate the uploader section of the pnw/eu model.
Keep "locating" until you reach the one for zip13.

It's in the <file_info> section for each model, just after <upload_when_present/>.

DON'T touch the second one! - It's in a signed (security) section!

Change the 4 characters in the string uploader.oerc.ox.ac.uk from oerc to oucs

Do this for all pnw/eu models THAT ARE RUNNING.

Save the file
Restart BOINC.
The files should now upload. (Been there, done that, as they say. )
========================

Don't bother with pnw models that haven't started yet. They have another problem, and should be aborted. (They're going to be regenerated.)
(The project may use a Killer trickle on these unstarted models.)

Also abort them if they've been started, but you haven't applied the fpops fix to them.

This is getting complicated, so feel free to post. The big problem is the people who don't read either of the boards.

Les Bayliss
Forum Admin and moderator
ClimatePrediction.Net (AKA CPDN)

Les Bayliss wrote:
Change the 4 characters in the string uploader.oerc.ox.ac.uk from oerc to oucs

Thyme Lawn wrote:
A DNS redirection is now in place and the final upload for HadAM3P regional models
(the *_13.zip file) should now work without this change.
If you continue to have problems uploading the final upload file please let us know.

Thyme Lawn
Forum Admin and moderator
ClimatePrediction.Net - (AKA CPDN)

I've just finished uploading the last file (zip13), for an eu model,
so the DNS redirect is working.

Les Bayliss
Forum Admin and moderator
ClimatePrediction.Net - (AKA CPDN)

Posts are starting to appear asking about the next lot of work, so this may be best answered as a news item.

At present there are only 2 groups of researchers, the RAPIT group, and the dual resolution (regional models) group.
The original models for both groups were created and sent out ages ago, and, as they get returned, the next step in the series is automatically created by an 'auto-regen' program.
Only those models that make it all the way to the end will be continued to the next stage. Those that become unstable and fail, and those that are abandoned or aborted won't go any further.

Occasionally, the researchers may ask for more new series to be created, if it looks like there won't be enough making it to form a good sample.

With only a few thousand models, and 35,000+ computers connected, you just need to be patient if you're after work.

Some of the latest batch of hadam3p models have a problem with 'permissions' on one of the mirror servers.
This will cause 2 of the ancillary files to have a temporary download failure.
And it's the weekend at Oxford again.
Les Bayliss
Forum moderator

The permissions problem causing HADAM3P download errors should now have been fixed.
If anyone gets a new occurence of this problem then please report it in this thread.
Iain Inglis
Forum moderator

On 18 October 2011 the CPDN project server was attacked.

We have had to respond by taking the project offline.

I will provide more info as I have it, but I will be devoting my time to fixing the issues and determining the extent of the problem.

Jonathan