Thread 'News on Project Outages'

Message boards : Projects : News on Project Outages
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 . . . 67 · Next

AuthorMessage
ProfileGary Charpentier
Avatar

Send message
Joined: 23 Feb 08
Posts: 2493
United States
Message 65006 - Posted: 20 Oct 2015, 22:09:54 UTC - in response to Message 65003.  

I have been uploading and downloading Einstein GPU work (BRP6-Beta for Nvidia), and also uploading FGRP4 work for several hours with no problems shown in the log here.

Einstein came back up about 1/2 hour after it went down.
ID: 65006 · Report as offensive     Reply Quote
Richard Haselgrove
Volunteer tester
Help desk expert

Send message
Joined: 5 Oct 06
Posts: 5129
United Kingdom
Message 65012 - Posted: 21 Oct 2015, 12:03:58 UTC - in response to Message 65006.  

And Einstein posted this at 12:00 UTC Wednesday 21 October:

We need to shut down the project for a few hours to do some urgent database maintenance. Your clients are able to handle that so the interruption should cause no harm.

Please note that during that time [the Einstein] website will be offline as well. As soon as it's available again, the project itself will be up and running as well.
ID: 65012 · Report as offensive     Reply Quote
Oliver Bock

Send message
Joined: 26 Jun 13
Posts: 8
Germany
Message 65014 - Posted: 21 Oct 2015, 13:45:44 UTC
Last modified: 21 Oct 2015, 13:46:40 UTC

Einstein@Home is available again. We restarted at about 13:25:00 UTC.
Einstein@Home Project
ID: 65014 · Report as offensive     Reply Quote
JIM

Send message
Joined: 19 Sep 10
Posts: 24
United States
Message 65050 - Posted: 23 Oct 2015, 3:43:40 UTC

Part of the CPDN site is up again. The “Homepage” and the “News” page are responding. The “Account” page and the Message Boards are still unresponsive. Zip file uploads are still not going through. Considering the backlog of zips waiting to upload after 6 days, the server may just be overloaded.
ID: 65050 · Report as offensive     Reply Quote
ProfileJord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 15563
Netherlands
Message 65055 - Posted: 23 Oct 2015, 16:56:41 UTC - in response to Message 64953.  

A further update on ClimatePrediction.net's predicaments:
A further update on the ongoing issues with the storage array on which the CPDN infrastructure resides:

Will in OeRC IT Support is continuing to work on restoring the storage array. The storage array has been rebuilt, and some services are being to be brought back online. These include a number of virtual machines, including the CPDN web server which is now back online. The CPDN project servers remain offline. Due to the expected back load of updates that will need to be performed with CPDN clients we will keep the CPDN project servers offline until we are confident that services on the storage array are sufficiently restored to be able to handle the load.

Once again, we apologise for the ongoing outage to services.
ID: 65055 · Report as offensive     Reply Quote
BarryAZ

Send message
Joined: 4 Sep 09
Posts: 381
United States
Message 65070 - Posted: 24 Oct 2015, 5:05:31 UTC

Collatz database went down about 5 hours ago (6PM west coast time).

It was steady for three weeks -- hope it bounces back up over the weekend.
ID: 65070 · Report as offensive     Reply Quote
Dayle Diamond

Send message
Joined: 5 Apr 13
Posts: 16
United States
Message 65081 - Posted: 26 Oct 2015, 1:31:53 UTC

Find Neglected Diseases is down.
WU aren't uploading, accounts and forums are down, and there's error messages all over the website.

http://findah.ucd.ie/
ID: 65081 · Report as offensive     Reply Quote
JIM

Send message
Joined: 19 Sep 10
Posts: 24
United States
Message 65083 - Posted: 26 Oct 2015, 15:08:10 UTC

A slight change in the responsiveness of the CPDN site. When I click the “Accounts”, “Help Desk”, “User Forum”, or “Team” button I not get the message :

“Forbidden

You don't have permission to access /cpdnboinc/team_display.php on this server.

Apache Server at climateapps2.oerc.ox.ac.uk Port 80”

Past attempts to access these pages have just resulted in a no connection time out message from the browser. So there does seem to be some life there, we are just not being let in yet.
ID: 65083 · Report as offensive     Reply Quote
ProfileByron Leigh Hatch @ team Carl ...
Avatar

Send message
Joined: 30 Aug 05
Posts: 505
Canada
Message 65085 - Posted: 26 Oct 2015, 15:43:40 UTC - in response to Message 65083.  

Just to confirm your reporting, yes I am getting the same.
ID: 65085 · Report as offensive     Reply Quote
Richard Haselgrove
Volunteer tester
Help desk expert

Send message
Joined: 5 Oct 06
Posts: 5129
United Kingdom
Message 65086 - Posted: 26 Oct 2015, 16:57:05 UTC

More CPDN progress to report:

We have now restarted the majority of services within the CPDN project including all central BOINC services. We are restarting the local (Oxford) upload server shortly in a manner that will allow us to control load into the main infrastructure so those waiting to upload there please be patient.
ID: 65086 · Report as offensive     Reply Quote
wol

Send message
Joined: 27 Oct 15
Posts: 1
Austria
Message 65098 - Posted: 27 Oct 2015, 14:18:18 UTC - in response to Message 65081.  
Last modified: 27 Oct 2015, 15:02:30 UTC

http://findah.ucd.ie/

find is working again, also the fora, accounts etc
ID: 65098 · Report as offensive     Reply Quote
ProfileByron Leigh Hatch @ team Carl ...
Avatar

Send message
Joined: 30 Aug 05
Posts: 505
Canada
Message 65104 - Posted: 27 Oct 2015, 18:10:52 UTC

Re: ClimatePrediction.Net (AKA CPDN)

I think the servers must be getting hammered with all the uploads,
and so I think I will suspend my network activity on BOINC,
for a day or so to give the servers a break.
patience is required :)

10/27/2015 9:33:42 AM | climateprediction.net | Temporarily failed upload of hadam3p_afr_e1uy_2011_1_010355437_0_3.zip: transient HTTP error
10/27/2015 9:33:42 AM | climateprediction.net | Backing off 03:43:10 on upload of hadam3p_afr_e1uy_2011_1_010355437_0_3.zip
10/27/2015 9:33:43 AM | | Internet access OK - project servers may be temporarily down.
10/27/2015 9:38:13 AM | climateprediction.net | Temporarily failed upload of hadam3p_eu_f4b0_1996_1_010197472_1_9.zip: transient HTTP error
10/27/2015 9:38:13 AM | climateprediction.net | Backing off 04:21:34 on upload of hadam3p_eu_f4b0_1996_1_010197472_1_9.zip
10/27/2015 9:38:14 AM | | Project communication failed: attempting access to reference site
10/27/2015 9:38:18 AM | | Internet access OK - project servers may be temporarily down.
ID: 65104 · Report as offensive     Reply Quote
ProfileDave
Help desk expert

Send message
Joined: 28 Jun 10
Posts: 2701
United Kingdom
Message 65121 - Posted: 28 Oct 2015, 12:16:12 UTC

Climateprediction.net AKA cpdn is down again

"Project is down
The project's database server is down. Please check back in a few hours. "
ID: 65121 · Report as offensive     Reply Quote
ProfileDave
Help desk expert

Send message
Joined: 28 Jun 10
Posts: 2701
United Kingdom
Message 65122 - Posted: 28 Oct 2015, 13:15:15 UTC - in response to Message 65121.  

And back up again.
ID: 65122 · Report as offensive     Reply Quote
BarryAZ

Send message
Joined: 4 Sep 09
Posts: 381
United States
Message 65127 - Posted: 28 Oct 2015, 19:52:46 UTC

Collatz was up for 5 days -- went down this morning -- and still is offline.

It's apparently the same data base crash that has bothered the project periodically for a long time.
ID: 65127 · Report as offensive     Reply Quote
ProfileByron Leigh Hatch @ team Carl ...
Avatar

Send message
Joined: 30 Aug 05
Posts: 505
Canada
Message 65151 - Posted: 30 Oct 2015, 15:41:01 UTC

Re: ClimatePrediction.Net (AKA CPDN)


The CPDN project will be taken offline for a scheduled downtime starting at 1000 UTC on Monday the 2nd November.
This is in order to perform firmware updates to the controllers of the storage array on which the CPDN servers reside. The firmware updates will correct an issue with the controllers that was a cause of the recent unscheduled downtime.
This work should not affect the climateprediction.net pages of the website, however the climateapps2.oerc portion of the website will be taken offline.
Will in OeRC support will be performing this work on the controllers of the storage array. At present we don't have an expected length of time that this firmware update will take, however we will endeavour to restore the project as soon as the updates have completed.

Source:
http://climateapps2.oerc.ox.ac.uk/cpdnboinc/forum_thread.php?id=8096
ID: 65151 · Report as offensive     Reply Quote
Profileritterm
Avatar

Send message
Joined: 4 Jul 08
Posts: 82
United States
Message 65279 - Posted: 5 Nov 2015, 18:21:46 UTC

CPDN seems to be making progress. I can access my account info, server status page, forums, etc., and reported a couple of recently completed tasks. However, I don't see any official looking announcement in the forum.
ID: 65279 · Report as offensive     Reply Quote
Richard Haselgrove
Volunteer tester
Help desk expert

Send message
Joined: 5 Oct 06
Posts: 5129
United Kingdom
Message 65280 - Posted: 5 Nov 2015, 19:19:50 UTC - in response to Message 65279.  

CPDN seems to be making progress. I can access my account info, server status page, forums, etc., and reported a couple of recently completed tasks. However, I don't see any official looking announcement in the forum.

There was a moderator briefing today:

The work to update the firmware has now completed and the project has now been partially restored. We are aware that a number of project services are still offline and we are working with OeRC support to bring these online as soon as possible. We apologise that the scheduled downtime took longer than expected.
ID: 65280 · Report as offensive     Reply Quote
BarryAZ

Send message
Joined: 4 Sep 09
Posts: 381
United States
Message 65321 - Posted: 8 Nov 2015, 16:09:20 UTC

Colattz seems to be under stress these days it appears difficult to get a full day run there without the database going offline -- as it is again this morning and has been fro a few hours.
ID: 65321 · Report as offensive     Reply Quote
James Lee

Send message
Joined: 6 Jan 14
Posts: 7
United States
Message 65324 - Posted: 9 Nov 2015, 1:03:38 UTC - in response to Message 65321.  

Yes.. my first day on adding this project, too. All my computers must have overwhelmed it. lol.. I hope Colattz works out.
ID: 65324 · Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 . . . 67 · Next

Message boards : Projects : News on Project Outages

Copyright © 2024 University of California.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.