News on project outages.

Message boards : Projects : News on project outages.
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 . . . 15 · Next

AuthorMessage
Profile Jord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 15477
Netherlands
Message 23123 - Posted: 15 Feb 2009, 4:11:03 UTC
Last modified: 15 Feb 2009, 4:15:49 UTC

It looks like Einstein@Home just got problems with their database. All it says is "Unable to connect to database - please try again later Error: 1040Too many connections" on their main page.

We'll wait for them to return.

Edit: talk about service. They're back already. ;-)
ID: 23123 · Report as offensive
mo.v
Avatar

Send message
Joined: 13 Aug 06
Posts: 778
United Kingdom
Message 23385 - Posted: 1 Mar 2009, 11:10:22 UTC

CPDN main project

A member has just reported the following BOINC Manager messages when trying to upload a CPDN model zip file:

01/03/2009 09:43:49|climateprediction.net|[error] Error reported by file upload server: can't write file /storage/incoming/uploader/hadcm3istd_cpbk_1920_160_06016948_6_1.zip: No space left on server

01/03/2009 09:43:49|climateprediction.net|Backing off 24 min 16 sec on upload of hadcm3istd_cpbk_1920_160_06016948_6_1.zip


The disk of one of the CPDN upload servers must be full. Milo now knows about the problem but doesn't yet know which upload server is affected. He will probably not be able to solve the problem immediately as he will need to move data to another disk.

As there are several upload servers, this will affect file uploads from some models but not others. If you receive the same or similar messages when trying to upload a model file, please suspend BOINC network activity for the time being to avoid repeated failed upload attempts. We realise that multi-project crunchers may not be able to do this for long periods.

Trickle uploads should not be affected.
ID: 23385 · Report as offensive
mo.v
Avatar

Send message
Joined: 13 Aug 06
Posts: 778
United Kingdom
Message 23386 - Posted: 1 Mar 2009, 12:49:21 UTC

CPDN main project - update to previous post.

Milo says 'Climateapps3.oucs is the culprit. This will mean moving around 500GB of files off to wyrm.oerc, which is itself 94% full and has only about 700GB free. I've set this job going and hopefully it will finish by tomorrow, at which point I will turn climateapps3.oucs back on again.'

The CPDN server status page now shows climateapps3.oucs down. When it's up and running again it should accept model file uploads.
ID: 23386 · Report as offensive
mo.v
Avatar

Send message
Joined: 13 Aug 06
Posts: 778
United Kingdom
Message 23425 - Posted: 3 Mar 2009, 15:07:03 UTC

CPDN upload server climateapps3.oucs is up and running.
ID: 23425 · Report as offensive
mo.v
Avatar

Send message
Joined: 13 Aug 06
Posts: 778
United Kingdom
Message 23503 - Posted: 7 Mar 2009, 3:06:48 UTC

CPDN Beta Project

The Beta server is down. The forum cannot be accessed. Uploads of trickles and files fail as well as the reporting of completed models. Milo or Tolu will be investigating the problem on Saturday.
ID: 23503 · Report as offensive
mo.v
Avatar

Send message
Joined: 13 Aug 06
Posts: 778
United Kingdom
Message 23526 - Posted: 8 Mar 2009, 2:39:31 UTC

CPDN Beta Project

The server is now partially active. The forum is back online and zip files can be uploaded. But trickles cannot be sent and completed models cannot be reported.
ID: 23526 · Report as offensive
mo.v
Avatar

Send message
Joined: 13 Aug 06
Posts: 778
United Kingdom
Message 23631 - Posted: 12 Mar 2009, 12:29:35 UTC

CPDN main project

Tolu has taken down at least one CPDN server, climateapps2, while he generates more workunits. At the moment at least some members' trickle-ups are failing and there's no CPDN-BOINC forum. It would be a good idea to suspend BOINC network activity temporarily if you can.
ID: 23631 · Report as offensive
mo.v
Avatar

Send message
Joined: 13 Aug 06
Posts: 778
United Kingdom
Message 23637 - Posted: 12 Mar 2009, 16:27:01 UTC

CPDN main project

The CPDN server programs are coming back up. The CPDN forum and other data-driven web pages are back but some computers still can't upload trickles.

CPDN credits haven't updated today so they can't be exported to the stats sites. They should catch up tomorrow.

CPDN Beta project

Beta credits haven't been transferred to members' CPDN accounts for about two weeks. Milo says a firewall restriction sprang up between the two machines. This is now fixed but the transfer script won't run immediately. Thanks to Beta crunchers for your patience in this respect.
ID: 23637 · Report as offensive
mo.v
Avatar

Send message
Joined: 13 Aug 06
Posts: 778
United Kingdom
Message 23659 - Posted: 13 Mar 2009, 11:50:40 UTC
Last modified: 13 Mar 2009, 11:56:26 UTC

CPDN main project server climateapps2 is down again. There's no access to the website or CPDN-BOINC forum and trickles from at least some computers are failing.

The CPDN Beta server is partially down with the website and forum inaccessible. But it is accepting trickles.

I seem to post here more regularly about server problems than many other projects. But CPDN and its subprojects in fact have a good record of server uptime compared with other BOINC projects.
ID: 23659 · Report as offensive
Ed Meadows
Avatar

Send message
Joined: 28 Jun 07
Posts: 22
United States
Message 23678 - Posted: 14 Mar 2009, 17:30:42 UTC

Does anyone have any news about Cosmology@home? They had a power failure last weekend, came back up but then circled the drain all week. Now everything is offline, including their website.
ID: 23678 · Report as offensive
Warped
Avatar

Send message
Joined: 25 Aug 08
Posts: 39
South Africa
Message 23679 - Posted: 14 Mar 2009, 18:58:52 UTC - in response to Message 23678.  

Does anyone have any news about Cosmology@home? They had a power failure last weekend, came back up but then circled the drain all week. Now everything is offline, including their website.


Good question. There's been no contact or news for a week.
ID: 23679 · Report as offensive
ZPM
Avatar

Send message
Joined: 14 Mar 09
Posts: 215
United States
Message 23680 - Posted: 14 Mar 2009, 20:56:52 UTC - in response to Message 23679.  

i just joined cosmo. and it goes compute this = were sry, the site you have reached is currently getting an oil job. please try again.
ID: 23680 · Report as offensive
mo.v
Avatar

Send message
Joined: 13 Aug 06
Posts: 778
United Kingdom
Message 23691 - Posted: 15 Mar 2009, 12:02:53 UTC

The servers for the CPDN main project and CPDN Beta are all working properly again.

Beta members' credits are now being transferred to CPDN and exported to the stats sites.
ID: 23691 · Report as offensive
LifeisAGame

Send message
Joined: 15 Mar 09
Posts: 1
Switzerland
Message 23693 - Posted: 15 Mar 2009, 15:52:02 UTC

from Cosmology in my Cache:
The Cosmology@home server has been suffering abnormal crashes ever few hours since the campus power outage on March 8, 2009. We will be taking to server ...

Thats now 7 Days ago, looks they have massive Problems
ID: 23693 · Report as offensive
Profile Stefan Ledwina
Avatar

Send message
Joined: 25 Nov 05
Posts: 55
Austria
Message 23694 - Posted: 15 Mar 2009, 16:40:02 UTC

Just got an answer from Ben Wandelt. No real news about Cosmo, but at least Anshul is looking into the problem...

"Ben Wandelt" wrote:
Re: Cosmo@home down

Thanks stefan

I am out of town but anshul is looking into it. Probably a consequence of the massive power outage last weekend.

Ben


ID: 23694 · Report as offensive
Profile Jord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 15477
Netherlands
Message 23703 - Posted: 15 Mar 2009, 21:33:25 UTC

Better yet, I got lots of replies. ;-)

Ben Wandelt wrote:
Hi Jord

I am out of town. Anshul noticed that our server had intermittent problems starting after the massive power outage on the whole campus last Sunday. There were downed power lines after a storm. NCSA is still recovering from the aftermath. We were up almost immediately but now it seems we may have hardware damage. Anshul was looking at replacing parts but I have not heard from him since the beginning of the most recent outage.

So we are certainly aware of the problem but since I am at Caltech at the moment I cannot give you a detailed status update.

Ben


Anshul Kanakia wrote:
Hi Jord,
I had put up a new article on the website before it went down about Cosmo@home server maintenance. We started experiencing sporadic server crashes after the campus power failure so I took the server down. We are currently in the process of sending it out for maintenance to get the hardware checked. I think its most probably a hard drive failure - some sector that the OS tries to access that got locked or corrupted. We should be back up in 2 weeks at most. 2 weeks because next week is spring break on campus and I will not be around when the server gets back. Thanks for sending me this email. I was trying to figure out how to send a message to our users that we are down, and so is the website.

Anshul Kanakia


Scott Kruger wrote:
I will try to be here when the server gets back so we can get it up as soon as possible. I'll also see about getting a temporary website up.

-Scott

ID: 23703 · Report as offensive
Profile Jord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 15477
Netherlands
Message 23822 - Posted: 20 Mar 2009, 23:44:05 UTC

Seti@Home has no work at the moment. Their databases are off line, since they crashed earlier on Friday. It's being worked on.
ID: 23822 · Report as offensive
Profile Jord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 15477
Netherlands
Message 23826 - Posted: 21 Mar 2009, 12:07:51 UTC

Hydrogen has temporary server problems. It would seem that the work I made last night filled up the upload server and has now crashed the database.

I am sure though that Jack will be on it once he wakes up. To be continued.
ID: 23826 · Report as offensive
Profile Jord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 15477
Netherlands
Message 23845 - Posted: 22 Mar 2009, 13:01:20 UTC - in response to Message 23822.  
Last modified: 22 Mar 2009, 13:01:32 UTC

Seti@Home has got their servers back online. Now they're being hammered by everyone. Work is available, but you have to have patience to get it.
ID: 23845 · Report as offensive
Profile Jord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 15477
Netherlands
Message 23846 - Posted: 22 Mar 2009, 13:02:17 UTC - in response to Message 23826.  

Hydrogen's server has been back up since yesterday.
ID: 23846 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 . . . 15 · Next

Message boards : Projects : News on project outages.

Copyright © 2024 University of California.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.