News on project outages

Message boards : Projects : News on project outages
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 . . . 10 · Next

AuthorMessage
Profile Jord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 15478
Netherlands
Message 30347 - Posted: 26 Dec 2009, 19:24:28 UTC
Last modified: 26 Dec 2009, 19:29:25 UTC

Due to some major power upgrades happening elsewhere on Berkeley campus, Seti@Home has a Scheduled Power Outage on Sunday (12/27/2009) at 12 noon (Pacific time) which is 20h UTC. Then they will begin a 24 hour power outage. All services will be off line during that period.

Since the BOINC servers live in the same building, it is very much possible that they will be off line during this period as well. I am trying to get confirmation on that.
ID: 30347 · Report as offensive
BarryAZ

Send message
Joined: 4 Sep 09
Posts: 381
United States
Message 30356 - Posted: 28 Dec 2009, 22:24:59 UTC - in response to Message 30347.  

Thanks for the notice, it appears, as often is the case with SETI, that 24 hours is perhaps best described as 24 airport hours. That is, 24 hours is the minimum length of the outage. Since nothing is up now at 2:30PM Pacific Time, I'd guess that we have several more hours before things are up, and then of course, an extended traffic jam period -- which might well extend into the weekly Tuesday outage.


Due to some major power upgrades happening elsewhere on Berkeley campus, Seti@Home has a Scheduled Power Outage on Sunday (12/27/2009) at 12 noon (Pacific time) which is 20h UTC. Then they will begin a 24 hour power outage. All services will be off line during that period.

Since the BOINC servers live in the same building, it is very much possible that they will be off line during this period as well. I am trying to get confirmation on that.

ID: 30356 · Report as offensive
BarryAZ

Send message
Joined: 4 Sep 09
Posts: 381
United States
Message 30357 - Posted: 29 Dec 2009, 0:51:40 UTC - in response to Message 30356.  

Looks like they got the power restored around 4PM PST (about 2 hours ago), the project is still bringing various servers back online. I'd guess the post outage rush will continue pretty deep into the night and perhaps extend to their Tuesday morning maintenance outage.

The same cycle will repeat next weekend as well (planned power outage on January 3 for 24+ hours).


Thanks for the notice, it appears, as often is the case with SETI, that 24 hours is perhaps best described as 24 airport hours. That is, 24 hours is the minimum length of the outage. Since nothing is up now at 2:30PM Pacific Time, I'd guess that we have several more hours before things are up, and then of course, an extended traffic jam period -- which might well extend into the weekly Tuesday outage.


Due to some major power upgrades happening elsewhere on Berkeley campus, Seti@Home has a Scheduled Power Outage on Sunday (12/27/2009) at 12 noon (Pacific time) which is 20h UTC. Then they will begin a 24 hour power outage. All services will be off line during that period.

Since the BOINC servers live in the same building, it is very much possible that they will be off line during this period as well. I am trying to get confirmation on that.


ID: 30357 · Report as offensive
Profile Jord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 15478
Netherlands
Message 30359 - Posted: 29 Dec 2009, 12:11:30 UTC

From Seti News:
Scheduled Power Outage - update
We are now recovering from a planned power outage, during which all services were off line. The workunit storage machine is undergoing a RAID resync and no new workunits will be generated until this has finished. The outage was for power upgrades on campus, and a similar outage will happen again this upcoming Sunday (01/03/2010) at 12 noon (Pacific time).
ID: 30359 · Report as offensive
Eric Myers
Avatar

Send message
Joined: 12 Feb 06
Posts: 232
United States
Message 30371 - Posted: 30 Dec 2009, 16:41:23 UTC

The Einstein@Home server is very sluggish. From the status page it looks like the transitioner is off-line, which would cause everything to come to a halt (basically the server seems to be constipated :-)

-- Eric Myers

"Education is not the filling of a pail, but the lighting of a fire." -- William Butler Yeats
ID: 30371 · Report as offensive
Profile Jord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 15478
Netherlands
Message 30376 - Posted: 30 Dec 2009, 19:15:28 UTC

Seti@Home News wrote:
A Fairly Short Outage
We are bringing the projects down for a MySQL database backup. No database compression will be done this week. The upload/download servers and some web pages are offline during this time.
ID: 30376 · Report as offensive
Profile Jord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 15478
Netherlands
Message 30377 - Posted: 30 Dec 2009, 21:30:33 UTC - in response to Message 30371.  

Einstein: David H. came back to me with the following, "There was a duplicate entry in the database that caused the transitioner to quit. I don't know what caused this problem and I asked Bernd if he understands. Hopefully it will not happen again before we figure out what happened."
ID: 30377 · Report as offensive
Eric Myers
Avatar

Send message
Joined: 12 Feb 06
Posts: 232
United States
Message 30379 - Posted: 30 Dec 2009, 22:12:08 UTC - in response to Message 30371.  

Eric Myers wrote:
The Einstein@Home server is very sluggish. From the status page it looks like the transitioner is off-line, which would cause everything to come to a halt (basically the server seems to be constipated :-)

And they've now got things moving again.

-- Eric Myers

"Education is not the filling of a pail, but the lighting of a fire." -- William Butler Yeats
ID: 30379 · Report as offensive
Profile Jord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 15478
Netherlands
Message 30381 - Posted: 30 Dec 2009, 22:19:26 UTC - in response to Message 30379.  

Or not, as it seems the scheduler is still down.

30-Dec-09 23:17:15 Einstein@Home Message from server: Project is temporarily shut down for maintenance
30-Dec-09 23:17:15 Einstein@Home Project requested delay of 3600 seconds
30-Dec-09 23:17:15 Einstein@Home [sched_op_debug] Deferring communication for 1 hr 0 min 0 sec
ID: 30381 · Report as offensive
Heidi1
Avatar

Send message
Joined: 30 Mar 08
Posts: 18
United States
Message 30383 - Posted: 31 Dec 2009, 6:53:17 UTC

I didn't see it here, so I'll add it to the list: Rosetta@Home is having a lot of problems right now, and has been since the SAN was upgraded on 23Dec. Most everyone can't get work and about half of the server stats are still in the red.
ID: 30383 · Report as offensive
Bernd

Send message
Joined: 24 Aug 09
Posts: 91
United States
Message 30399 - Posted: 1 Jan 2010, 7:33:22 UTC

Seti seems to be having problems. The Master DB is disabled. The front page and anything pulling data from that DB is not working. Graphs show a falloff in data going in and out of the project.
ID: 30399 · Report as offensive
BarryAZ

Send message
Joined: 4 Sep 09
Posts: 381
United States
Message 30404 - Posted: 1 Jan 2010, 17:37:34 UTC - in response to Message 30399.  

Indeed -- their master database crashed -- and not only do they have the long weekend to contend with, but also on Sunday there will be a 24 hour power outage.

The power outage was announced, so some 'SETI only' folks were likely maxing out their cache when the database went boom. It isn't that big a deal for me -- in that I run 10 active projects, and with the really long due dates that SETI has, I simply suspend its processing for the duration, sending CPU and GPU cycles to other projects.

With SETI being by far the largest BOINC project in terms of active users and (I suspect) in terms of budget and infrastructure, when it goes bump, there is a fair amount of work shifting that goes on for those with multiproject configurations. Those that remain SETI only do get a tad discomfited though.


Seti seems to be having problems. The Master DB is disabled. The front page and anything pulling data from that DB is not working. Graphs show a falloff in data going in and out of the project.

ID: 30404 · Report as offensive
Profile Gundolf Jahn

Send message
Joined: 20 Dec 07
Posts: 1069
Germany
Message 30407 - Posted: 1 Jan 2010, 18:28:27 UTC - in response to Message 30399.  

Seti seems to be having problems. The Master DB is disabled. The front page and anything pulling data from that DB is not working. Graphs show a falloff in data going in and out of the project.

Meanwhile, the front page says:
Projects are down due to a database machine crash.
The machine that serves the BOINC database crashed last night. Until this is fixed we cannot send out work.
1 Jan 2010 16:27:45 UTC

Gruß,
Gundolf
ID: 30407 · Report as offensive
BarryAZ

Send message
Joined: 4 Sep 09
Posts: 381
United States
Message 30410 - Posted: 1 Jan 2010, 21:09:16 UTC - in response to Message 30407.  

Correct -- as I noted previously.

My own sense, given the long weekend, is that they simply won't be able to even look at the database problems until AFTER the power outage (24+ hours starting sometime on 1/3). That being the likely scenario, it may well be until after the regular Tuesday outage (4+ hours on 1/5) that SETI will be back online. And, given the very thin margin for I/O handling that SETI works with, and a likely extended (as in 5 days) outage, we may well be looking at perhaps sometime on Wednesday (perhaps late Wednesday) before things settle down.

Seti seems to be having problems. The Master DB is disabled. The front page and anything pulling data from that DB is not working. Graphs show a falloff in data going in and out of the project.

Meanwhile, the front page says:
Projects are down due to a database machine crash.
The machine that serves the BOINC database crashed last night. Until this is fixed we cannot send out work.
1 Jan 2010 16:27:45 UTC

Gruß,
Gundolf

ID: 30410 · Report as offensive
Aurora Borealis
Avatar

Send message
Joined: 8 Jan 06
Posts: 448
Canada
Message 30416 - Posted: 2 Jan 2010, 8:42:12 UTC

It looks like most of the Seti servers are up again. The boys must have spent a good part of their new years day working as things came up one piece at a time. I just hope no one actually had to trek out to the campus to kick them.

Boinc V 7.4.36
Win7 i5 3.33G 4GB NVidia 470
ID: 30416 · Report as offensive
Profile Jord
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 29 Aug 05
Posts: 15478
Netherlands
Message 30427 - Posted: 3 Jan 2010, 13:38:58 UTC

Warning: Tonight from 20.00 hours UTC the Berkeley Campus will have another 24 hour scheduled power outage. This will mean that Seti@Home, Seti Beta and all of the BOINC domain will be off line again during that time period.
ID: 30427 · Report as offensive
Profile Gary Charpentier
Avatar

Send message
Joined: 23 Feb 08
Posts: 2462
United States
Message 30430 - Posted: 4 Jan 2010, 23:00:33 UTC - in response to Message 30427.  

Warning: Tonight from 20.00 hours UTC the Berkeley Campus will have another 24 hour scheduled power outage. This will mean that Seti@Home, Seti Beta and all of the BOINC domain will be off line again during that time period.

and looks like SETI is staying down
Scheduled Power Outage - Update
We mostly recovered from our scheduled power outage this weekend, but due to minor problems we decided it would be best to keep the projects offline another day - however we're folding in the usual Tuesday outage today. So if all goes well we'll be fully up again by tomorrow (Tuesday) morning (if not sooner). 4 Jan 2010 22:19:26 UTC
ID: 30430 · Report as offensive
Profile Gary Charpentier
Avatar

Send message
Joined: 23 Feb 08
Posts: 2462
United States
Message 30469 - Posted: 9 Jan 2010, 7:27:53 UTC

Not that they had any work, but LHC seems to be unreachable.
ID: 30469 · Report as offensive
mo.v
Avatar

Send message
Joined: 13 Aug 06
Posts: 778
United Kingdom
Message 30494 - Posted: 11 Jan 2010, 23:14:34 UTC

CPDN main project

Many members are still processing HadAM3P models which are not currently available for download. Their trickles go to the scheduler and are uploading without problems. But there's an upload problem with one of the three final files.

#1 file uploads to upload server uploader.oerc
#2 file to cpdn-uploader1.comlab
#3 file to uploader1.atm

Uploader.oerc has filled up as can be seen on the server status page. #2 and #3 files should upload without problems but #1 file cannot. Please do not try to force this file to upload.

Milo is moving data from the uploader.oerc server and hopes it will be able to accept uploads sometime on Tuesday.

You could

* do nothing and let Boinc attempt the upload at intervals (current attempts will fail)
or * suspend HadAM3P models before completion so they do not produce the final files while the server is down
or * suspend network activity in the Boinc Manager Activity menu

ID: 30494 · Report as offensive
Keith T
Avatar

Send message
Joined: 26 Feb 07
Posts: 71
United Kingdom
Message 30498 - Posted: 12 Jan 2010, 14:16:30 UTC

SETI@home has had an unscheduled outage today. SETI Beta and SETI classic sites are also down, so is seti.berkeley.edu

Keith
ID: 30498 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 . . . 10 · Next

Message boards : Projects : News on project outages

Copyright © 2024 University of California.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.