Thread 'News on project outages'

Author	Message
Jord Volunteer tester Help desk expert Send message Joined: 29 Aug 05 Posts: 15542	Message 30347 - Posted: 26 Dec 2009, 19:24:28 UTC Last modified: 26 Dec 2009, 19:29:25 UTC Due to some major power upgrades happening elsewhere on Berkeley campus, Seti@Home has a Scheduled Power Outage on Sunday (12/27/2009) at 12 noon (Pacific time) which is 20h UTC. Then they will begin a 24 hour power outage. All services will be off line during that period. Since the BOINC servers live in the same building, it is very much possible that they will be off line during this period as well. I am trying to get confirmation on that. ID: 30347 ·

BarryAZ Send message Joined: 4 Sep 09 Posts: 381	Message 30356 - Posted: 28 Dec 2009, 22:24:59 UTC - in response to Message 30347. Thanks for the notice, it appears, as often is the case with SETI, that 24 hours is perhaps best described as 24 airport hours. That is, 24 hours is the minimum length of the outage. Since nothing is up now at 2:30PM Pacific Time, I'd guess that we have several more hours before things are up, and then of course, an extended traffic jam period -- which might well extend into the weekly Tuesday outage. Due to some major power upgrades happening elsewhere on Berkeley campus, Seti@Home has a Scheduled Power Outage on Sunday (12/27/2009) at 12 noon (Pacific time) which is 20h UTC. Then they will begin a 24 hour power outage. All services will be off line during that period. Since the BOINC servers live in the same building, it is very much possible that they will be off line during this period as well. I am trying to get confirmation on that. ID: 30356 ·

BarryAZ Send message Joined: 4 Sep 09 Posts: 381	Message 30357 - Posted: 29 Dec 2009, 0:51:40 UTC - in response to Message 30356. Looks like they got the power restored around 4PM PST (about 2 hours ago), the project is still bringing various servers back online. I'd guess the post outage rush will continue pretty deep into the night and perhaps extend to their Tuesday morning maintenance outage. The same cycle will repeat next weekend as well (planned power outage on January 3 for 24+ hours). Thanks for the notice, it appears, as often is the case with SETI, that 24 hours is perhaps best described as 24 airport hours. That is, 24 hours is the minimum length of the outage. Since nothing is up now at 2:30PM Pacific Time, I'd guess that we have several more hours before things are up, and then of course, an extended traffic jam period -- which might well extend into the weekly Tuesday outage. Due to some major power upgrades happening elsewhere on Berkeley campus, Seti@Home has a Scheduled Power Outage on Sunday (12/27/2009) at 12 noon (Pacific time) which is 20h UTC. Then they will begin a 24 hour power outage. All services will be off line during that period. Since the BOINC servers live in the same building, it is very much possible that they will be off line during this period as well. I am trying to get confirmation on that. ID: 30357 ·

Jord Volunteer tester Help desk expert Send message Joined: 29 Aug 05 Posts: 15542	Message 30359 - Posted: 29 Dec 2009, 12:11:30 UTC From Seti News: Scheduled Power Outage - update We are now recovering from a planned power outage, during which all services were off line. The workunit storage machine is undergoing a RAID resync and no new workunits will be generated until this has finished. The outage was for power upgrades on campus, and a similar outage will happen again this upcoming Sunday (01/03/2010) at 12 noon (Pacific time). ID: 30359 ·

Eric Myers Send message Joined: 12 Feb 06 Posts: 232	Message 30371 - Posted: 30 Dec 2009, 16:41:23 UTC The Einstein@Home server is very sluggish. From the status page it looks like the transitioner is off-line, which would cause everything to come to a halt (basically the server seems to be constipated :-) -- Eric Myers "Education is not the filling of a pail, but the lighting of a fire." -- William Butler Yeats ID: 30371 ·

Jord Volunteer tester Help desk expert Send message Joined: 29 Aug 05 Posts: 15542	Message 30376 - Posted: 30 Dec 2009, 19:15:28 UTC Seti@Home News wrote: A Fairly Short Outage We are bringing the projects down for a MySQL database backup. No database compression will be done this week. The upload/download servers and some web pages are offline during this time. ID: 30376 ·

Jord Volunteer tester Help desk expert Send message Joined: 29 Aug 05 Posts: 15542	Message 30377 - Posted: 30 Dec 2009, 21:30:33 UTC - in response to Message 30371. Einstein: David H. came back to me with the following, "There was a duplicate entry in the database that caused the transitioner to quit. I don't know what caused this problem and I asked Bernd if he understands. Hopefully it will not happen again before we figure out what happened." ID: 30377 ·

Eric Myers Send message Joined: 12 Feb 06 Posts: 232	Message 30379 - Posted: 30 Dec 2009, 22:12:08 UTC - in response to Message 30371. Eric Myers wrote: The Einstein@Home server is very sluggish. From the status page it looks like the transitioner is off-line, which would cause everything to come to a halt (basically the server seems to be constipated :-) And they've now got things moving again. -- Eric Myers "Education is not the filling of a pail, but the lighting of a fire." -- William Butler Yeats ID: 30379 ·

Jord Volunteer tester Help desk expert Send message Joined: 29 Aug 05 Posts: 15542	Message 30381 - Posted: 30 Dec 2009, 22:19:26 UTC - in response to Message 30379. Or not, as it seems the scheduler is still down. 30-Dec-09 23:17:15 Einstein@Home Message from server: Project is temporarily shut down for maintenance 30-Dec-09 23:17:15 Einstein@Home Project requested delay of 3600 seconds 30-Dec-09 23:17:15 Einstein@Home [sched_op_debug] Deferring communication for 1 hr 0 min 0 sec ID: 30381 ·

Heidi1 Send message Joined: 30 Mar 08 Posts: 18	Message 30383 - Posted: 31 Dec 2009, 6:53:17 UTC I didn't see it here, so I'll add it to the list: Rosetta@Home is having a lot of problems right now, and has been since the SAN was upgraded on 23Dec. Most everyone can't get work and about half of the server stats are still in the red. ID: 30383 ·

Bernd Send message Joined: 24 Aug 09 Posts: 91	Message 30399 - Posted: 1 Jan 2010, 7:33:22 UTC Seti seems to be having problems. The Master DB is disabled. The front page and anything pulling data from that DB is not working. Graphs show a falloff in data going in and out of the project. ID: 30399 ·

BarryAZ Send message Joined: 4 Sep 09 Posts: 381	Message 30404 - Posted: 1 Jan 2010, 17:37:34 UTC - in response to Message 30399. Indeed -- their master database crashed -- and not only do they have the long weekend to contend with, but also on Sunday there will be a 24 hour power outage. The power outage was announced, so some 'SETI only' folks were likely maxing out their cache when the database went boom. It isn't that big a deal for me -- in that I run 10 active projects, and with the really long due dates that SETI has, I simply suspend its processing for the duration, sending CPU and GPU cycles to other projects. With SETI being by far the largest BOINC project in terms of active users and (I suspect) in terms of budget and infrastructure, when it goes bump, there is a fair amount of work shifting that goes on for those with multiproject configurations. Those that remain SETI only do get a tad discomfited though. Seti seems to be having problems. The Master DB is disabled. The front page and anything pulling data from that DB is not working. Graphs show a falloff in data going in and out of the project. ID: 30404 ·

Gundolf Jahn Send message Joined: 20 Dec 07 Posts: 1069	Message 30407 - Posted: 1 Jan 2010, 18:28:27 UTC - in response to Message 30399. Seti seems to be having problems. The Master DB is disabled. The front page and anything pulling data from that DB is not working. Graphs show a falloff in data going in and out of the project. Meanwhile, the front page says: Projects are down due to a database machine crash. The machine that serves the BOINC database crashed last night. Until this is fixed we cannot send out work. 1 Jan 2010 16:27:45 UTC Gruß, Gundolf ID: 30407 ·

BarryAZ Send message Joined: 4 Sep 09 Posts: 381	Message 30410 - Posted: 1 Jan 2010, 21:09:16 UTC - in response to Message 30407. Correct -- as I noted previously. My own sense, given the long weekend, is that they simply won't be able to even look at the database problems until AFTER the power outage (24+ hours starting sometime on 1/3). That being the likely scenario, it may well be until after the regular Tuesday outage (4+ hours on 1/5) that SETI will be back online. And, given the very thin margin for I/O handling that SETI works with, and a likely extended (as in 5 days) outage, we may well be looking at perhaps sometime on Wednesday (perhaps late Wednesday) before things settle down. Seti seems to be having problems. The Master DB is disabled. The front page and anything pulling data from that DB is not working. Graphs show a falloff in data going in and out of the project. Meanwhile, the front page says: Projects are down due to a database machine crash. The machine that serves the BOINC database crashed last night. Until this is fixed we cannot send out work. 1 Jan 2010 16:27:45 UTC Gruß, Gundolf ID: 30410 ·

Aurora Borealis Send message Joined: 8 Jan 06 Posts: 448	Message 30416 - Posted: 2 Jan 2010, 8:42:12 UTC It looks like most of the Seti servers are up again. The boys must have spent a good part of their new years day working as things came up one piece at a time. I just hope no one actually had to trek out to the campus to kick them. Boinc V 7.4.36 Win7 i5 3.33G 4GB NVidia 470 ID: 30416 ·

Jord Volunteer tester Help desk expert Send message Joined: 29 Aug 05 Posts: 15542	Message 30427 - Posted: 3 Jan 2010, 13:38:58 UTC Warning: Tonight from 20.00 hours UTC the Berkeley Campus will have another 24 hour scheduled power outage. This will mean that Seti@Home, Seti Beta and all of the BOINC domain will be off line again during that time period. ID: 30427 ·

Gary Charpentier Send message Joined: 23 Feb 08 Posts: 2486	Message 30430 - Posted: 4 Jan 2010, 23:00:33 UTC - in response to Message 30427. Warning: Tonight from 20.00 hours UTC the Berkeley Campus will have another 24 hour scheduled power outage. This will mean that Seti@Home, Seti Beta and all of the BOINC domain will be off line again during that time period. and looks like SETI is staying down Scheduled Power Outage - Update We mostly recovered from our scheduled power outage this weekend, but due to minor problems we decided it would be best to keep the projects offline another day - however we're folding in the usual Tuesday outage today. So if all goes well we'll be fully up again by tomorrow (Tuesday) morning (if not sooner). 4 Jan 2010 22:19:26 UTC ID: 30430 ·

Gary Charpentier Send message Joined: 23 Feb 08 Posts: 2486	Message 30469 - Posted: 9 Jan 2010, 7:27:53 UTC Not that they had any work, but LHC seems to be unreachable. ID: 30469 ·

mo.v Send message Joined: 13 Aug 06 Posts: 778	Message 30494 - Posted: 11 Jan 2010, 23:14:34 UTC CPDN main project Many members are still processing HadAM3P models which are not currently available for download. Their trickles go to the scheduler and are uploading without problems. But there's an upload problem with one of the three final files. #1 file uploads to upload server uploader.oerc #2 file to cpdn-uploader1.comlab #3 file to uploader1.atm Uploader.oerc has filled up as can be seen on the server status page. #2 and #3 files should upload without problems but #1 file cannot. Please do not try to force this file to upload. Milo is moving data from the uploader.oerc server and hopes it will be able to accept uploads sometime on Tuesday. You could * do nothing and let Boinc attempt the upload at intervals (current attempts will fail) or * suspend HadAM3P models before completion so they do not produce the final files while the server is down or * suspend network activity in the Boinc Manager Activity menu ID: 30494 ·

Keith T Send message Joined: 26 Feb 07 Posts: 71	Message 30498 - Posted: 12 Jan 2010, 14:16:30 UTC SETI@home has had an unscheduled outage today. SETI Beta and SETI classic sites are also down, so is seti.berkeley.edu Keith ID: 30498 ·

Copyright © 2024 University of California.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.