Thread 'News on Project Outages'

Message boards : Projects : News on Project Outages
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 46 · 47 · 48 · 49 · 50 · 51 · 52 . . . 67 · Next

AuthorMessage
Grumpy Swede
Avatar

Send message
Joined: 30 Mar 20
Posts: 408
Sweden
Message 111151 - Posted: 2 Mar 2023, 3:53:12 UTC
Last modified: 2 Mar 2023, 3:56:52 UTC

No further news from the WCG team. Still down, very very deep down......
Since it's almost 23:00 hours, in Toronto now, WCG is probably going to stay down, at least over the (Canadian) night.
ID: 111151 · Report as offensive     Reply Quote
Warped
Avatar

Send message
Joined: 25 Aug 08
Posts: 40
South Africa
Message 111152 - Posted: 2 Mar 2023, 13:49:16 UTC - in response to Message 111151.  

No further news from the WCG team. Still down, very very deep down......
Since it's almost 23:00 hours, in Toronto now, WCG is probably going to stay down, at least over the (Canadian) night.

The lack of news (and progress) is somewhat concerning.
Also, why would the forums be down at the same time?
I thought they were hosted separately.
ID: 111152 · Report as offensive     Reply Quote
Dr Who Fan
Avatar

Send message
Joined: 10 May 07
Posts: 1436
United States
Message 111155 - Posted: 2 Mar 2023, 20:33:28 UTC - in response to Message 111152.  

The lack of news (and progress) is somewhat concerning.
Also, why would the forums be down at the same time?
I thought they were hosted separately.

It was when IBM ran the show. Don't know how much physical hardware Krembil consolidated project on, but my guess is it's a lot less than IBM.

Being 30+ hours since last Facebook update/post from Krembil, who have not been good at communicating things since they took over 18 months ago... We don't have a clue what's happening.
ID: 111155 · Report as offensive     Reply Quote
Phillip Spencer

Send message
Joined: 3 Mar 23
Posts: 10
France
Message 111156 - Posted: 3 Mar 2023, 12:14:47 UTC - in response to Message 111151.  

To: Grumpy Swede
Thank you for posting updates on WCG here. I don't use Facebook or Twitter so, with the forums down as well, this thread has become my source of information.
Cheers
Phillip
ID: 111156 · Report as offensive     Reply Quote
Grumpy Swede
Avatar

Send message
Joined: 30 Mar 20
Posts: 408
Sweden
Message 111157 - Posted: 3 Mar 2023, 13:02:52 UTC - in response to Message 111156.  
Last modified: 3 Mar 2023, 13:22:58 UTC

To: Grumpy Swede
Thank you for posting updates on WCG here. I don't use Facebook or Twitter so, with the forums down as well, this thread has become my source of information.
Cheers
Phillip
No problems Phillip. I do not "use" Facebook or Twitter either (I have no accounts), but one can always read the postings on most public Facebook and Twitter accounts.
Sadly though, it has not been any more updates on the WCG FB account. They really need to improve their communication skills. This silence towards us volunteers,
are simply not acceptable.

Just a few words would make a big difference. Like "We're having big problems with the RAID array, and it will take about xx hours/days to rebuild the array", or
"The entire RAID array is toast, and we have to rebuild it from backups, and it will take about xx hours/days" (Let's hope their backups has been handled in a professional way.)

To have this kind of SPOF (Single Point Of Failure), in such a big system as WCG, is just ridiculously unprofessional. When/if, they recover from this, the first thing they need to do,
is hire a real IT-professional, to look through their entire setup, and identify other weaknesses, and other SPOF's.
ID: 111157 · Report as offensive     Reply Quote
Grumpy Swede
Avatar

Send message
Joined: 30 Mar 20
Posts: 408
Sweden
Message 111158 - Posted: 3 Mar 2023, 13:31:22 UTC
Last modified: 3 Mar 2023, 13:44:24 UTC

Just in, New Update:

"Update #2: The borrowed RAID card worked and the drive layout was recognized, so we have all data safe
(there is also a tape backup, but accessing that would be slower). Data center managed a full boot and we
expect we will resume operation later today."


Edit, added: the last part "we expect we will resume operation later today", is something I would take with more than one grain of salt
ID: 111158 · Report as offensive     Reply Quote
ProfileDave
Help desk expert

Send message
Joined: 28 Jun 10
Posts: 2672
United Kingdom
Message 111159 - Posted: 3 Mar 2023, 13:53:36 UTC

To have this kind of SPOF (Single Point Of Failure), in such a big system as WCG, is just ridiculously unprofessional. When/if, they recover from this, the first thing they need to do,
is hire a real IT-professional, to look through their entire setup, and identify other weaknesses, and other SPOF's.
I suspect the real issue is the amount of cash going in is a fraction of what there was when IBM had it. If they had more cash, I am sure their infrastructure would have more redundancy built in.
ID: 111159 · Report as offensive     Reply Quote
Grumpy Swede
Avatar

Send message
Joined: 30 Mar 20
Posts: 408
Sweden
Message 111164 - Posted: 4 Mar 2023, 0:13:15 UTC
Last modified: 4 Mar 2023, 0:14:00 UTC

New Update, about 4-hours ago:

"Update #3: All disks have finished rebuilding and the MySQL and DB2 filesystems are mounted.
We are currently checking the /science filesystem integrity."


WCG is still down though......
ID: 111164 · Report as offensive     Reply Quote
Grumpy Swede
Avatar

Send message
Joined: 30 Mar 20
Posts: 408
Sweden
Message 111165 - Posted: 4 Mar 2023, 1:10:55 UTC
Last modified: 4 Mar 2023, 1:14:01 UTC

New update, just in:

"Update #4: We have confirmed all the data is intact and have replaced the RAID controller,
but we are still having some issues with getting the new hardware production ready.
Unfortunately, data center staff will not be able to help us over the weekend.

Additionally, the deadline of all existing WUs that are partially done will be extended and accepted once the hardware change is done."


So, as I expected, WCG will likely continue to be down, until at least some time next week.
(I wouldn't be surprise though, if WCG was down for another week, or two)
ID: 111165 · Report as offensive     Reply Quote
Grumpy Swede
Avatar

Send message
Joined: 30 Mar 20
Posts: 408
Sweden
Message 111174 - Posted: 5 Mar 2023, 11:41:55 UTC
Last modified: 5 Mar 2023, 12:03:05 UTC

A slight difference now on the WCG home page. You no longer get the normal homepage, but instead directly the Errror
"System error World Community Grid is currently experiencing an unexpected error. Please check Facebook or Twitter for more information."

Also the forum ink (or other links needing access to a database) , does no longer show "Error 500: javax.servlet.ServletException: net.myvietnam.mvncore.exception.DatabaseException: Error executing SQL in MVNForumPermissionWebHelper.getPermissionsForGroupGuest."
The forum link now also shows: "System error World Community Grid is currently experiencing an unexpected error. Please check Facebook or Twitter for more information."

So, somebody is possibly up early in Toronto, trying to perhaps restart the system, and restore connections to the various databases, and filesystems.

Edit, added: No life from BOINC yet though. BOINC is still replying to a request with "Server error: feeder not running"
ID: 111174 · Report as offensive     Reply Quote
[CSF] Aleksey Belkov

Send message
Joined: 3 Mar 23
Posts: 14
Russia
Message 111177 - Posted: 5 Mar 2023, 16:26:00 UTC - in response to Message 111165.  

Thx for info ; )
ID: 111177 · Report as offensive     Reply Quote
Grumpy Swede
Avatar

Send message
Joined: 30 Mar 20
Posts: 408
Sweden
Message 111180 - Posted: 6 Mar 2023, 10:32:20 UTC

And it's finally Monday. Let's see if anything at all happens with WCG today.
It's only 05:33 in Toronto yet, so maybe something happens in a couple of hours.
ID: 111180 · Report as offensive     Reply Quote
Phillip Spencer

Send message
Joined: 3 Mar 23
Posts: 10
France
Message 111181 - Posted: 6 Mar 2023, 10:50:39 UTC - in response to Message 111180.  

And it's finally Monday. Let's see if anything at all happens with WCG today.
It's only 05:33 in Toronto yet, so maybe something happens in a couple of hours.

More in hope than certainty! Do you have pigs that fly in Sweden? I just saw several Porcine Flying Objects overhead in South West France and distinctly heard them squealing "(B)oink, (B)oink"
Fingers crossed!
Phillip
ID: 111181 · Report as offensive     Reply Quote
ProfileDave
Help desk expert

Send message
Joined: 28 Jun 10
Posts: 2672
United Kingdom
Message 111182 - Posted: 6 Mar 2023, 11:22:13 UTC - in response to Message 111181.  

More in hope than certainty! Do you have pigs that fly in Sweden? I just saw several Porcine Flying Objects overhead in South West France and distinctly heard them squealing "(B)oink, (B)oink"
Fingers crossed!
Phillip
I don't know the velocity of flying pigs, they may take a while to reach Toronto.
ID: 111182 · Report as offensive     Reply Quote
Grumpy Swede
Avatar

Send message
Joined: 30 Mar 20
Posts: 408
Sweden
Message 111183 - Posted: 6 Mar 2023, 12:23:11 UTC - in response to Message 111181.  
Last modified: 6 Mar 2023, 12:33:01 UTC


More in hope than certainty! Do you have pigs that fly in Sweden? I just saw several Porcine Flying Objects overhead in South West France and distinctly heard them squealing "(B)oink, (B)oink"
Fingers crossed!
Phillip
Well, we do have some flying pigs here in Sweden, most of them are politicians :-)
Yeah, fingers crossed for WCG returning before the next ice age.
ID: 111183 · Report as offensive     Reply Quote
robsmith
Volunteer tester
Help desk expert

Send message
Joined: 25 May 09
Posts: 1298
United Kingdom
Message 111184 - Posted: 6 Mar 2023, 12:33:28 UTC - in response to Message 111183.  

Only a matter of hours until the next ice age in the north of the UK......
ID: 111184 · Report as offensive     Reply Quote
jhseltzer

Send message
Joined: 8 Feb 17
Posts: 7
United States
Message 111186 - Posted: 6 Mar 2023, 13:44:34 UTC - in response to Message 111184.  

I'm very upset over the current state of affairs. I've been crunching files since the start of WCG. I'm now doing Rosetta files hoping WCG will come back soon. Glad to have found this forum since FB and the WCG forum are down.
ID: 111186 · Report as offensive     Reply Quote
Grumpy Swede
Avatar

Send message
Joined: 30 Mar 20
Posts: 408
Sweden
Message 111187 - Posted: 6 Mar 2023, 15:13:16 UTC
Last modified: 6 Mar 2023, 15:13:48 UTC

New Update from WCG on Facebook, a couple of minutes ago:

"Hello everyone, hope you had a great weekend. We are still working with data centre to resolve the hardware failure so we can
restart the storage, BOINC and website ASAP. We will post updates as we receive them. Thank you for your patience."
ID: 111187 · Report as offensive     Reply Quote
Dr Who Fan
Avatar

Send message
Joined: 10 May 07
Posts: 1436
United States
Message 111189 - Posted: 6 Mar 2023, 17:52:41 UTC - in response to Message 111187.  

... Another day wasted because Krembil's incompetence and running around chasing it's tail :(
ID: 111189 · Report as offensive     Reply Quote
Grumpy Swede
Avatar

Send message
Joined: 30 Mar 20
Posts: 408
Sweden
Message 111190 - Posted: 6 Mar 2023, 18:55:52 UTC

It's soon 14:00 hours in Toronto, and I seriously doubt that we'll see any working WCG today either.
Sigh......
ID: 111190 · Report as offensive     Reply Quote
Previous · 1 . . . 46 · 47 · 48 · 49 · 50 · 51 · 52 . . . 67 · Next

Message boards : Projects : News on Project Outages

Copyright © 2024 University of California.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.