And it seems the working day in Oxford still hasn't finished. Forums and server status not accessible and project servers may be down message in event log.

Andy tells me the work is still ongoing and they expect things to be restored sometime later today.

However:

Hi Dave,

We've been told that this work is still ongoing unfortunately and is going to be continued to be worked on into today.

Best regards,

Andy


This about 3/4 hour ago.

CPDN website is back up but getting

Fri 06 Aug 2021 17:21:54 BST | climateprediction.net | Project is temporarily shut down for maintenance

On project update. Unless someone is doing overtime, it will be Monday before everything stands a chance of returning to normal.

On project update. Unless someone is doing overtime, it will be Monday before everything stands a chance of returning to normal.

1 hour later, 9 completed tasks uploaded and 8 new ones now downloading.

Clearly someone is doing overtime.

Two tasks reported, four tasks allocated, four sets of downloads failed.

You'd have thought they knew about that one by now :-(

Got this from Andy.

Hi Dave,

Thanks. Engineering IT Support have partially restored networking to a number of machines, but a number of key machines still have no networking access following the switch work on Tuesday. I have submitted a ticket to them for the other machines and I will follow this up on Monday with them.

Best regards,

Andy

I'm not very impressed by the Oxford University Engineering IT Support team. They will have scheduled this work for the summer vacation, when the undergraduate demand is low: but university postgrad and faculty research continues 52 weeks of the year. This is also a very busy time of year for university administration, dealing with applications from next year's intake of new students.

Letting a planned infrastructure upgrade over-run by a week is bad management, to say the least.

I'm not very impressed by the Oxford University Engineering IT Support team. They will have scheduled this work for the summer vacation, when the undergraduate demand is low: but university postgrad and faculty research continues 52 weeks of the year. This is also a very busy time of year for university administration, dealing with applications from next year's intake of new students.

Letting a planned infrastructure upgrade over-run by a week is bad management, to say the least.

And then not working the weekend to clear the problem - I know I’d never have got away with that.

And then not working the weekend to clear the problem - I know I’d never have got away with that.

Sadly, when I worked in the NHS they were as bad or worse about sorting out problems after, "upgrades." However having had a normal work day to sort things out and no signs of progress I am beginning to despair of them.

And then not working the weekend to clear the problem - I know I’d never have got away with that.

Sadly, when I worked in the NHS they were as bad or worse about sorting out problems after, "upgrades." However having had a normal work day to sort things out and no signs of progress I am beginning to despair of them.

When I was supporting system upgrades you worked until the system worked - either fix forward or pull the upgrade and fall back to the starting position. You did not break the system then go home.

So, does anyone know whether the CPDN download servers have been re-connected to the internet yet? I'm on Andy Bowery's email distribution list, and I haven't seen anything yet - and I've completed upgrading my machines to Linux Mint v20.2

Memo to project staff: the project shouldn't be restarted after maintenance until all components are tested and working.

It doesn't appear to be.
And Andy is probably "in a mood" by now, so I'm staying well away from it.

If Oxford IT hired external workers to do this, the air in the place has probably turned blue by now. :)

I am enabling internet access if I have an upload or two ready to go. One is almost uploading at the moment. I am going to suspend it again when it has finished as no movement on the downloads.

If it were possible to just suspend uploads or downloads I could leave internet access on and just check once a day to see whether the download server problem was fixed.

CPDN needs to be aware that BOINC is designed to manage multiple projects in parallel, and that many of us use it that way. There was once a proposal by, I think, user 'Thyme Lawn' to allow/suspend transfers by project: he coded it for precisely this scenario, but it was rejected by the gatekeepers.

For that reason, I can't follow your example: all my recent tasks have declared their download errors to be permanent and have reported their task status as 'download failed'. I've set 'no new tasks' until I receive positive confirmation that the network is operating properly again.

CPDN needs to be aware that BOINC is designed to manage multiple projects in parallel, and that many of us use it that way. There was once a proposal by, I think, user 'Thyme Lawn' to allow/suspend transfers by project: he coded it for precisely this scenario, but it was rejected by the gatekeepers.

For that reason, I can't follow your example: all my recent tasks have declared their download errors to be permanent and have reported their task status as 'download failed'. I've set 'no new tasks' until I receive positive confirmation that the network is operating properly again.

MY downloads are now shifting - two 10MB atmos.gz files have downloaded. The slow speed is I think my bored band rather than the servers getting hammered though I guess that is probably happening as well.

Edit: the trickle server isn't running again yet though.

Edit2:Well the server status page says that at least. I will know in about ten minutes whether trickles are going through as well. One task has finished downloading so that side seems to have been fixed.

Edit3: Does suspending the project stop the uploads/downloads?

Trickle server still showing as down after last update to server status page.

Edit3: Does suspending the project stop the uploads/downloads?

I think not.

I turned my net access back on a few hours ago, and the four that I had from before downloaded while I was sleeping.

Edit3: Does suspending the project stop the uploads/downloads?

I think not.

A shame. that would be a simple solution.

At 15:24 UTC on 12 Aug 2021 Andy Bowery wrote:

All services have been restored now to climateprediction.net infrastructure. The Department of Engineering IT Support decided to roll back the changes they made to the networking. This has allowed us to restore all the CPDN services.

Edit: Yes, I can confirm that all files for new tasks are being downloaded cleanly.