Early this morning there was a cooling system failure at the data center on campus where we have most of our systems. Cooling has been restored, but failures during the event mean that all planned work for today is deferred as we asses damage and make any necessary fixes. If you noticed a failure of an ibiblio system that started between 3am and 5am EDT today, and it still has not been resolved, please reply here.
Hi, nomproject1 has an issue where / is mounted read only so none of the services will start, e.g. Apache. The first alert was seen at:
September 16, 2020 02:15:51 (GMT -04:00)
Thanks for the heads-up @bac. Above is how I found the host.
Filesystems still not RW/clean after one reboot. Above is me working on a reboot after a hard power cycle. That seemed to work fine, and now I’ve applied various updates, but I see you’ve got a shell again. Okay to reboot one more time?
We are back! Thank you.
Do whatever you need to do!
1 Like
Done. Whenever you see it back up, it’s all yours!
Back again. Thanks CMP.
I assume login is broken right now as ibiblio comes back up? I cannot connect to login.ibiblio.org, and get this message:
kex_exchange_identification: Connection closed by remote host
Connection closed by 2610:28:3090:3001:0:bad:cafe:136 port 22login.ibiblio.org still down?
Date/Time: 2020-09-16 15:26:13 +0000
1: Transmit 5.6.7 (x86_64) Session Transcript [Version 10.15.6 (Build 19G2021)] (16/09/2020, 17:26)
1: Connecting to login.ibiblio.org user:xxxxxx port:22
1: Failure Event: -43 - Failed getting banner
1: Canceling…
@freax @jhall login.ibiblio is still down. There are several dependencies I’m still working on. I’ll reply again shortly when I believe it’s ready for y’all.
@jhall @freax login.ibiblio.org should be all set now. Please let us know if not.
2 Likes
It works now - thanks!
Hi – not sure if this is related to today’s situation or not but I am unable to add images to my WordPress website (username vlalib) and WordPress is tossing me “Failed to write file to disk.” errors. I can add the images by hand via sftp but they are not showing up in the media viewer. Path is this…
https://www.vermontlibraries.org/wp-content/uploads/2020/06/
I did check to make sure that the permissions on the directory were okay. No rush on my end. I add content to this site infrequently, so it’s possible that this is an earlier issue. Thank you.
@Jessamyn This is related to the outage of Aug 27 (details at Ibiblio Outage Announcement). If you would like your site to be one of the early/test cases with restored read-write access to the filesystem (to allow file uploads via the website), please let us know. The current workaround is manual uploading using sftp - it may take some searching to find the correct path to put the files for WordPress to access them. (I will try to look into this when I can.)
Thank you – yes I’d like restored access when you can. Since I run a site with multiple users who all use the website and don’t have access to, or knowledge of, sftp, I can’t view that as a workable solution.
It’s funny, I am not having this issue with my other ibiblio site but possibly they are on different servers. Appreciate the speedy reply.
no problem logging in now, thanks! 
Thanks for volunteering. Let us know whether the wordpress management interface works now, please.
It does not, but a new error
"Screen Shot 2020-04-10 at 14.02.09 .jpg The uploaded file could not be moved to wp-content/uploads/2020/09. "
Let me know if you’d like an admin account so you can test it.
Could this be why www.metalab.unc.edu is down? Also, e-mail addressed to metalab.unc.edu isn’t going through; my ibiblio.org mail goes through. I know, I know, I’m one of the (annoying?) folks still using metalab.unc.edu for some of my incoming mail.
I think I’ve traced it to ns1.int.ibiblio.org no longer responding to DNS requests for metalab.unc.edu, and the secondaries, ns.unc.edu and ns2.unc.edu, not giving out the A and MX entries needed.
Thoughts?
I can assist via private message