Ibiblio Outage Announcement

Hi All , ibiblio is currently experiencing a system-wide problem with our v-host sites. We are working on it and will update when it is remedied. Thanks for your patience.

Hi All - the outage has been resolved!

1 Like

Thanks a lot! The connection issues are now all gone.

Hi, the forum on our account (www.oesf.org) cannot be reached since ~1 hr, pinging it also gives no result, it appears as the Apache server is offline. Can somebody please confirm this?

Thanks and regards

Hello, I’m unable to access http://www.guidetojapanese.org today. Is this due to this issue?

Hi ibiblio.org, my sites also seem be be down. User is sauvd. One site is levee67.org.

Thank you in advance.

Damon

Our forum is also again down (www.oesf.org).

Thanks and regards
Daniel

Our site is also down:
http://norfolksouthernhs.org/

Hi all. This seems to be a continuation of the previous outage, meaning that our assessment of the level of re-deployment that would solve the issue in the very short term was not sufficient. While this is frustrating both for us and for you, our analysis of the root cause and the level of change necessary within the time frame of a week or sooner was correct. It was simply that the “week” part was too long a timescale.

Unfortunately for ibiblio, some of that change involves request(s) to campus provider(s) who will not respond to our tickets on a Sunday. We will do our best to have our vhosts fully functional as soon as possible. Unfortunately, that may mean business hours Eastern Time on Monday the 31st of August. As I write this, that’s just under 13 hours from now that would be the earliest our tickets would be seen. We will keep you posted here in this thread.

5 Likes

We are having to make storage changes that affect both /public/vhost and /public/html, and to be safe our changes involve finishing a filesystem scan. We hope to have another update in a few hours.

When sites using our vhost.ibiblio.org infrastructure are available again, it will be with filesystems set to read-only for the backend web nodes. This will break functionality like file uploads for many of your sites. We’re prioritizing bringing the infrastructure back up in a way that gives some public access over having a possibly multi-day ongoing downtime before making anything public again.

We have put the ball back in the court of a campus partner, and will send an update again when we either need to take more action on the ibiblio side or when we believe things should be back up.

2 Likes

We should be back up. If your vhost for your collection is still completely down, we may need to deal with that separately and in a private ticket. As previously mentioned, many of you will notice partial breakage since the web servers cannot write to disk. For example, if you are running a CMS that allowed you to upload photos or user avatars that were stored in a directory under your document root instead of in the database, that will be broken.

Hi @Tae_Kim. Yes, this was your issue. Hopefully your site works now. If not, please check for PHP changes that happened between 5.6 and 7.3 and make the relevant changes. If you need help, you’re welcome to start a topic in the Help Category.

Thanks for all your hard work!

Hi. My site www.huevaluechroma.com is responding again but all pages are broken. I am also getting an access denied response when I attempt to login to login.ibiblio.org.

Hi @djcbriggs! Thanks for joining here and reaching out. For the login question, I’m going to open a ticket in our private ticketing system for you, but for the benefit of others in this thread, it would be good if folks were aware of this separate but related change affecting the login host. For the broken pages part, unfortunately part of restoring services required an abrupt jump in PHP version to 7.3. Given that the previous version in use across our vhost cluster was not even 7.0, It’s not surprising that you are having some issues. This page shows the breaking changes starting in php 7 compared to previous releases. If you have specific questions about changes, I suggest you start a new topic in the help category on this site.

Just to clarify: Is this restriction (filesystems set to read-only) just temporary, until you get the site fully back online. Or will this be a permanent change?

@rf-confluence While this is intended to be temporary, we do not have a clear time horizon for making vhost.ibiblio read-write again. It may be that some collections get moved to a separate set of web hosts that are read-write, or some other significant change that will involve active steps taken by contributors to make their collections work properly again. We are still in a phase of assessing and minimizing damage. We are balancing that work with work to ensure that as many collections as possible are available to the public, and maintainers of those collections are able to manage their collections.

In the short to medium term, please plan on using CLI alternatives to processes that would normally happen via the web interface of your collections. For example, if a collection is managed using WordPress, previously the maintainer may have used the built-in media management for adding banners and other images. For now, such a maintainer would need to use sftp, scp, or rsync to upload their media to their shell account at login.ibiblio.org. We understand this presents a significant challenge to collections that allow for distributed/delegated contributions by the public, because you may not share ssh credentials.

Confluence is an example of a collection with custom software that may require interventions that don’t generalize well for other vhosts, so it may be worth opening a separate topic in the Help category about this. Thanks for writing here and asking a very good question; let’s continue the conversation in a way that helps get your collection available to the public again soon.

The “confluence” project needs HTTP POST support (to store some user-supplied data), so the data for this collection will need to be moved to a web host that supports this. Please let us know ASAP when this becomes possible.

In the meantime, the PHP version upgrade seems to have broken our site. I can probably fix this, but I’ll first need to be able to log in to “login.ibiblio.org” once again. See my separate posting (on the “Breaking change: login.ibiblio.org SSH key fingerprint changed” thread) about why I currently can’t do this.

Making three separate changes (login.ibiblio.org key fingerprint; PHP version; lack of read-write web site access) all at once was extremely disruptive. In the future, please consider trying to stagger these changes, and giving us plenty of warning.

Yes. If this had been a planned set of changes, we would have made each of these changes separately, announced them widely, and communicated in advance how to test under the new system. We had begun preparations for all of these changes, but had not set dates for them. Unfortunately, breakage happened that forced our hand. These changes were applied in order to restore service, not because we felt like it was a good time.

Thank you for working with our staff and student workers to restore your collection. We look forward to facilitating that, including by communicating about upgrades and changes in advance, and by structuring them in a phased, planned fashion where possible in the future.

1 Like