Network change making IPv6 unreachable

An after-hours change made yesterday, Wednesday the 14th around 6p local time has left IPv6 resources at ibiblio unreachable beyond the UNC-CH campus. ibiblio staff are working with other UNC-CH staff to troubleshoot, identify the specific needed changes, and restore service. We apologize for the inconvenience and will provide updates as we are able.

Multiple staff members with domain expertise are reviewing networking configurations affecting ibiblio systems. While we do not have a time to resolution, we can confirm that most ibiblio services are functional. A subset of vhost and login services are impacted. We will update again when we are confident we have a resolution or if we can give a more detailed timeline to resolution. Once again, please understand that multiple professionals are working to restore service, and we regret the outage.

We believe services are restored. If we have missed something, please let us know. We are happy to have this issue resolved, and we share our regret that resolution took so long.

For the curious, the issue was a legacy IPv6 range in use on a subset of hosts. Moving those hosts to the current range required significant, careful work.

Again, please accept our regret for the extended outage.

C:\Users\kersh>curl -v --ipv6 www.ibiblio.org

  • Host www.ibiblio.org:80 was resolved.
  • IPv6: 2610:28:3090:3000:0:bad:cafe:23
  • IPv4: (none)
  • Trying [2610:28:3090:3000:0:bad:cafe:23]:80…
  • Established connection to www.ibiblio.org (2610:28:3090:3000:0:bad:cafe:23 port 80) from 2600:1700:f91:2940:5a:100a:ff94:399c port 49732
  • using HTTP/1.x

GET / HTTP/1.1
Host: www.ibiblio.org
User-Agent: curl/8.16.0
Accept: /

  • Request completely sent off
  • Recv failure: Connection was reset
  • closing connection #0
    curl: (56) Recv failure: Connection was reset

The WCOM podcast files stored are unreachable. I noted the failure from my home and found the issue is not isolated. I am able to reach the site on my CELL with a Safari browser, but using a laptop and Chrome, Edge, and DuckDuckGo browsers the connection is reset.

Virtual AGC Home Page is still unreachable here, if I change my network settings to not use ipv6 I can access the site, so it seems the ipv6 issue is still present.

We are actively making changes. Unfortunately because of our current architecture, the effects of some changes lag. We have taken steps to shorten the turnaround time between changes and live production status, but I do not yet have a specific timeline for resolution. As an added frustration, the severity of this outage for services running off of the primary www.ibiblio.org cluster may go up in the short term.

To provide a bit more detail, here’s what you see if you talk directly to UNC’s DNS infrastructure:

dig -t AAAA @ns.unc.edu +noall +answer www.ibiblio.org
www.ibiblio.org.        600     IN      AAAA    2606:f640:6000:321:0:bad:cafe:23

And here is what I see when I just use the native DNS for a non-UNC hosting provider:

dig -t AAAA +noall +answer www.ibiblio.org
www.ibiblio.org.        83452   IN      AAAA    2610:28:3090:3000:0:bad:cafe:23

Note that the non-UNC DNS has a TTL of almost a full calendar day (more than 23 hours) and the campus one has 600 seconds (ie. ten minutes). The campus value is the correct value, and when that propagates to the rest of the internet, which can take as much as a day, things should stabilize. If there is still a problem tomorrow, changes we make will propagate in minutes instead of hours.

Please stick with us. We’re working on it!

support.ibiblio (this site) was impacted and we have made a change to avoid this recurring.