RESOLVED: UNPLANNED: load balancer outage

We are having an issue with a primary load balancer. Hosted sites and are impacted. Will update here as we make progress.

UPDATE 17:10 Tuesday - primary is back up. We do not seem to have a proper secondary. We are going to leave the primary as-is and investigate. We are scheduling maintenance for tomorrow (Wednesday) morning to resolve and have a proper High Availability (HA) pair, but making any more changes before we investigate further seems unwise at this time.

This morning, Wednesday, at 9:30am local, we will be making a switch to a new secondary load balancer. This may involve an outage of up to one minute.

EDIT 9:37am: after a correction to a configuration error, the secondary load balancer is now handling all traffic that the primary was handling yesterday. This includes traffic for and collections in directories under that domain, as well as the majority of our hosted domains. If your collection is not responsive (times out, gives a 503 error, etc.) please reach out.

This morning we are deploying a replacement primary load balancer. Outages of 30 seconds to a minute are possible, but should be fully resolved by 10:30am local.