From about 4:15p Eastern to about 5:21p Eastern yesterday, Thursday the 19th, the Apache httpd children on the vhosts_rw backend were in a state that caused them to be unresponsive, but the parent did not cause them to exit, and existing monitoring did not catch that Apache was not functioning properly. This meant that a small number of highly visible vhosts were showing ‘503 Service Unavailable’ to clients for that time period. A simple restart of the Apache service resolved the immediate issue.
We have implemented two changes in order to avoid a repeat of this extended unplanned outage.
-
We have changed the Apache MaxRequestsPerChild value to cycle through children faster in an effort to avoid hitting the bug that caused this again, and we will monitor how this changes service over the next week.
-
We have deployed an application level canary that will cause a service restart if it is unavailable for longer than two minutes.
Apologies for the outage. It is our hope that the fixes we deployed, the manner in which we deployed them, and our ability to communicate these changes to you publicly will increase your confidence in management of vhosts infrastructure moving forward.