Hello,
My users have reported that for several hours etreedb.org has been down, mostly.
Is there a load-balanced server which has gone bad?
Please advise,
Tom H Anderson
From a phpinfo screen:
| System | Linux etreeweb1.lb.ibiblio.org 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC 2020 x86_64 |
|---|---|
| Build Date | Dec 1 2020 21:53:22 |
I was not informed of a planned upgrade of the server PHP version. The PHP version on login.ibiblio.org is 7.0.27. The etreedb.org tomb is 21 years old and not ready for a PHP upgrade. If this was a deliberate change please tell me. Updating the etreedb.org code base was not on my scheduled holiday events.
Looking forward to more information,
Tom
Hi @tom.h.anderson, no this was not a planned upgrade. This was unfortunately a rescue operation that attempted to make the smallest possible change in order to mitigate an ImageMagick issue, but that unfortunately did not work as planned across our clusters, with the result that you see: PHP went from 7.1.x to 7.3.x. For more background, we have a management system, puppet, that after a change I made left the etreeweb nodes in different wedged states with regard to package updates, resulting in the flapping you saw (one node was working but would have failed at the next reboot or log rotation). Since 7.1 was pulled from the repositories as an option, the only way with those existing nodes to quickly move forward so that they wouldn’t yield 503s to visitors was to make them match other known-working nodes, which meant 7.3.x. This was an emergency move, and you’re right that I should have posted here in support.ibiblio about it. I’m sorry I failed to do that last night.
First let me address login.ibiblio and then we’ll come back to etreeweb{1,2}.lb.
On login.ibiblio, you can use several different versions of PHP from your shell session. You’ve seen the default packages, but I for example have set 7.3 as my default:
[cmpalmer@login2 ~]$ which php
/opt/rh/rh-php73/root/usr/bin/php
[cmpalmer@login2 ~]$ php --version
PHP 7.3.20 (cli) (built: Dec 1 2020 12:22:19) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.20, Copyright (c) 1998-2018 Zend Technologies
with Zend OPcache v7.3.20, Copyright (c) 1999-2018, by Zend Technologies
with Xdebug v2.7.2, Copyright (c) 2002-2019, by Derick Rethans
[cmpalmer@login2 ~]$ cat .enable_scls
source scl_source enable rh-php73
source scl_source enable rh-mariadb101
[cmpalmer@login2 ~]$ grep enable .bashrc
. ~/.enable_scls
And you could use 7.2 if you preferred:
[cmpalmer@login2 ~]$ source scl_source enable rh-php72
[cmpalmer@login2 ~]$ php --version
PHP 7.2.24 (cli) (built: Nov 4 2019 10:23:08) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
with Zend OPcache v7.2.24, Copyright (c) 1999-2018, by Zend Technologies
But I would recommend 7.3, as 7.2 is also EOL. See here for EOL dates for PHP versions.
As for the etreeweb nodes, we could replace them with debian-based nodes (our current default for new nodes), which would give us the option of going as far back as PHP 7.2 with a fresh install since debian still has those (now EOL) packages in their repos, but we cannot go back to where you were (7.1) without two risks. First and foremost, 7.1 is EOL and has been for over a year, so we cannot know the sum total of the risks of running it. Second, and more immediately, the campus security group has already blocked an entire cluster earlier this year, and it is not entirely clear to us how much risk we can shoulder without them blocking your IPs at the campus border.
My strong preference would be to shoulder as much of the load in getting your sites fully functional on PHP 7.3 as possible, and subsequently working on mechanisms both on our end and yours to make future upgrades more comfortable and planned.
I should add that I’m happy to reach out directly so we’re not putting error messages and logs on the public support site. Mark has already reached out, and I’ll do my best to make sure you’re both included.