Page MenuHomePhabricator

Migrate dbmonitor hosts to Stretch/Buster
Open, Stalled, NormalPublic

Description

The dbmonitor* hosts are currently running jessie:

  • dbmonitor1001.wikimedia.org
  • dbmonitor2001.wikimedia.org

Event Timeline

This should be easy unless apache changes are drastic like last time (as this are rather standard web frontend hosts).

Apache should be harmless, it's just different versions of Apache 2.4, but I vaguely remember an issue with something requiring PHP5. But I might be completely off track here, it's just a vague recollection.

ArielGlenn triaged this task as Normal priority.Jun 11 2019, 7:52 AM

Change 545246 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb/backups: Prepare dbmonitor[12]001 to reimage to buster

https://gerrit.wikimedia.org/r/545246

jcrespo claimed this task.Oct 22 2019, 11:48 AM

Change 545246 merged by Jcrespo:
[operations/puppet@production] mariadb/backups: Prepare dbmonitor[12]001 to reimage to buster

https://gerrit.wikimedia.org/r/545246

Mentioned in SAL (#wikimedia-operations) [2019-10-22T12:14:59Z] <jynus> reimage to buster dbmonitor2001.wikimedia.org T224589

Change 545273 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbmonitor: Install the right apache module packages for >jessie

https://gerrit.wikimedia.org/r/545273

Change 545273 merged by Jcrespo:
[operations/puppet@production] dbmonitor: Install the right apache module packages for >jessie

https://gerrit.wikimedia.org/r/545273

2 blockers:

  • Exec of /usr/sbin/a2enmod php7.0 fails, as ther right module would be php7.3- No support for buster on the http module? Httpd/Httpd::Mod_conf[php7.0]/Exec[ensure_present_mod_php7.0]
  • Error: '/usr/bin/git clone -b master https://gerrit.wikimedia.org/r/operations/software/tendril /srv/tendril' returned 1 instead of one of [0] /srv/tendril/.git: Permission denied, (why?)
Joe added a subscriber: Joe.Oct 22 2019, 1:38 PM

2 blockers:

  • Exec of /usr/sbin/a2enmod php7.0 fails, as ther right module would be php7.3- No support for buster on the http module? Httpd/Httpd::Mod_conf[php7.0]/Exec[ensure_present_mod_php7.0]

That would be the php module, or whatever else you're using, not the httpd module, that has no support for php7.3.

Change 545282 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbmonitor: Deploy git repo as mwdeploy, otherwise no write permission

https://gerrit.wikimedia.org/r/545282

Change 545286 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbmonitor: Install the right apache modules for buster

https://gerrit.wikimedia.org/r/545286

Thanks, joe, I didn't see your comment so it tool me more time than I thought to find it. The above 2 patches should fix it?

Change 545282 merged by Jcrespo:
[operations/puppet@production] dbmonitor: Deploy git repo as mwdeploy, otherwise no write permission

https://gerrit.wikimedia.org/r/545282

Change 545286 merged by Jcrespo:
[operations/puppet@production] dbmonitor: Install the right apache modules for buster

https://gerrit.wikimedia.org/r/545286

🤔

Notice: /Stage[main]/Httpd/Httpd::Conf[defaults]/File[/etc/apache2/conf-enabled/00-defaults.conf]/ensure: created
Info: /Stage[main]/Httpd/Httpd::Conf[defaults]/File[/etc/apache2/conf-enabled/00-defaults.conf]: Scheduling refresh of Service[apache2]                                                                                                         
Notice: /Stage[main]/Httpd/Httpd::Mod_conf[rewrite]/Exec[ensure_present_mod_rewrite]/returns: executed successfully
Info: /Stage[main]/Httpd/Httpd::Mod_conf[rewrite]/Exec[ensure_present_mod_rewrite]: Scheduling refresh of Service[apache2]                                                                                                                      
Notice: /Stage[main]/Httpd/Httpd::Mod_conf[headers]/Exec[ensure_present_mod_headers]/returns: executed successfully
Info: /Stage[main]/Httpd/Httpd::Mod_conf[headers]/Exec[ensure_present_mod_headers]: Scheduling refresh of Service[apache2]                                                                                                                      
Notice: /Stage[main]/Httpd/Httpd::Mod_conf[ssl]/Exec[ensure_present_mod_ssl]/returns: executed successfully
Info: /Stage[main]/Httpd/Httpd::Mod_conf[ssl]/Exec[ensure_present_mod_ssl]: Scheduling refresh of Service[apache2]
Notice: /Stage[main]/Httpd/Httpd::Mod_conf[php7.3]/Exec[ensure_present_mod_php7.3]/returns: ERROR: Module mpm_event is enabled - cannot proceed due to conflicts. It needs to be disabled first!
Notice: /Stage[main]/Httpd/Httpd::Mod_conf[php7.3]/Exec[ensure_present_mod_php7.3]/returns: ERROR: Could not enable dependency mpm_prefork for php7.3, aborting
Notice: /Stage[main]/Httpd/Httpd::Mod_conf[php7.3]/Exec[ensure_present_mod_php7.3]/returns: Considering dependency mpm_prefork for php7.3:
Notice: /Stage[main]/Httpd/Httpd::Mod_conf[php7.3]/Exec[ensure_present_mod_php7.3]/returns: Considering conflict mpm_event for mpm_prefork:
Notice: /Stage[main]/Httpd/Httpd::Mod_conf[php7.3]/Exec[ensure_present_mod_php7.3]/returns: Considering conflict mpm_worker for mpm_prefork:
Error: '/usr/sbin/a2enmod php7.3' returned 1 instead of one of [0]
Error: /Stage[main]/Httpd/Httpd::Mod_conf[php7.3]/Exec[ensure_present_mod_php7.3]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/a2enmod php7.3' returned 1 instead of one of [0]
Notice: /Stage[main]/Httpd/Httpd::Mod_conf[authnz_ldap]/Exec[ensure_present_mod_authnz_ldap]/returns: executed successfully

I think I know what happened: Initially the Puppet code was wrong and libapache-mod-php didn't get installed (which needs mpm_prefork). But "apache" still got installed which detects a new installation in the postinst in is_fresh_install() and then sets up mpm as the default mpm in the postinst. Now that Puppet is fixed, you can do a "dpkg --purge apache2 apache2-bin apache2-data apache2-utils libapache2-mod-php libapache2-mod-php7.3" and then re-running Puppet should set this up correctly.

That was more or less what I tried before, but it installs event version rather than prefork. Just to be sure, I tried your exact purges again, and I got the same error:

Notice: /Stage[main]/Httpd/Httpd::Mod_conf[php7.3]/Exec[ensure_present_mod_php7.3]/returns: ERROR: Module mpm_event is enabled - cannot proceed due to conflicts. It needs to be disabled first!
Notice: /Stage[main]/Httpd/Httpd::Mod_conf[php7.3]/Exec[ensure_present_mod_php7.3]/returns: ERROR: Could not enable dependency mpm_prefork for php7.3, aborting
Notice: /Stage[main]/Httpd/Httpd::Mod_conf[php7.3]/Exec[ensure_present_mod_php7.3]/returns: Considering dependency mpm_prefork for php7.3:
Notice: /Stage[main]/Httpd/Httpd::Mod_conf[php7.3]/Exec[ensure_present_mod_php7.3]/returns: Considering conflict mpm_event for mpm_prefork:
Notice: /Stage[main]/Httpd/Httpd::Mod_conf[php7.3]/Exec[ensure_present_mod_php7.3]/returns: Considering conflict mpm_worker for mpm_prefork:
Error: '/usr/sbin/a2enmod php7.3' returned 1 instead of one of [0]
Error: /Stage[main]/Httpd/Httpd::Mod_conf[php7.3]/Exec[ensure_present_mod_php7.3]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/a2enmod php7.3' returned 1 instead of one of [0]

My suspicion is that event or one of its used modules is chosen somewhere in the httpd class dependencies?

jcrespo added a comment.EditedOct 23 2019, 9:04 AM

I ran manually a2dismod mpm_event and now it worked. I will check if this happens again on a clean install of dbmonitor1001 and add code to handle it. You (we) still may be right, but the config maybe hidden on a -common package or something.

Now we "only" need to fix the php, with I would prefer not to, not because it would be difficult, but because it would be a waste of time, and I would prefer to create a simple flash + d3 microsite, specially for dbtree:

[Wed Oct 23 09:20:25.549861 2019] [php7:error] [pid 23096] [client 208.80.153.74:36964] PHP Fatal error:  Uncaught Error: Call to undefined function mysql_connect() in /srv/dbtree/index.php:31\nStack trace:\n#0 /srv/dbtree/inc/sanity.php(279): db()\n#1 /srv/dbtree/inc/sanity.php(951): sql->__construct('tendril.servers')\n#2 /srv/dbtree/inc/tree.php(88): sql::query('tendril.servers')\n#3 /srv/dbtree/index.php(41): Tree->generate()\n#4 {main}\n  thrown in /srv/dbtree/index.php on line 31

Change 545505 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/dbtree@master] mysql: Migrate away from long-deprecated mysql module to mysqli

https://gerrit.wikimedia.org/r/545505

Change 545505 merged by Jcrespo:
[operations/software/dbtree@master] mysql: Migrate away from long-deprecated mysql module to mysqli

https://gerrit.wikimedia.org/r/545505

Mentioned in SAL (#wikimedia-operations) [2019-10-23T10:11:03Z] <jynus> deploying new version of dbtree T224589

Mentioned in SAL (#wikimedia-operations) [2019-10-23T10:13:47Z] <jynus> reverting dbtree revision to HEAD~1 T224589

[Wed Oct 23 10:17:48.055752 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_select_db() expects exactly 2 parameters, 1 given in /srv/dbtree/index.php on line 33
[Wed Oct 23 10:17:48.055976 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_real_escape_string() expects exactly 2 parameters, 1 given in /srv/dbtree/inc/sanity.php on line 286
[Wed Oct 23 10:17:48.056122 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_query() expects parameter 1 to be mysqli, string given in /srv/dbtree/inc/sanity.php on line 783
[Wed Oct 23 10:17:48.056213 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_errno() expects exactly 1 parameter, 0 given in /srv/dbtree/inc/sanity.php on line 788
[Wed Oct 23 10:17:48.056297 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_error() expects exactly 1 parameter, 0 given in /srv/dbtree/inc/sanity.php on line 789
[Wed Oct 23 10:17:48.056377 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_fetch_array() expects parameter 1 to be mysqli_result, null given in /srv/dbtree/inc/sanity.php on line 852
[Wed Oct 23 10:17:48.060413 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_select_db() expects exactly 2 parameters, 1 given in /srv/dbtree/index.php on line 33
[Wed Oct 23 10:17:48.060750 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_real_escape_string() expects exactly 2 parameters, 1 given in /srv/dbtree/inc/sanity.php on line 286
[Wed Oct 23 10:17:48.060941 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_query() expects parameter 1 to be mysqli, string given in /srv/dbtree/inc/sanity.php on line 783
[Wed Oct 23 10:17:48.061083 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_errno() expects exactly 1 parameter, 0 given in /srv/dbtree/inc/sanity.php on line 788
[Wed Oct 23 10:17:48.061236 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_error() expects exactly 1 parameter, 0 given in /srv/dbtree/inc/sanity.php on line 789
[Wed Oct 23 10:17:48.061352 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_fetch_array() expects parameter 1 to be mysqli_result, null given in /srv/dbtree/inc/sanity.php on line 852
[Wed Oct 23 10:17:48.062785 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_select_db() expects exactly 2 parameters, 1 given in /srv/dbtree/index.php on line 33
[Wed Oct 23 10:17:48.062926 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_real_escape_string() expects exactly 2 parameters, 1 given in /srv/dbtree/inc/sanity.php on line 286
[Wed Oct 23 10:17:48.063107 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_query() expects parameter 1 to be mysqli, string given in /srv/dbtree/inc/sanity.php on line 783
[Wed Oct 23 10:17:48.063231 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_errno() expects exactly 1 parameter, 0 given in /srv/dbtree/inc/sanity.php on line 788
[Wed Oct 23 10:17:48.063341 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_error() expects exactly 1 parameter, 0 given in /srv/dbtree/inc/sanity.php on line 789
[Wed Oct 23 10:17:48.063453 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_fetch_array() expects parameter 1 to be mysqli_result, null given in /srv/dbtree/inc/sanity.php on line 852
[Wed Oct 23 10:17:48.064791 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_select_db() expects exactly 2 parameters, 1 given in /srv/dbtree/index.php on line 33
[Wed Oct 23 10:17:48.064942 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_real_escape_string() expects exactly 2 parameters, 1 given in /srv/dbtree/inc/sanity.php on line 286
[Wed Oct 23 10:17:48.065104 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_real_escape_string() expects exactly 2 parameters, 1 given in /srv/dbtree/inc/sanity.php on line 286
[Wed Oct 23 10:17:48.065229 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_real_escape_string() expects exactly 2 parameters, 1 given in /srv/dbtree/inc/sanity.php on line 286
[Wed Oct 23 10:17:48.065380 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_query() expects parameter 1 to be mysqli, string given in /srv/dbtree/inc/sanity.php on line 783
[Wed Oct 23 10:17:48.065489 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_errno() expects exactly 1 parameter, 0 given in /srv/dbtree/inc/sanity.php on line 788
[Wed Oct 23 10:17:48.065596 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_error() expects exactly 1 parameter, 0 given in /srv/dbtree/inc/sanity.php on line 789
[Wed Oct 23 10:17:48.065707 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_fetch_array() expects parameter 1 to be mysqli_result, null given in /srv/dbtree/inc/sanity.php on line 852
[Wed Oct 23 10:17:48.067187 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_select_db() expects exactly 2 parameters, 1 given in /srv/dbtree/index.php on line 33
[Wed Oct 23 10:17:48.067320 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_real_escape_string() expects exactly 2 parameters, 1 given in /srv/dbtree/inc/sanity.php on line 286
[Wed Oct 23 10:17:48.067507 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_query() expects parameter 1 to be mysqli, string given in /srv/dbtree/inc/sanity.php on line 783
[Wed Oct 23 10:17:48.067633 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_errno() expects exactly 1 parameter, 0 given in /srv/dbtree/inc/sanity.php on line 788
[Wed Oct 23 10:17:48.067751 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_error() expects exactly 1 parameter, 0 given in /srv/dbtree/inc/sanity.php on line 789
[Wed Oct 23 10:17:48.067896 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_fetch_array() expects parameter 1 to be mysqli_result, null given in /srv/dbtree/inc/sanity.php on line 852
[Wed Oct 23 10:17:48.070537 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_select_db() expects exactly 2 parameters, 1 given in /srv/dbtree/index.php on line 33
[Wed Oct 23 10:17:48.070809 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_query() expects parameter 1 to be mysqli, string given in /srv/dbtree/inc/sanity.php on line 783
[Wed Oct 23 10:17:48.070914 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_errno() expects exactly 1 parameter, 0 given in /srv/dbtree/inc/sanity.php on line 788
[Wed Oct 23 10:17:48.071025 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_error() expects exactly 1 parameter, 0 given in /srv/dbtree/inc/sanity.php on line 789
[Wed Oct 23 10:17:48.071139 2019] [:error] [pid 10017] [client 10.64.32.67:64543] PHP Warning:  mysqli_fetch_array() expects parameter 1 to be mysqli_result, null given in /srv/dbtree/inc/sanity.php on line 852

So normally the fix for the above would be trivial, but the design decisions of making sql class a singleton are in my opinion not worthy fixing, because it would force to either a deeper refactoring or a global scope hack. I would prefer to spend more time to refactor tendril into not using the Google API: T96499

jcrespo changed the task status from Open to Stalled.Tue, Nov 12, 9:39 AM
jcrespo removed jcrespo as the assignee of this task.