Page MenuHomePhabricator

Global User Contributions complains about replica conf file
Closed, DuplicatePublic

Description

Warning: parse_ini_file(/replica.my.cnf): failed to open stream: No such file or directory in /data/project/guc/labs-tools-guc/src/Settings.php on line 35

See https://tools.wmflabs.org/guc/?user=83.220.237.95 as example for this.

Event Timeline

Wiki13 created this task.Mar 6 2018, 1:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 6 2018, 1:46 PM

By the time this issue started, I was doing package upgrades in toolforge (T188994), so this could be related to some package upgrade.

According to @Chicocvenancio, this is running at tools-worker-1020.tools.eqiad.wmflabs, which was part of the upgrade:

[13:48 UTC] tools.guc at tools-bastion-03.tools.eqiad.wmflabs in ~
$ kubectl get pod -o wide
NAME                   READY     STATUS    RESTARTS   AGE       IP              NODE
guc-3456753348-bjdt9   1/1       Running   0          31d       192.168.206.7   tools-worker-1020.tools.eqiad.wmflabs
[13:49 UTC] tools.guc at tools-bastion-03.tools.eqiad.wmflabs in ~

The involved packages:

aborrero@tools-worker-1020:~$ tail /var/log/apt/history.log | grep Upgrade | sed s/"Upgrade: "//g | sed s/"), "/")"'\n'/g | sort 
base-files:amd64 (8+deb8u5, 8+deb8u10)
bash:amd64 (4.3-11+b1, 4.3-11+deb8u1)
binutils:amd64 (2.25-5, 2.25-5+deb8u1)
ca-certificates:amd64 (20141019+deb8u1, 20141019+deb8u3)
dbus:amd64 (1.8.20-0+deb8u1, 1.8.22-0+deb8u1)
debconf:amd64 (1.5.56, 1.5.56+deb8u1)
debconf-i18n:amd64 (1.5.56, 1.5.56+deb8u1)
debian-archive-keyring:amd64 (2014.3, 2017.5~deb8u1)
e2fslibs:amd64 (1.42.12-1.1, 1.42.12-2+b1)
e2fsprogs:amd64 (1.42.12-1.1, 1.42.12-2+b1)
file:amd64 (5.22+15-2+deb8u1, 5.22+15-2+deb8u3)
gnupg2:amd64 (2.0.26-6, 2.0.26-6+deb8u1)
gnupg-agent:amd64 (2.0.26-6, 2.0.26-6+deb8u1)
initramfs-tools:amd64 (0.120+deb8u2, 0.120+deb8u3)
jq:amd64 (1.4-2.1, 1.4-2.1+deb8u1)
krb5-locales:amd64 (1.12.1+dfsg-19+deb8u2, 1.12.1+dfsg-19+deb8u4)
libcairo2:amd64 (1.14.0-2.1+deb8u1, 1.14.0-2.1+deb8u2)
libcairo-gobject2:amd64 (1.14.0-2.1+deb8u1, 1.14.0-2.1+deb8u2)
libc-ares2:amd64 (1.10.0-2+deb8u1, 1.10.0-2+deb8u2)
libcomerr2:amd64 (1.42.12-1.1, 1.42.12-2+b1)
libcups2:amd64 (1.7.5-11+deb8u1, 1.7.5-11+deb8u2)
libdb5.3:amd64 (5.3.28-9, 5.3.28-9+deb8u1)
libdbus-1-3:amd64 (1.8.20-0+deb8u1, 1.8.22-0+deb8u1)
libgnutls-deb0-28:amd64 (3.3.8-6+deb8u6, 3.3.8-6+deb8u7)
libgnutls-openssl27:amd64 (3.3.8-6+deb8u6, 3.3.8-6+deb8u7)
libgssapi-krb5-2:amd64 (1.12.1+dfsg-19+deb8u2, 1.12.1+dfsg-19+deb8u4)
libgtk2.0-0:amd64 (2.24.25-3+deb8u1, 2.24.25-3+deb8u2)
libgtk2.0-common:amd64 (2.24.25-3+deb8u1, 2.24.25-3+deb8u2)
libhogweed2:amd64 (2.7.1-5+deb8u1, 2.7.1-5+deb8u2)
libicu52:amd64 (52.1-8+deb8u5, 52.1-8+deb8u6)
libk5crypto3:amd64 (1.12.1+dfsg-19+deb8u2, 1.12.1+dfsg-19+deb8u4)
libkrb5-3:amd64 (1.12.1+dfsg-19+deb8u2, 1.12.1+dfsg-19+deb8u4)
libkrb5support0:amd64 (1.12.1+dfsg-19+deb8u2, 1.12.1+dfsg-19+deb8u4)
libltdl7:amd64 (2.4.2-1.11, 2.4.2-1.11+b1)
libmagic1:amd64 (5.22+15-2+deb8u1, 5.22+15-2+deb8u3)
libncurses5:amd64 (5.9+20140913-1+b1, 5.9+20140913-1+deb8u2)
libncursesw5:amd64 (5.9+20140913-1+b1, 5.9+20140913-1+deb8u2)
libnettle4:amd64 (2.7.1-5+deb8u1, 2.7.1-5+deb8u2)
libnss-ldapd:amd64 (0.9.4-3+deb8u1, 0.9.4-3+deb8u2)
libpng12-0:amd64 (1.2.50-2+deb8u2, 1.2.50-2+deb8u3)
libpython2.7:amd64 (2.7.9-2, 2.7.9-2+deb8u1)
libpython2.7-minimal:amd64 (2.7.9-2, 2.7.9-2+deb8u1)
libpython2.7-stdlib:amd64 (2.7.9-2, 2.7.9-2+deb8u1)
libruby2.1:amd64 (2.1.5-2+deb8u2, 2.1.5-2+deb8u3)
libsqlite3-0:amd64 (3.8.7.1-1+deb8u1, 3.8.7.1-1+deb8u2)
libss2:amd64 (1.42.12-1.1, 1.42.12-2+b1)
libsystemd0:amd64 (215-17+deb8u4, 215-17+deb8u7)
libtinfo5:amd64 (5.9+20140913-1+b1, 5.9+20140913-1+deb8u2)
libudev1:amd64 (215-17+deb8u4, 215-17+deb8u7)
libx11-6:amd64 (1.6.2-3, 1.6.2-3+deb8u1)
libx11-data:amd64 (1.6.2-3, 1.6.2-3+deb8u1)
libxfixes3:amd64 (5.0.1-2+b2, 5.0.1-2+deb8u1)
libxi6:amd64 (1.7.4-1+b2, 1.7.4-1+deb8u1)
libxrandr2:amd64 (1.4.2-1+b1, 1.4.2-1+deb8u1)
libxslt1.1:amd64 (1.1.28-2+deb8u2, 1.1.28-2+deb8u3)
ncurses-base:amd64 (5.9+20140913-1, 5.9+20140913-1+deb8u2)
ncurses-bin:amd64 (5.9+20140913-1+b1, 5.9+20140913-1+deb8u2)
ncurses-term:amd64 (5.9+20140913-1, 5.9+20140913-1+deb8u2)
openssh-client:amd64 (6.7p1-5+deb8u3, 6.7p1-5+deb8u4)
openssh-server:amd64 (6.7p1-5+deb8u3, 6.7p1-5+deb8u4)
openssh-sftp-server:amd64 (6.7p1-5+deb8u3, 6.7p1-5+deb8u4)
python2.7:amd64 (2.7.9-2, 2.7.9-2+deb8u1)
python2.7-minimal:amd64 (2.7.9-2, 2.7.9-2+deb8u1)
python-crypto:amd64 (2.6.1-5+b2, 2.6.1-5+deb8u1)
ruby2.1:amd64 (2.1.5-2+deb8u2, 2.1.5-2+deb8u3)
sed:amd64 (4.2.2-4+b1, 4.2.2-4+deb8u1)
sudo-ldap:amd64 (1.8.10p3-1+deb8u4, 1.8.10p3-1+deb8u5)
systemd:amd64 (215-17+deb8u4, 215-17+deb8u7)
systemd-sysv:amd64 (215-17+deb8u4, 215-17+deb8u7)
udev:amd64 (215-17+deb8u4, 215-17+deb8u7)
vim:amd64 (7.4.488-7+deb8u2, 7.4.488-7+deb8u3)
vim-common:amd64 (7.4.488-7+deb8u2, 7.4.488-7+deb8u3)
vim-runtime:amd64 (7.4.488-7+deb8u2, 7.4.488-7+deb8u3)
vim-tiny:amd64 (7.4.488-7+deb8u2, 7.4.488-7+deb8u3)
w3m:amd64 (0.5.3-19, 0.5.3-19+deb8u2)
chasemp added a subscriber: chasemp.Mar 6 2018, 1:59 PM

Warning: parse_ini_file(/replica.my.cnf)

I can't verify the working path in the context of the operations here but that's an especially bad relative path to use I expect. './replica.my.cnf' would at least be better.

However

[13:58 UTC] tools.guc at tools-bastion-03.tools.eqiad.wmflabs in ~
1 $ file /data/project/guc/replica.my.cnf
/data/project/guc/replica.my.cnf: ASCII text
chasemp triaged this task as Normal priority.Mar 6 2018, 2:00 PM

There are no pending puppet changes right now in this host:

aborrero@tools-worker-1020:~$ sudo puppet agent -t -v
Info: Using configured environment 'future'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for tools-worker-1020.tools.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1520344645'
Notice: Applied catalog in 9.93 seconds
Jeff_G added a subscriber: Jeff_G.Mar 6 2018, 2:08 PM

I am getting "Error: MySQL login data not found at", in addition to the originally reported warning.

revi added a subscriber: revi.Mar 6 2018, 2:20 PM

https://tools.wmflabs.org/replag/

(and #wikimedia-operations discussion seems to indicate it's related to wikireplica incident which is ongoing as of now.)

Possible offending packages:

  • libnss-ldapd
  • sudo-ldap

Warning: parse_ini_file(/replica.my.cnf)

I can't verify the working path in the context of the operations here but that's an especially bad relative path to use I expect. './replica.my.cnf' would at least be better.
However

[13:58 UTC] tools.guc at tools-bastion-03.tools.eqiad.wmflabs in ~
1 $ file /data/project/guc/replica.my.cnf
/data/project/guc/replica.my.cnf: ASCII text

It seems the tool does not use a relative path, but gets the home dir for the tool user and appends /replica.my.cnf to it. Somehow posix_getpwuid(posix_geteuid()) failed and left the path as /replica.my.cnf.

Warning: parse_ini_file(/replica.my.cnf)

I can't verify the working path in the context of the operations here but that's an especially bad relative path to use I expect. './replica.my.cnf' would at least be better.
However

[13:58 UTC] tools.guc at tools-bastion-03.tools.eqiad.wmflabs in ~
1 $ file /data/project/guc/replica.my.cnf
/data/project/guc/replica.my.cnf: ASCII text

It seems the tool does not use a relative path, but gets the home dir for the tool user and appends /replica.my.cnf to it. Somehow posix_getpwuid(posix_geteuid()) failed and left the path as /replica.my.cnf.

ack, thanks :)