I've found the same issue while trying to migrate jobs from the trusty grid to the new one (or better deleting the tool at all) :/

Tools like qstat-full also missed.

Is there a work-around?

Workarounds:

• become mytoolname: sudo -niu tools.mytoolname
• crontab -l: ssh cat /etc/toollabs-cronhost` crontab -l`

In T218494#5030165, @Kanashimi wrote:

Tools like qstat-full also missed.

This seems a different issue (a tool missing on Stretch while it was available on Trusty), so I created a separate task for this: T218504

I forced the missing misctools package back with a manual puppet run.

$ sudo -i puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for tools-bastion-03.tools.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1552861777'
Notice: /Stage[main]/Toollabs::Exec_environ/Package[mariadb-client]/ensure: created
Notice: /Stage[main]/Toollabs::Exec_environ/Package[misctools]/ensure: created
Notice: Applied catalog in 61.30 seconds

It looks like this next automated puppet run removed it again. Another manual run restored it:

$ sudo -i puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for tools-bastion-03.tools.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1552862910'
Notice: /Stage[main]/Toollabs::Exec_environ/Package[mariadb-client]/ensure: created
Notice: /Stage[main]/Toollabs::Exec_environ/Package[misctools]/ensure: created
Notice: Applied catalog in 70.90 seconds

My current guess is that we have a non-deterministic state defined in Puppet such that some package we are installing is forcing removal of the mariadb-client package which then in turn removes the misctools package. Seems like this might be related to T218009: Puppet failure emails sent to non-admin members of tools project causing user confusion but that is not confirmed yet.

I caught Puppet causing the removal, and yes this is from T218009:

$ sudo -i puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for tools-bastion-03.tools.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1552863224'
Notice: /Stage[main]/Openstack::Clientpackages::Mitaka::Trusty/Package[mysql-client-5.5]/ensure: created
Notice: Applied catalog in 66.48 seconds

Change 497210 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] cloud-vps: Remove mysql packages from openstack::clientpackages::mitaka::*

https://gerrit.wikimedia.org/r/497210

Mentioned in SAL (#wikimedia-cloud) [2019-03-17T23:41:53Z] <bd808> Cherry-picked https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/497210/ as a quick fix for T218494

In T218494#5031052, @bd808 wrote:

I caught Puppet causing the removal, and yes this is from T218009:

$ sudo -i puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for tools-bastion-03.tools.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1552863224'
Notice: /Stage[main]/Openstack::Clientpackages::Mitaka::Trusty/Package[mysql-client-5.5]/ensure: created
Notice: Applied catalog in 66.48 seconds

Confirmed that in Trusty this happens because:

aborrero@tools-bastion-03:~$ dpkg -s mariadb-client | grep Depends
Depends: mariadb-client-5.5 (>= 5.5.63-1ubuntu0.14.04.1)

aborrero@tools-bastion-03:~$ dpkg -s mariadb-client-5.5 | grep Breaks
Breaks: mariadb-server-5.5 (<< 5.5.63-1ubuntu0.14.04.1), mysql-client, mysql-client-5.5, mysql-client-5.6, virtual-mysql-client

i.e, mariadb-client and mysql-client can't be installed at the same time. These 2 tend to be problematic. We had a similar issue in T215578: Weird dependency issue on the stretch grid around libmariadb-dev-compat and friends
We should probably standardize to one or the other across all the puppet code, if even possible (I'm aware this may have consequences for toolforge exec nodes).

In T218494#5031116, @Stashbot wrote:

Mentioned in SAL (#wikimedia-cloud) [2019-03-17T23:41:53Z] <bd808> Cherry-picked https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/497210/ as a quick fix for T218494

I'm fine with this patch, +1'd

Change 497210 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloud-vps: Remove mysql packages from openstack::clientpackages::mitaka::*

https://gerrit.wikimedia.org/r/497210

This is solved now right?

Change 497445 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Follow-up I71678b27: Remove stray MariaDB reference in openstack::clientpackages::newton::stretch

https://gerrit.wikimedia.org/r/497445

Change 497445 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] openstack: Follow-up I71678b27: Remove stray MariaDB reference

https://gerrit.wikimedia.org/r/497445

This happened over the weekend on Stretch hosts as well, and then the next puppet run resolved it. It was weird.
I'm so glad it makes more sense now. :)

Change 497445 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: Follow-up I71678b27: Remove stray MariaDB reference

https://gerrit.wikimedia.org/r/497445