Weird dependency issue on the stretch grid around libmariadb-dev-compat and friends
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Bstorm
	Feb 8 2019, 12:46 AM

Project Tags

Referenced Files

None

Subscribers

Description

So something is happening on the stretch grid that has baffled me for some time, but it has broken puppet and apt repeatedly on tools-sgebastion-06.

libgdal-dev (a gis, mappy kinda library) is installed because someone needed it at some point on the old grid. I presume they will still require it on the new grid. So that depends on libmysqlclient-dev which actually installs libmariadbclient-dev-compat (which just creates a few symlinks to convince things they can link against mysql libs). We also have libmariadb-dev-compat which creates the same damned symlinks and a few more.

The problem is that those two packages (libmariadbclient-dev-compat and libmariadb-dev-compat) are mutually exclusive packages. If I could choose one, I'd choose libmariadb-dev-compat because it is a superset of the other and might be better for developers as a result. Sadly puppet is reinstalling things back and forth it seems. On the bastion, it has begun dying on install. You end up with broken apt, and you have to run dpkgs --configure -a to get it working again. Then it breaks on the next run. It's really fun.

To make this story all the better, in Debian Sid, libmariadbclient-dev-compat is a virtual package that points to libmariadb-dev-compat ❤️. This seems like what it should be everywhere, but it is not. Also, a backport would mean our mariadb packages are at 10.3 rather than 10.1 like the actual databases are. I think the solution might be to remove libmariadb-dev-compat from modules/profile/manifests/toolforge/grid/exec_environ.pp even though it seems like the better one, but I wanted to create this ticket to get help from @aborrero to make sure.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	toolforge: grid: exec_environ: use libmariadbclient-dev* packages	operations/puppet	production	+2 -2

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	• bd808	T55704 Packages to be added to toollabs puppet
Resolved	• bd808	T213646 Stretch grid problem: Please install packages libmariadbclient-dev-compat and libssl-dev
Resolved	• aborrero	T215578 Weird dependency issue on the stretch grid around libmariadb-dev-compat and friends

Event Timeline

• Bstorm created this task.Feb 8 2019, 12:46 AM

• Bstorm triaged this task as Medium priority.

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptFeb 8 2019, 12:46 AM

• bd808 subscribed.Feb 8 2019, 12:53 AM

I have realized that the worst of this issue (the crashing and requiring dpkg --configure -a) may be because of the cgroup limits. Puppet is getting SIGKILLed during apt runs because I'm running it using sudo. The rest is probably just a matter of removing libmariadb-dev-compat.

Noticed the problems around cgroups here: T215434 and have greatly expanded the limits for now.

Yes! That was it. It just dances back and forth installing one dependency or the other on each puppet run like it always did now :)

TL;DR: I think we can have this combo installed: libgdal-dev libmariadbclient-dev-compat libmariadbclient-dev (even though I have doubts we need the -compat one). Will create a patch proposal for this.

From a quick reading, my understanding is that the '-compat' package provides some sort of compatibility with mysql, specifically to allow co-existance of mysql and mariadb client libraries in the system.
Since you mentioned the database is mariadb, perhaps we can forget about this compatibility?

To make things even more confusing, there are 2 flavors of the -dev packages, identified by the client keyword. Apparently they have different licensing:

libmariadb-dev: LGPL
libmariadbclient-dev: GPL

root@tools-sgebastion-06:~# aptitude show libmariadb-dev
Package: libmariadb-dev           
Version: 2.3.2-2
[...]
The main reason to use this package <<libmariadb-dev>>, rather than libmariadbclient-dev, is differences in licensing: this package uses the LGPL, while libmariadbclient-dev uses the GPL.

And we also have 2 flavors for the -compat version:

libmariadb-dev-compat: LGPL, allows co-existence with mysql libs
libmariadbclient-dev-compat: GPL, allows co-existence with mysql libs

I just did a quick installation test to see how the resolver behaves in this situation:

test1 -- NOOK

root@tools-sgebastion-06:~# apt-get install libgdal-dev libmariadb-dev-compat libmariadb-dev -sy
Reading package lists... Done
Building dependency tree       
Reading state information... Done
libmariadb-dev is already the newest version (2.3.2-2).
libmariadb-dev-compat is already the newest version (2.3.2-2).
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 libgdal-dev : Depends: default-libmysqlclient-dev but it is not going to be installed
 libmariadb-dev-compat : Conflicts: libmariadbclient-dev but 10.1.37-0+deb9u1 is to be installed
E: Unable to correct problems, you have held broken packages.

test2 -- OK

root@tools-sgebastion-06:~# apt-get install libgdal-dev libmariadbclient-dev-compat libmariadbclient-dev -sy
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  default-libmysqlclient-dev
Suggested packages:
  libgdal-doc
The following packages will be REMOVED:
  libmariadb-dev-compat
The following NEW packages will be installed:
  default-libmysqlclient-dev libgdal-dev libmariadbclient-dev libmariadbclient-dev-compat
0 upgraded, 4 newly installed, 1 to remove and 22 not upgraded.
Remv libmariadb-dev-compat [2.3.2-2]
Inst libmariadbclient-dev (10.1.37-0+deb9u1 Debian-Security:9/stable [amd64])
Inst libmariadbclient-dev-compat (10.1.37-0+deb9u1 Debian-Security:9/stable [amd64])
Inst default-libmysqlclient-dev (1.0.2 Debian:9.7/stable [amd64])
Inst libgdal-dev (2.1.2+dfsg-5 Debian:9.7/stable [amd64])
Conf libmariadbclient-dev (10.1.37-0+deb9u1 Debian-Security:9/stable [amd64])
Conf libmariadbclient-dev-compat (10.1.37-0+deb9u1 Debian-Security:9/stable [amd64])
Conf default-libmysqlclient-dev (1.0.2 Debian:9.7/stable [amd64])
Conf libgdal-dev (2.1.2+dfsg-5 Debian:9.7/stable [amd64])

So this combination seems right: libgdal-dev libmariadbclient-dev-compat libmariadbclient-dev, which only introduce the licensing change from what we have in puppet right now.

Change 489195 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: grid: exec_environ: use libmariadbclient* packages

https://gerrit.wikimedia.org/r/489195

gerritbot added a project: Patch-For-Review.Feb 8 2019, 12:55 PM

• aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Feb 8 2019, 12:58 PM

Change 489195 merged by GTirloni:
[operations/puppet@production] toolforge: grid: exec_environ: use libmariadbclient-dev* packages

https://gerrit.wikimedia.org/r/489195

• Bstorm awarded a token.Feb 8 2019, 4:48 PM

Closing task. Feel free to reopen if required.

• aborrero mentioned this in T218494: `become`, `crontab` et al missing from Trusty hosts.Mar 18 2019, 10:27 AM