Review and make librdkafka-0.11.6 installable from stretch-wikimedia
Closed, ResolvedPublic

Description

As part of the logging goal this quarter the rsyslog-kafka pacakge will be gradually rolled out to virtually all hosts. On stretch this carries a dependency of librdkafka1 (>= 0.9.4) However, the version currently available in stretch is 0.9.3-1, with 0.11.6-1~bpo9+1 in stretch-backports.

As discussed in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/472694/ it will be preferable to backport librdkafka-0.11.6-1~bpo9+1 to stretch-wikimedia (as opposed to apt pinning the package to stretch-backports on stretch hosts)

Creating this task to:

  • Review the librdkafka_0.11.6-1~bpo9+1+wikimedia1 package (currently in the stretch-amd64 pbuilder result directory on boron) before importing to stretch-wikimedia and deploying to the fleet
  • Discuss if it is necessary to align librdkafka versions between jessie and stretch.
herron created this task.Nov 12 2018, 5:25 PM
herron triaged this task as Normal priority.
herron added a subscriber: elukey.Nov 12 2018, 5:27 PM

@Ottomata and @elukey what do you think?

I believe we had this problem (and discussion) before...and we decided that apt pinning to backports was better than importing our own version. This allows us to more easily upgrade and keep track of what versions are used where.

@MoritzMuehlenhoff might have an opinion here?

I believe we had this problem (and discussion) before...and we decided that apt pinning to backports was better than importing our own version. This allows us to more easily upgrade and keep track of what versions are used where.

@MoritzMuehlenhoff might have an opinion here?

Fwiw he left a quick comment about this in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/472694/

@Ottomata would it be safe to upgrade stretch hosts currently using librdkafka 0.9.3-1 to 0.11.6-1~bpo9+1 across the board? I'm hoping for yes because otherwise we'll have to figure out how to satisfy the newer librdkafka version for rsyslog without disrupting hosts that depend on the older version.

fdans moved this task from Incoming to Radar on the Analytics board.Nov 12 2018, 5:38 PM
Ottomata added a comment.EditedNov 12 2018, 5:40 PM

I'm fairly certain there shouldn't be any stretch hosts using 0.9.3-1, or at least not any running prod services. It should be fine.

I'm fairly certain there shouldn't be any stretch hosts using 0.9.3-1, or at least not any running prod services. It should be fine.

There's quite a few, e.g. stat hosts, full list at https://debmonitor.wikimedia.org/packages/librdkafka1

I believe we had this problem (and discussion) before...and we decided that apt pinning to backports was better than importing our own version. This allows us to more easily upgrade and keep track of what versions are used where.

@MoritzMuehlenhoff might have an opinion here?

It has pros and cons: The downside of using backports is that it is a moving target, while we don't necessary want to follow up update. With a locally imported package we can more easily control when to upgrade (which is particularly important for fleet-wide installed packages where we try to keep oldstable and stable at the same version).

Oh hm. There are no prod services running on the stat boxes. We can (and should) upgrade there anyway.

The only one there that we should check on for sure is wdqs1009.eqiad.wmnet, but I'm pretty sure they use the Java client, not librdkafka for their updater service.

I think there are also some inconsistencies in the application of the apt pin. e.g. on an-coord1001, stat100[46] there's _no_ upgrade candidate, while there's one for stat1007.

It has pros and cons: The downside of using backports is that it is a moving target, while we don't necessary want to follow up update. With a locally imported package we can more easily control when to upgrade (which is particularly important for fleet-wide installed packages where we try to keep oldstable and stable at the same version).

I'm fine with either, but I'm not really sure there is a difference. Having our own version will give us control over when we upgrade, but there will still be various uses across the whole fleet for different systems that we'll have to upgrade carefully.

I'm inclined to think if the version we want is in backports, let's just pin to backports. If there is some new version that isn't yet available, then let's consider uploading to our own apt. But I'm not strongly opinionated either way :)

The only one there that we should check on for sure is wdqs1009.eqiad.wmnet, but I'm pretty sure they use the Java client, not librdkafka for their updater service.

Just checked and not seeing librdkafka in lsof, so that's good!

It has pros and cons: The downside of using backports is that it is a moving target, while we don't necessary want to follow up update. With a locally imported package we can more easily control when to upgrade (which is particularly important for fleet-wide installed packages where we try to keep oldstable and stable at the same version).

I'm fine with either, but I'm not really sure there is a difference. Having our own version will give us control over when we upgrade, but there will still be various uses across the whole fleet for different systems that we'll have to upgrade carefully.

I'm inclined to think if the version we want is in backports, let's just pin to backports. If there is some new version that isn't yet available, then let's consider uploading to our own apt. But I'm not strongly opinionated either way :)

I don't have a strong preference either, but Moritz and Filippo did both suggest going the import route, and the apt pin patch has been abandoned in favor of the package that's been built.

It sounds like it's expected to be safe and there isn't opposition to importing librdkafka_0.11.6-1~bpo9+1+wikimedia1 so I'll plan tentatively on doing that tomorrow (11-13-2018) afternoon (Eastern)

If anyone thinks this needs more review or should happen at a different time just shout!

Mentioned in SAL (#wikimedia-operations) [2018-11-13T19:31:25Z] <herron> uploaded librdkafka_0.11.6-1~bpo9+1+wikimedia1 packages to stretch-wikimedia T209300

herron closed this task as Resolved.Nov 13 2018, 8:11 PM
herron claimed this task.

Mental note for @Ottomata @Pchelolo and myself: the current node-rdkafka driver is based on librdkafka 0.11.5, so we will have to be careful when putting the new EB service in prod as well as when moving CP to k8s.