@Ottomata and @elukey what do you think?

I believe we had this problem (and discussion) before...and we decided that apt pinning to backports was better than importing our own version. This allows us to more easily upgrade and keep track of what versions are used where.

@MoritzMuehlenhoff might have an opinion here?

In T209300#4740111, @Ottomata wrote:

I believe we had this problem (and discussion) before...and we decided that apt pinning to backports was better than importing our own version. This allows us to more easily upgrade and keep track of what versions are used where.

@MoritzMuehlenhoff might have an opinion here?

Fwiw he left a quick comment about this in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/472694/

@Ottomata would it be safe to upgrade stretch hosts currently using librdkafka 0.9.3-1 to 0.11.6-1~bpo9+1 across the board? I'm hoping for yes because otherwise we'll have to figure out how to satisfy the newer librdkafka version for rsyslog without disrupting hosts that depend on the older version.

I'm fairly certain there shouldn't be any stretch hosts using 0.9.3-1, or at least not any running prod services. It should be fine.

In T209300#4740139, @Ottomata wrote:

I'm fairly certain there shouldn't be any stretch hosts using 0.9.3-1, or at least not any running prod services. It should be fine.

There's quite a few, e.g. stat hosts, full list at https://debmonitor.wikimedia.org/packages/librdkafka1

In T209300#4740111, @Ottomata wrote:

I believe we had this problem (and discussion) before...and we decided that apt pinning to backports was better than importing our own version. This allows us to more easily upgrade and keep track of what versions are used where.

@MoritzMuehlenhoff might have an opinion here?

It has pros and cons: The downside of using backports is that it is a moving target, while we don't necessary want to follow up update. With a locally imported package we can more easily control when to upgrade (which is particularly important for fleet-wide installed packages where we try to keep oldstable and stable at the same version).

Oh hm. There are no prod services running on the stat boxes. We can (and should) upgrade there anyway.

The only one there that we should check on for sure is wdqs1009.eqiad.wmnet, but I'm pretty sure they use the Java client, not librdkafka for their updater service.

I think there are also some inconsistencies in the application of the apt pin. e.g. on an-coord1001, stat100[46] there's _no_ upgrade candidate, while there's one for stat1007.

It has pros and cons: The downside of using backports is that it is a moving target, while we don't necessary want to follow up update. With a locally imported package we can more easily control when to upgrade (which is particularly important for fleet-wide installed packages where we try to keep oldstable and stable at the same version).

I'm fine with either, but I'm not really sure there is a difference. Having our own version will give us control over when we upgrade, but there will still be various uses across the whole fleet for different systems that we'll have to upgrade carefully.

I'm inclined to think if the version we want is in backports, let's just pin to backports. If there is some new version that isn't yet available, then let's consider uploading to our own apt. But I'm not strongly opinionated either way :)

In T209300#4740170, @Ottomata wrote:

The only one there that we should check on for sure is wdqs1009.eqiad.wmnet, but I'm pretty sure they use the Java client, not librdkafka for their updater service.

Just checked and not seeing librdkafka in lsof, so that's good!

In T209300#4740199, @Ottomata wrote:

It has pros and cons: The downside of using backports is that it is a moving target, while we don't necessary want to follow up update. With a locally imported package we can more easily control when to upgrade (which is particularly important for fleet-wide installed packages where we try to keep oldstable and stable at the same version).

I'm fine with either, but I'm not really sure there is a difference. Having our own version will give us control over when we upgrade, but there will still be various uses across the whole fleet for different systems that we'll have to upgrade carefully.

I'm inclined to think if the version we want is in backports, let's just pin to backports. If there is some new version that isn't yet available, then let's consider uploading to our own apt. But I'm not strongly opinionated either way :)

I don't have a strong preference either, but Moritz and Filippo did both suggest going the import route, and the apt pin patch has been abandoned in favor of the package that's been built.

It sounds like it's expected to be safe and there isn't opposition to importing librdkafka_0.11.6-1~bpo9+1+wikimedia1 so I'll plan tentatively on doing that tomorrow (11-13-2018) afternoon (Eastern)

If anyone thinks this needs more review or should happen at a different time just shout!

Ok +1

Mentioned in SAL (#wikimedia-operations) [2018-11-13T19:31:25Z] <herron> uploaded librdkafka_0.11.6-1~bpo9+1+wikimedia1 packages to stretch-wikimedia T209300

Mental note for @Ottomata @Pchelolo and myself: the current node-rdkafka driver is based on librdkafka 0.11.5, so we will have to be careful when putting the new EB service in prod as well as when moving CP to k8s.