After a chat with Filippo, IIUC 8.2008.0-1 is used only on centrallog nodes (hence the component) but we might want to use 8.19011 provided on Buster and add the custom bits for rsyslog-kubernetes, uploading all to main. The alternative would be to modify 8.2008.0 and use the component instead (so adding rsyslog-kubernetes to it).

Any preference? :)

I have no idea why 8.2008.0-1 is used on centrallog , but if it is compatible with 8.1901 and can have kubernetes support, I 'd just upload it to main.

I am a little confused from the debian/stretch-wikimedia branch of the rsyslog repo, because the debian changelog seems to mention 8.38.0-1~bpo9+1wmf1 (it corresponds to a commit from Filippo to enable mmkubernetes).

In T277739#6924397, @elukey wrote:

I am a little confused from the debian/stretch-wikimedia branch of the rsyslog repo, because the debian changelog seems to mention 8.38.0-1~bpo9+1wmf1 (it corresponds to a commit from Filippo to enable mmkubernetes).

Self answering: I tried apt-get source rsyslog and its changelog mentions:

rsyslog (8.1901.0-1~bpo9+wmf1) stretch-backports; urgency=medium

  * Enable mmkubernetes (build depends on libcurl and liblognorm) and build
    rsyslog-kubernetes

 -- Filippo Giunchedi <filippo@wikimedia.org>  Mon, 22 Oct 2018 14:40:32 +0200

rsyslog (8.1901.0-1~bpo9+1) stretch-backports; urgency=medium

So I guess that we took the rsyslog in stretch-backports and applied Filippo's patch on top, but it wasn't saved in our repository. I would do the same for Buster as well, seems the easiest:

1. get the source of https://packages.debian.org/buster/rsyslog
2. Apply Filippo's patch
3. upload to buster wikimedia with +wmf1

In T277739#6924819, @elukey wrote:

In T277739#6924397, @elukey wrote:

I am a little confused from the debian/stretch-wikimedia branch of the rsyslog repo, because the debian changelog seems to mention 8.38.0-1~bpo9+1wmf1 (it corresponds to a commit from Filippo to enable mmkubernetes).

Self answering: I tried apt-get source rsyslog and its changelog mentions:

rsyslog (8.1901.0-1~bpo9+wmf1) stretch-backports; urgency=medium

  * Enable mmkubernetes (build depends on libcurl and liblognorm) and build
    rsyslog-kubernetes

 -- Filippo Giunchedi <filippo@wikimedia.org>  Mon, 22 Oct 2018 14:40:32 +0200

rsyslog (8.1901.0-1~bpo9+1) stretch-backports; urgency=medium

So I guess that we took the rsyslog in stretch-backports and applied Filippo's patch on top, but it wasn't saved in our repository. I would do the same for Buster as well, seems the easiest:

1. get the source of https://packages.debian.org/buster/rsyslog
2. Apply Filippo's patch
3. upload to buster wikimedia with +wmf1

+1

It probably makes sense to figure out what to do with the repo then though. archive it? sync it?

In T277739#6924365, @akosiaris wrote:

After a chat with Filippo, IIUC 8.2008.0-1 is used only on centrallog nodes (hence the component) but we might want to use 8.19011 provided on Buster and add the custom bits for rsyslog-kubernetes, uploading all to main. The alternative would be to modify 8.2008.0 and use the component instead (so adding rsyslog-kubernetes to it).

Any preference? :)

I have no idea why 8.2008.0-1 is used on centrallog , but if it is compatible with 8.1901 and can have kubernetes support, I 'd just upload it to main.

I've tried it on centrallog at upstream' suggestion to see if it'd fix T199406 (it doesn't) and T259780, however it was decided to leave the rest of the fleet unchanged unless we have to (and thus needed to upgrade the fleet). The package needed no changes though, as it was a straight backport from the version in testing (now at 8.2102.0).

In T277739#6924397, @elukey wrote:

I am a little confused from the debian/stretch-wikimedia branch of the rsyslog repo, because the debian changelog seems to mention 8.38.0-1~bpo9+1wmf1 (it corresponds to a commit from Filippo to enable mmkubernetes).

Indeed, it looks like we've upgraded rsyslog fleetwide 8.1901.0 but never committed the change to operations/debs/rsyslog

In T277739#6924860, @akosiaris wrote:

In T277739#6924819, @elukey wrote:

In T277739#6924397, @elukey wrote:

I am a little confused from the debian/stretch-wikimedia branch of the rsyslog repo, because the debian changelog seems to mention 8.38.0-1~bpo9+1wmf1 (it corresponds to a commit from Filippo to enable mmkubernetes).

Self answering: I tried apt-get source rsyslog and its changelog mentions:

rsyslog (8.1901.0-1~bpo9+wmf1) stretch-backports; urgency=medium

  * Enable mmkubernetes (build depends on libcurl and liblognorm) and build
    rsyslog-kubernetes

 -- Filippo Giunchedi <filippo@wikimedia.org>  Mon, 22 Oct 2018 14:40:32 +0200

rsyslog (8.1901.0-1~bpo9+1) stretch-backports; urgency=medium

So I guess that we took the rsyslog in stretch-backports and applied Filippo's patch on top, but it wasn't saved in our repository. I would do the same for Buster as well, seems the easiest:

1. get the source of https://packages.debian.org/buster/rsyslog
2. Apply Filippo's patch
3. upload to buster wikimedia with +wmf1

+1

It probably makes sense to figure out what to do with the repo then though. archive it? sync it?

I'm +1 on keeping the repo since we have to carry the patch anyways. On which version to enable mmkubernetes I'm not sure, I see the following solutions though:

1. Import 8.2102.0 in operations/debs/rsyslog (to have a version likely shipped in a Debian release) and enable mmkubernetes, upload to component/rsyslog, then enable the component on kubernetes hosts as well.
2. Enable mmkubernetes on 8.2008.0 and upload to component/rsyslog, then enable the component on kubernetes hosts as well.
3. Enable mmkubernetes on 8.1901.0 and upload to main in buster-wikimedia, and upgrade the fleet.

I think overall I prefer solution 1, and can of course assist with its implementation.

This is what I did on deneb before reading Filippo's answer :)

• apt-get source rsyslog -t buster
• applied Filippo's patch manually (git diff HEAD~1 HEAD on the debian/stretch-wikimedia branch of the rsyslog gerrit repo)
• DIST=buster pdebuild (all tests passed)

elukey@deneb:~/rsyslog-8.1901.0$ ls /var/cache/pbuilder/result/buster-amd64/rsyslog*8.1901.0-1+wmf1*  | grep kubernetes
/var/cache/pbuilder/result/buster-amd64/rsyslog-kubernetes_8.1901.0-1+wmf1_amd64.deb
/var/cache/pbuilder/result/buster-amd64/rsyslog-kubernetes-dbgsym_8.1901.0-1+wmf1_amd64.deb

Update after a chat with Filippo:

• a fleetwide update of rsyslog is painful, we can avoid it with the component solution (extremely wise point)
• update the rsyslog repo seems good - master branch to version 8.2102 (like centrallog / bulleye / unstable) and then custom buster-wikimedia branch with Filippo's patch

I have updated the operations/debs/rsyslog from salsa.debian.org, now it contains 8.2102.0. I then tried something simple:

• created a local branch debian/buster-wikimedia from master
• applied Filippo's patch to create 8.2102.0+wmf1
• ran DIST=buster pdebuild (relying on the .orig file from apt-get source rsyslog -t unstable)

It failed, due to debhelper-compact=13, so I lowered it down in control and kicked off the build again. A new error came up:

The following packages have unmet dependencies:
pbuilder-satisfydepends-dummy : Depends: librelp-dev (>= 1.4.0) but it is not going to be installed

Filippo must have dealt with this when creating 8.2008.0-1~bpo10+1 for buster-backports, since I see:

elukey@centrallog1001:~$ apt-cache policy librelp-dev
librelp-dev:
  Installed: (none)
  Candidate: 1.7.0-1~bpo10+1
  Version table:
     1.7.0-1~bpo10+1 1001
       1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/component/rsyslog amd64 Packages
     1.3.0-1 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages

I am pretty sure that the problem here is my limited debian packaging knowledge, so how can I build rsyslog 8.2102.0 using librelp-dev 1.7.0-1~bpo10+1 ?

Change 673442 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] aptrepo: add a new rsyslog-k8s component for buster-wikimedia

https://gerrit.wikimedia.org/r/673442

Change 673442 merged by Elukey:
[operations/puppet@production] aptrepo: add a new rsyslog-k8s component for buster-wikimedia

https://gerrit.wikimedia.org/r/673442

After a chat with Moritz we decided to create a specific component with 8.1901 for buster:

root@apt1001:/srv/wikimedia# reprepro lsbycomponent rsyslog
rsyslog | 8.1901.0-1~bpo8+wmf1 |  jessie-wikimedia |                  main | amd64, source
rsyslog | 8.1901.0-1~bpo9+wmf1 | stretch-wikimedia |                  main | amd64, source
rsyslog |   8.2008.0-1~bpo10+1 |  buster-wikimedia |     component/rsyslog | amd64, source
rsyslog |      8.1901.0-1+wmf1 |  buster-wikimedia | component/rsyslog-k8s | amd64, source

I'll file another code review to apply it for kubernetes on buster, and then we'll need to do something similar for bullseye. Filing a change for https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911299 ourselves could be good as well, so post bullseye we'll not have this issue.

Change 673450 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::rsyslog::kubernetes: add component for buster

https://gerrit.wikimedia.org/r/673450

Change 673450 merged by Elukey:
[operations/puppet@production] profile::rsyslog::kubernetes: add component for buster

https://gerrit.wikimedia.org/r/673450

Coming back to this task since Janis has a patch that needs to be added on top of 8.1901.0-1+wmf1 :)

At the time I recall that I just picked buster's upstream version for rsyslog 8.1901.0-1 and I applied on top of it Filippo's fix to get rsyslog-kubernetes. The idea was to avoid a major rebuild of rsyslog (and then related deploy across all nodes) only for the kubernetes use case.

We have some options on the table:

1. Figure out where 8.2008.0-1~bpo10+1 comes from in buster-wikimedia, create a branch based on it in the rsyslog repo and apply Filippo's and Janis' patches on top. Rebuild and rollout everywhere.
2. Keep going with the current scheme (Buster upstream version with manual patches on top), but of course it is not really scalable.
3. Something else.

Bullseye is out and there is not rsyslog-kubernetes in it, maybe we could start working with upstream to have it in unstable first and possibly in backports?

In T277739#7344522, @elukey wrote:

Bullseye is out and there is not rsyslog-kubernetes in it, maybe we could start working with upstream to have it in unstable first and possibly in backports?

+1

I'd say we do 2. for short term (easier to do, less nodes to update, more in line with what we currently have) plus 3. in form of trying to get a sponsored upload to debian upstream that enables mmkubernetes (if that's possible).

Unfortunately I've not heard back from upstream as to if my patch actually makes sense. So I'm not sure if that will land in debian.

New plan that was discussed between me and Janis on IRC:

1. In our rsyslog repo, git pull upstream debian/master to get the last updates on master.
2. Create a new branch in our rsyslog repo called debian/buster-wikimedia-k8s (name entirely debatable). The idea is to avoid confusion to people checking out the repo, since we use another version for all the non-k8s hosts on Buster.
3. The branch will be cut from e0f331dc32a45ccf73c92aefe7022f2c8a468b55 (tag: debian/8.1901.0-1) and we'll cherry-pick 84797fd8fbb46c83687833e0ee3503780fbc27f2 and 26f0024dcccc540207d85705054e4211afa92b56 (the two patches that we are interested in).
4. Push the branch
5. Build the package and upload it to its component.

We'll wait till Monday to get feedback from people, then we'll move forward if nobody disagrees :)

In T277739#7344860, @elukey wrote:

New plan that was discussed between me and Janis on IRC:

1. In our rsyslog repo, git pull upstream debian/master to get the last updates on master.

I think this has already been done. Our master is at "8.2102.0-2" (af12eb3ab346b0d758cac56590b4ba4111f9f730).

In T277739#7344860, @elukey wrote:

2. Create a new branch in our rsyslog repo called debian/buster-wikimedia-k8s (name entirely debatable). The idea is to avoid confusion to people checking out the repo, since we use another version for all the non-k8s hosts on Buster.
3. The branch will be cut from e0f331dc32a45ccf73c92aefe7022f2c8a468b55 (tag: debian/8.1901.0-1) and we'll cherry-pick 84797fd8fbb46c83687833e0ee3503780fbc27f2 and 26f0024dcccc540207d85705054e4211afa92b56 (the two patches that we are interested in).

3.5 Fix version to be 8.1901.0-1+wmf2 ;-)

In T277739#7344860, @elukey wrote:

4. Push the branch
5. build the package and upload it to its component.

6. Write about what/why on https://wikitech.wikimedia.org/wiki/Rsyslog to lower chances of insanity for our future selves.

FWIW I'm +1 and happy to assist/review/etc

I pushed manually to debian/buster-wikimedia-k8s the base 8.1901.0-1 version and published two patches starting from https://gerrit.wikimedia.org/r/c/operations/debs/rsyslog/+/720669/1

I've added a couple of lines to https://wikitech.wikimedia.org/wiki/Rsyslog#Packaging
I'm not sure what the source for 8.2008.0 is but if we want to keep it around we should potentially push a branch for that as well to have something to refer to.

root@apt1001:/srv/wikimedia# reprepro lsbycomponent rsyslog
rsyslog | 8.1901.0-1~bpo9+wmf2 | stretch-wikimedia |                  main | amd64, source
rsyslog |   8.2008.0-1~bpo10+1 |  buster-wikimedia |     component/rsyslog | amd64, source
rsyslog |      8.1901.0-1+wmf2 |  buster-wikimedia | component/rsyslog-k8s | amd64, source

root@apt1001:/srv/wikimedia# reprepro lsbycomponent rsyslog-kubernetes
rsyslog-kubernetes | 8.1901.0-1~bpo9+wmf2 | stretch-wikimedia |                  main | amd64
rsyslog-kubernetes |      8.1901.0-1+wmf2 |  buster-wikimedia | component/rsyslog-k8s | amd64

Should be done!

Mentioned in SAL (#wikimedia-operations) [2021-09-13T09:11:50Z] <elukey> upload rsyslog* 8.1901.0-1+wmf2 to buster-wikimedia component/rsyslog-k8s - T277739