mcrouter doesn't seem to be available on Buster:
# reprepro ls mcrouter mcrouter | 0.37.0-1~jessie0 | jessie-wikimedia | amd64, source mcrouter | 0.37.0-1 | stretch-wikimedia | amd64, source
Shall I just forward the stretch package to buster, or does it need a rebuild?
| Project | Branch | Lines +/- | Subject | |
|---|---|---|---|---|
| operations/debs/mcrouter | master | +534 -39 | docker build: update the build process to us docker |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Stalled | None | T302086 Set scap minimum python version to 3.7 | |||
| Open | None | T247045 Migrate all of production metal and VMs to Buster or later | |||
| Resolved | MoritzMuehlenhoff | T224549 Track remaining jessie systems in production | |||
| Resolved | None | T244852 Upgrade and improve our application object caching service (memcached) | |||
| Resolved | jijiki | T213089 Upgrade memcached cluster to Debian Buster | |||
| Resolved | Andrew | T251294 Upgrade cloud-vps control plane to Debian Buster | |||
| Resolved | jbond | T251574 Package mcrouter 0.41 for Debian Buster |
In the SRE meeting today, @Joe mentioned that this might be messy -- weird build process and also we might want to do version upgrades. So whoever works on this should check in with him first.
@elukey I'm don't remember why I assigned this to you :) Are you actively working on it or does it need a new home?
@Andrew I have actully made a bit of progress on this here
https://gerrit.wikimedia.org/r/c/operations/debs/mcrouter/+/596779
Sorry i didn't link the task as it was still a bit WIP
@jbond feel free to assign the task to yourself, I was added by mistake, not working on it :)
So I think there are separate things here
(I tried to rebuild our current package for Buster, but 0.37.0 is incompatible with OpenSSL 1.1, this is likely fixed in current releases.)
As for the approach to build it using Docker: I think the current repo that we use is actually quite nice, it bundles the thirdparty addons without a stable ABI, which was the same approach that was used by Faidon when HHVM (which also uses all those Facebook-internal libs) was packaged in Debian (which was eventually abandoned when we moved off HHVM). It's a clean package by Debian standards (ofc ideally folly et. al. would have stable interfaces and proper sonames, but it is what it is). There's also an RFP bug for mcrouter in Debian (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=946495) and we could even consider to upload our package to Debian once we have updated to current releases.
If the current approach with the bundled libs turns out to be unfeasible with current mcrouter releases, then for doing the Docker build route I think it would it would make sense to follow the upstream build process for the vendor debs provided by Facebook. They are at https://github.com/facebook/mcrouter/tree/packaging/packaging/ubuntu and we can probably easily adapt them to use Debian.
Correct the 0.41 version in the docker process builds correctly
As for the approach to build it using Docker: I think the current repo that we use is actually quite nice, it bundles the thirdparty addons without a stable ABI, which was the same approach that was used by Faidon when HHVM (which also uses all those Facebook-internal libs) was packaged in Debian (which was eventually abandoned when we moved off HHVM). It's a clean package by Debian standards (ofc ideally folly et. al. would have stable interfaces and proper sonames, but it is what it is).
The reason i went this route, after chatting withg @Joe because instead of 2 third party dependencies the 0.41 version has 5 dependencies. Further there where changes to the build process which meant one needs to rework the patch. I certainly found trying to update the current process much harder then creating this docker script (of course not i have this script i probably could hack it back into the rules file) and i feel that future updates would be equally or more problematic.
Of course there are many here much more familiar with the debian build process then I but i think there are equally many with less knowledge then I. My gut feeling is that this has potentially not been tackled and updated because it is a complicate process and difficult for many to navigate. Going forward it would be good to have something that anyone in SRE could fairly easily update
If the current approach with the bundled libs turns out to be unfeasible with current mcrouter releases, then for doing the Docker build route I think it would it would make sense to follow the upstream build process for the vendor debs provided by Facebook. They are at https://github.com/facebook/mcrouter/tree/packaging/packaging/ubuntu and we can probably easily adapt them to use Debian.
That's exactly what i have used, docker_entry.sh is just a cleaned up version of the FB packaging scripts
That's exactly what i have used, docker_entry.sh is just a cleaned up version of the FB packaging scripts
Correction i based my script on https://github.com/facebook/mcrouter/tree/master/mcrouter/scripts, will take a look at the link
Change 596779 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/debs/mcrouter@master] docker build: update the build process to us docker
Change 596779 merged by Jbond:
[operations/debs/mcrouter@master] docker build: update the build process to us docker
mcrouter 0.4.1 is now available in buster-wikimedia
$ sudo -E reprepro ls mcrouter mcrouter | 0.37.0-1~jessie0 | jessie-wikimedia | amd64, source mcrouter | 0.37.0-1 | stretch-wikimedia | amd64, source mcrouter | 0.41.0-1 | buster-wikimedia | amd64
I did a simple hand test of this (setting on one host using the local mcrouter port, getting on another host using that host's local mcrouter) and it looks good. I've also get designate-producer working which uses mcrouter for coordination and it seems happy.
Thank you for this!