Page MenuHomePhabricator

prepare mcrouter package for Debian Buster
Closed, ResolvedPublic

Description

mcrouter doesn't seem to be available on Buster:

# reprepro ls mcrouter
mcrouter | 0.37.0-1~jessie0 |  jessie-wikimedia | amd64, source
mcrouter |         0.37.0-1 | stretch-wikimedia | amd64, source

Shall I just forward the stretch package to buster, or does it need a rebuild?

Event Timeline

Andrew created this task.Apr 30 2020, 9:28 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 30 2020, 9:28 PM
Andrew added a subscriber: Joe.Apr 30 2020, 9:30 PM
Andrew added a comment.May 4 2020, 4:21 PM

In the SRE meeting today, @Joe mentioned that this might be messy -- weird build process and also we might want to do version upgrades. So whoever works on this should check in with him first.

jbond added a subscriber: jbond.May 6 2020, 2:59 PM

@elukey I'm don't remember why I assigned this to you :) Are you actively working on it or does it need a new home?

jbond added a comment.May 19 2020, 9:07 AM

@Andrew I have actully made a bit of progress on this here

https://gerrit.wikimedia.org/r/c/operations/debs/mcrouter/+/596779

Sorry i didn't link the task as it was still a bit WIP

elukey removed elukey as the assignee of this task.May 19 2020, 12:26 PM
elukey added a subscriber: elukey.

@jbond feel free to assign the task to yourself, I was added by mistake, not working on it :)

jbond claimed this task.May 19 2020, 12:59 PM
jbond triaged this task as Medium priority.

So I think there are separate things here

  • How to rebuild it for Buster
  • How to build mcrouter in general

(I tried to rebuild our current package for Buster, but 0.37.0 is incompatible with OpenSSL 1.1, this is likely fixed in current releases.)

As for the approach to build it using Docker: I think the current repo that we use is actually quite nice, it bundles the thirdparty addons without a stable ABI, which was the same approach that was used by Faidon when HHVM (which also uses all those Facebook-internal libs) was packaged in Debian (which was eventually abandoned when we moved off HHVM). It's a clean package by Debian standards (ofc ideally folly et. al. would have stable interfaces and proper sonames, but it is what it is). There's also an RFP bug for mcrouter in Debian (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=946495) and we could even consider to upload our package to Debian once we have updated to current releases.

If the current approach with the bundled libs turns out to be unfeasible with current mcrouter releases, then for doing the Docker build route I think it would it would make sense to follow the upstream build process for the vendor debs provided by Facebook. They are at https://github.com/facebook/mcrouter/tree/packaging/packaging/ubuntu and we can probably easily adapt them to use Debian.

jbond added a comment.May 19 2020, 3:15 PM

(I tried to rebuild our current package for Buster, but 0.37.0 is incompatible with OpenSSL 1.1, this is likely fixed in current releases.)

Correct the 0.41 version in the docker process builds correctly

As for the approach to build it using Docker: I think the current repo that we use is actually quite nice, it bundles the thirdparty addons without a stable ABI, which was the same approach that was used by Faidon when HHVM (which also uses all those Facebook-internal libs) was packaged in Debian (which was eventually abandoned when we moved off HHVM). It's a clean package by Debian standards (ofc ideally folly et. al. would have stable interfaces and proper sonames, but it is what it is).

The reason i went this route, after chatting withg @Joe because instead of 2 third party dependencies the 0.41 version has 5 dependencies. Further there where changes to the build process which meant one needs to rework the patch. I certainly found trying to update the current process much harder then creating this docker script (of course not i have this script i probably could hack it back into the rules file) and i feel that future updates would be equally or more problematic.

Of course there are many here much more familiar with the debian build process then I but i think there are equally many with less knowledge then I. My gut feeling is that this has potentially not been tackled and updated because it is a complicate process and difficult for many to navigate. Going forward it would be good to have something that anyone in SRE could fairly easily update

If the current approach with the bundled libs turns out to be unfeasible with current mcrouter releases, then for doing the Docker build route I think it would it would make sense to follow the upstream build process for the vendor debs provided by Facebook. They are at https://github.com/facebook/mcrouter/tree/packaging/packaging/ubuntu and we can probably easily adapt them to use Debian.

That's exactly what i have used, docker_entry.sh is just a cleaned up version of the FB packaging scripts

jbond added a comment.May 19 2020, 3:19 PM

That's exactly what i have used, docker_entry.sh is just a cleaned up version of the FB packaging scripts

Correction i based my script on https://github.com/facebook/mcrouter/tree/master/mcrouter/scripts, will take a look at the link

Change 596779 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/debs/mcrouter@master] docker build: update the build process to us docker

https://gerrit.wikimedia.org/r/596779

Change 596779 merged by Jbond:
[operations/debs/mcrouter@master] docker build: update the build process to us docker

https://gerrit.wikimedia.org/r/596779

mcrouter 0.4.1 is now available in buster-wikimedia

$ sudo -E reprepro ls mcrouter                                           
mcrouter | 0.37.0-1~jessie0 |  jessie-wikimedia | amd64, source
mcrouter |         0.37.0-1 | stretch-wikimedia | amd64, source
mcrouter |         0.41.0-1 |  buster-wikimedia | amd64
jbond closed this task as Resolved.May 27 2020, 12:30 PM

I did a simple hand test of this (setting on one host using the local mcrouter port, getting on another host using that host's local mcrouter) and it looks good. I've also get designate-producer working which uses mcrouter for coordination and it seems happy.

Thank you for this!

jbond added a comment.May 28 2020, 8:48 AM

Awesome thanks