@MoritzMuehlenhoff: Is it acceptable to download the pre-build .debs, and upload them into our apt repo?

We do have packages we source from external repos, but they mostly fall under any of the categories below:

• Commercial FLOSS which bundles an huge stack of components with little to no control to fix things (Elastic, Kibana, Cloudera)
• The special case of Jenkins which do releases every 1-2 weeks
• External repos we trust (e.g. Casssandra where Eric Evans is one of the upstream devs/uploaders or the Postgres/Tor repos which are maintained by the respective Debian developers)
• Debs of closed-source hardware monitoring software (the only case where we do exceptions from our FLOSS policy) released by Dell/HP/RAID companies

There's three major reasons why we strife to avoid external repos if we have the choice:

• Security is one of them: Every deb can contain pre/postrun scripts which run with root privileges. There's ongoing work to reduce these in Debian tooling (by turning more of the steps typically run in a pre/post scripts into declarative logic defined in debian/* files, e.g. to create system users), but in the foreseeable future these will continue to exist. Backdoors can certainly also exist in source code releases, but it's harder to sneak through a git repo, while the deb build process is often separate and all it takes it to steal the credentials of a user in charge. It even happened to Red Hat's openssh in the past: https://www.securityfocus.com/bid/30794/discuss

• That said, security is not the predominant obstacle here: The bigger issue here is that relying on an external deb without an option to rebuild a custom package drastically hampers your FLOSS freedoms. Without a way to rebuild a deb, we're stuck with the distro supported by the upstream and we can't apply local patches (either because upstream made a poor choice, because we need to apply a security fix and upstream only fixed the latest release we can't use yet, something entirely site/WMF-specific or maybe because we wrote a feature we want to test in prod to eventually submit it upstream). This has bitten us many times before, e.g. large parts of analytics are currently stuck with Stretch due to many issues with the Cloudera debs (T231067 and several others tickets), with things like Hue/Cloudera bundling Django 1.6.10 with a vintage Python.

• There's also significant quality issues with vendor debs, especially when created with meta tooling like fpm like missing dependencies, incomplete use of Debian-specific tooling resulting in poor interoperability. Or there was a case of some external deb (forgot the name) 1-2 years ago which wiped users root partitions because someone used something like "rm -rf $APPBASEDIR/" in postinst and there were a few code paths which left $APPBASEDIR unset.... Or the case Kibana embedding full Java starting with ELK7: https://phabricator.wikimedia.org/T252913

Essentially it boils down to technical debt; we need to weigh the downsides of a vendor deb against the effort required to fix things cleanly: Some external repos have no practical downsides (e.g. back in the days we ran a Tor relay and were using the debs provided by the Tor project which allowed us to follow the latest stable branch released by Tor, the debs were created by the Debian Tor maintainer with the same quality standards as the main Debian archive), sometimes it's a bullet we have to bite (like Elastic, which would be a Herculean task to package the right way (this was even attempted inside Debian, but eventually abandoned https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892466#21), yet we definitely need a search feature on the wikis) and then there's various middleground.

From my PoV: Feel free to start with the vendor deb if it unblocks your work to get started with deploying Orchestrator (especially if it's still in prototype stage and we're not certain yet whether Orchestrator can/will replace Tendril and Orchestrator only running on a single server anyway), but I'd strongly suggest to account for some time later on to create a Debian source package. Orchestrator looks like a run-of-the-mill Go package with vendored deps and it shouldn't take long.

Change 635964 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] aptrepo: Add thirdparty/orchestrator

https://gerrit.wikimedia.org/r/635964

Change 635964 merged by Kormat:
[operations/puppet@production] aptrepo: Add thirdparty/orchestrator

https://gerrit.wikimedia.org/r/635964

v3.2.3 has been uploaded:

# apt policy orchestrator
orchestrator:
  Installed: (none)
  Candidate: 1:3.2.3
  Version table:
     1:3.2.3 1001
       1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/thirdparty/orchestrator amd64 Packages
# apt policy orchestrator-cli
orchestrator-cli:
  Installed: (none)
  Candidate: 1:3.2.3
  Version table:
     1:3.2.3 1001
       1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/thirdparty/orchestrator amd64 Packages
# apt policy orchestrator-client
orchestrator-client:
  Installed: (none)
  Candidate: 1:3.2.3
  Version table:
     1:3.2.3 1001
       1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/thirdparty/orchestrator amd64 Packages

Mentioned in SAL (#wikimedia-operations) [2020-10-23T10:57:08Z] <kormat> uploaded orchestrator v3.2.3 to apt.wikimedia.org buster-wikimedia - T266023 (forgot to log this earlier)

Change 638443 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] orchestrator: Drop the obsolete thirdparty/orchestrator component.

https://gerrit.wikimedia.org/r/638443

Change 638443 merged by Kormat:
[operations/puppet@production] orchestrator: Drop the obsolete thirdparty/orchestrator component.

https://gerrit.wikimedia.org/r/638443