Currently we are downloading Flink from Apache via wget. Sometimes the gpg key check fails, and the blubber build has to be restarted. @Ottomata shared a way his team created Deb packages for Presto: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/presto/+/refs/heads/debian/debian/README.Debian. We also have our own example for the Elasticsearch plugins.
AC
Flink can be installed via a Debian package
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| aptrepo: add flink to the bigtop14 package list | operations/puppet | production | +1 -1 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Gehel | T244590 [Epic] Rework the WDQS updater as an event driven application | |||
| Resolved | Gehel | T264006 Deploy Flink (rdf-streaming-updater) to kubernetes (k8s) | |||
| Declined | None | T266495 Create Debian Package for Flink |
I don't know if it's relevant at all, but anlytics is in the process of switching to apache bigtop (from cloudera hadoop). That includes flink debs, would we want to use that?
apache bigtop (from cloudera hadoop). That includes flink debs, would we want to use that?
OH HO! I did not know that. Hopefully that would be best! @elukey FYI
Change 636944 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] aptrepo: add flink to the bigtop14 package list
Change 636944 merged by Elukey:
[operations/puppet@production] aptrepo: add flink to the bigtop14 package list
root@apt1001:/srv/wikimedia# reprepro lsbycomponent flink flink | 1.6.4-1 | stretch-wikimedia | thirdparty/bigtop14 | amd64 flink | 1.6.4-1 | buster-wikimedia | thirdparty/bigtop14 | amd64
This is Bigtop 1.4 version, the last upstream, but they are about to release 1.5 so flink will probably be more up to date.
I see https://issues.apache.org/jira/browse/BIGTOP-3382 to add 1.11.1, but it seems that the work will be done after Bigtop 1.5. We could think about contributing to upstream if we feel the need to have that included to Bigtop 1.5!
In the interim (until we have the deb package) @dcausse suggested putting the Flink packages on people.wikimedia.org since packages on the official Flink download sites get updated and removed pretty frequently.
(See https://phabricator.wikimedia.org/T273097#6805355 for why this ticket has been closed)