Change 435740 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: reimage maps-test2004 to stretch

https://gerrit.wikimedia.org/r/435740

Change 435741 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: set cassandra version based on role, not based on regexes

https://gerrit.wikimedia.org/r/435741

Change 435746 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: enable prometheus metrics for cassandra on maps-test instance

https://gerrit.wikimedia.org/r/435746

Change 435740 merged by Gehel:
[operations/puppet@production] maps: reimage maps-test2004 to stretch

https://gerrit.wikimedia.org/r/435740

Change 435741 merged by Gehel:
[operations/puppet@production] maps: set cassandra version based on role, not based on regexes

https://gerrit.wikimedia.org/r/435741

Change 435746 merged by Gehel:
[operations/puppet@production] maps: enable prometheus metrics for cassandra on maps-test2004

https://gerrit.wikimedia.org/r/435746

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2004.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805281252_gehel_26403.log.

Change 435789 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: don't fix postgresql version

https://gerrit.wikimedia.org/r/435789

Change 435789 merged by Gehel:
[operations/puppet@production] maps: don't fix postgresql version

https://gerrit.wikimedia.org/r/435789

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2004.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805281429_gehel_15485.log.

Completed auto-reimage of hosts:

['maps-test2004.codfw.wmnet']

Of which those FAILED:

['maps-test2004.codfw.wmnet']

It looks like we have a few missing dependencies in Stretch:

• jvm-tools (only available on jessie, could probably just be copied to stretch, need to check if the java7-runtime-headless dep is satisfied by openjdk-8)
• python-support >= 0.90.0 (required by Cassandra=2.2.6-wmf3, can't find it anywhere, seems to have disappeared from reprepro, looks like it should be a package-time dependency, but I'm a bit lost here...)
• osmborder (only available on jessie, not sure if it will need rebuilding, see https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/osmborder)

In T195741#4237089, @Gehel wrote:

It looks like we have a few missing dependencies in Stretch:

• jvm-tools (only available on jessie, could probably just be copied to stretch, need to check if the java7-runtime-headless dep is satisfied by openjdk-8)

It's not satisfied, we'll need to rebuild the package for stretch. Should be straightforward, though.

• python-support >= 0.90.0 (required by Cassandra=2.2.6-wmf3, can't find it anywhere, seems to have disappeared from reprepro, looks like it should be a package-time dependency, but I'm a bit lost here...)

python-support was replaced by dh-python. We should not reintroduce it in stretch, but instead import cassandra 2.2.6 into a separate component for stretch and fix it to use dh-python (which I suppose was already done for cassandra 3 anyway).

• osmborder (only available on jessie, not sure if it will need rebuilding, see https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/osmborder)

Yeah, it's has binary dependencies against libstdc++ and that changed between jessie and stretch. It will need to be rebuild. Should be straightforward, let me know if I can help.

We may need to deploy different binaries for kartotherian and tilerator on Jessie and Stretch. The package contains at least one hard-coded reference to jessie

In T195741#4237117, @MoritzMuehlenhoff wrote:

In T195741#4237089, @Gehel wrote:

It looks like we have a few missing dependencies in Stretch:

• jvm-tools (only available on jessie, could probably just be copied to stretch, need to check if the java7-runtime-headless dep is satisfied by openjdk-8)

It's not satisfied, we'll need to rebuild the package for stretch. Should be straightforward, though.

It looks like openjdk-8-jre-headless does provide java7-runtime-headless:

gehel@maps-test2004:~$ apt-cache show openjdk-8-jre-headless=8u171-b11-1~deb9u1 | grep Provides
Provides: java2-runtime-headless, java5-runtime-headless, java6-runtime-headless, java7-runtime-headless, java8-runtime-headless

Is there something I don't understand here (well, I'm sure there are tons of things I don't understand...) ?

Anyway, I can rebuild the package, no big deal.

• python-support >= 0.90.0 (required by Cassandra=2.2.6-wmf3, can't find it anywhere, seems to have disappeared from reprepro, looks like it should be a package-time dependency, but I'm a bit lost here...)

python-support was replaced by dh-python. We should not reintroduce it in stretch, but instead import cassandra 2.2.6 into a separate component for stretch and fix it to use dh-python (which I suppose was already done for cassandra 3 anyway).

I have only a very vague idea what "fix it to use dh-python" means. @MoritzMuehlenhoff / @elukey: help would be appreciated on this one.

• osmborder (only available on jessie, not sure if it will need rebuilding, see https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/osmborder)

Yeah, it's has binary dependencies against libstdc++ and that changed between jessie and stretch. It will need to be rebuild. Should be straightforward, let me know if I can help.

Rebuilding osmborder should be no issue, I'll try and let you know.

In T195741#4237539, @Pnorman wrote:

We may need to deploy different binaries for kartotherian and tilerator on Jessie and Stretch. The package contains at least one hard-coded reference to jessie

There is probably a conversation to have with @Mholloway to see if we want to create a "stretch" branch in the -package and -deploy repos during the transition.

In T195741#4237841, @Gehel wrote:

gehel@maps-test2004:~$ apt-cache show openjdk-8-jre-headless=8u171-b11-1~deb9u1 | grep Provides
Provides: java2-runtime-headless, java5-runtime-headless, java6-runtime-headless, java7-runtime-headless, java8-runtime-headless

Is there something I don't understand here (well, I'm sure there are tons of things I don't understand...) ?

Ah, you're right. It missed the alternate dependency, this could also simply be copied over. Note that there's also WIP to move to a new upstream release in https://phabricator.wikimedia.org/T178839

• python-support >= 0.90.0 (required by Cassandra=2.2.6-wmf3, can't find it anywhere, seems to have disappeared from reprepro, looks like it should be a package-time dependency, but I'm a bit lost here...)

python-support was replaced by dh-python. We should not reintroduce it in stretch, but instead import cassandra 2.2.6 into a separate component for stretch and fix it to use dh-python (which I suppose was already done for cassandra 3 anyway).

I have only a very vague idea what "fix it to use dh-python" means. @MoritzMuehlenhoff / @elukey: help would be appreciated on this one.

dh-python is now the standard way to build Python code for Debian. Historically there used to be several tools (with python-support being one of them), but eventually they all got coalesced to dh-python. It's quite likely that Eric already fixed that in the Cassandra 3 packages, I can have a look later on.

jvm-tools tested and copied from jessie-wikimedia to stretch-wikimedia

osmborder rebuilt and uploaded to reprepro. Only Cassandra left.

I can't find a repo in gerrit for cassandra packaging. @elukey / @Eevans any idea where it could be?

tail-kartotherian shows a undefined symbol error. The best way to reproduce this is cd /srv/deployment/kartotherian/deploy-cache/current && nodejs -e "require('mapnik')"

module.js:597
  return process.dlopen(module, path._makeLong(filename));
                 ^

Error: /srv/deployment/kartotherian/deploy-cache/revs/9fc09ef37894cbcac437ba48209cf21a8c6829c7/node_modules/mapnik/lib/binding/node-v48-linux-x64/mapnik.node: undefined symbol: _ZN6mapnik14save_to_stringINS_5imageINS_7rgba8_tEEEEESsRKT_RKSsRKNS_12rgba_paletteE
    at Error (native)
    at Object.Module._extensions..node (module.js:597:18)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/srv/deployment/kartotherian/deploy-cache/revs/9fc09ef37894cbcac437ba48209cf21a8c6829c7/node_modules/mapnik/lib/mapnik.js:31:15)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)

This occurs because the kartotherian and tilerator deploy repos were built on jessie, while we're running on stretch. Specifically, https://gerrit.wikimedia.org/r/plugins/gitiles/maps/kartotherian/deploy/+/master/node_modules/mapnik/ has binaries which are built against stretch.

It was suggested we do a stretch branch of package and deploy repos, so I'm going to try that and see if I can get the service-runner build stuff, scap, and git-review to all cooperate and work on a branch that isn't named master.

I'm working on the scap3 stuff, but leaving the issue assigned to @Gehel since he's still got the Cassandra/etc stuff.

Just to note so it doesn't disappear into a long .bash_history, someone trying to follow the stretch work I'm doing will probably need to do git config deploy.srcbranch stretch && git config deploy.deploybranch stretch and switch that setting back and forth.

@Eevans packaged a new cassandra-2.2.6-wmf4 on https://people.wikimedia.org/~eevans/debian, we can test it and see how it works...

In T195741#4238201, @Gehel wrote:

I can't find a repo in gerrit for cassandra packaging. @elukey / @Eevans any idea where it could be?

For posterity sake: https://github.com/wikimedia/cassandra/tree/wmf/2.2.6

In T195741#4241483, @Pnorman wrote:

Just to note so it doesn't disappear into a long .bash_history, someone trying to follow the stretch work I'm doing will probably need to do git config deploy.srcbranch stretch && git config deploy.deploybranch stretch and switch that setting back and forth.

Thanks, noted.

Change 436308 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: upgrade to cassandra 2.2.6-wmf4

https://gerrit.wikimedia.org/r/436308

Change 436308 merged by Gehel:
[operations/puppet@production] maps: upgrade to cassandra 2.2.6-wmf4

https://gerrit.wikimedia.org/r/436308

cassandra and cassandra-tools-wmf are manually installed from https://people.wikimedia.org/~eevans/debian/ on maps-test2004, after testing I'll upload them to reprepro.

I'm getting what looks to be https://issues.apache.org/jira/browse/CASSANDRA-11850 when I try to connect to Cassandra.

Yes, it's confirmed to be that issue. The workaround was @Gehel installing the python-cassandra package, then using CQLSH_NO_BUNDLED=TRUE cqlsh to connect