Page MenuHomePhabricator

reimage maps-test2004 to stretch and cassandra 2.2
Closed, ResolvedPublic

Description

It make sense to upgrade to stretch as part of the migration to vector tiles, since we need to reimage the servers anyway. Bring newer packages, in particular fonts is needed, and upgrading to a more recent version of cassandra will help on the long term.

Details

Related Gerrit Patches:

Event Timeline

Gehel triaged this task as Normal priority.May 28 2018, 8:38 AM
Gehel created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 28 2018, 8:38 AM
Gehel moved this task from Backlog to In progress on the Maps-Sprint board.May 28 2018, 8:39 AM

Change 435740 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: reimage maps-test2004 to stretch

https://gerrit.wikimedia.org/r/435740

Change 435741 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: set cassandra version based on role, not based on regexes

https://gerrit.wikimedia.org/r/435741

Change 435746 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: enable prometheus metrics for cassandra on maps-test instance

https://gerrit.wikimedia.org/r/435746

Change 435740 merged by Gehel:
[operations/puppet@production] maps: reimage maps-test2004 to stretch

https://gerrit.wikimedia.org/r/435740

Change 435741 merged by Gehel:
[operations/puppet@production] maps: set cassandra version based on role, not based on regexes

https://gerrit.wikimedia.org/r/435741

Change 435746 merged by Gehel:
[operations/puppet@production] maps: enable prometheus metrics for cassandra on maps-test2004

https://gerrit.wikimedia.org/r/435746

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2004.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805281252_gehel_26403.log.

Change 435789 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: don't fix postgresql version

https://gerrit.wikimedia.org/r/435789

Change 435789 merged by Gehel:
[operations/puppet@production] maps: don't fix postgresql version

https://gerrit.wikimedia.org/r/435789

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2004.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805281429_gehel_15485.log.

Completed auto-reimage of hosts:

['maps-test2004.codfw.wmnet']

Of which those FAILED:

['maps-test2004.codfw.wmnet']
Gehel added a comment.May 28 2018, 5:40 PM

It looks like we have a few missing dependencies in Stretch:

  • jvm-tools (only available on jessie, could probably just be copied to stretch, need to check if the java7-runtime-headless dep is satisfied by openjdk-8)
  • python-support >= 0.90.0 (required by Cassandra=2.2.6-wmf3, can't find it anywhere, seems to have disappeared from reprepro, looks like it should be a package-time dependency, but I'm a bit lost here...)
  • osmborder (only available on jessie, not sure if it will need rebuilding, see https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/osmborder)

It looks like we have a few missing dependencies in Stretch:

  • jvm-tools (only available on jessie, could probably just be copied to stretch, need to check if the java7-runtime-headless dep is satisfied by openjdk-8)

It's not satisfied, we'll need to rebuild the package for stretch. Should be straightforward, though.

  • python-support >= 0.90.0 (required by Cassandra=2.2.6-wmf3, can't find it anywhere, seems to have disappeared from reprepro, looks like it should be a package-time dependency, but I'm a bit lost here...)

python-support was replaced by dh-python. We should not reintroduce it in stretch, but instead import cassandra 2.2.6 into a separate component for stretch and fix it to use dh-python (which I suppose was already done for cassandra 3 anyway).

Yeah, it's has binary dependencies against libstdc++ and that changed between jessie and stretch. It will need to be rebuild. Should be straightforward, let me know if I can help.

We may need to deploy different binaries for kartotherian and tilerator on Jessie and Stretch. The package contains at least one hard-coded reference to jessie

Gehel added a subscriber: elukey.May 29 2018, 7:26 AM

It looks like we have a few missing dependencies in Stretch:

  • jvm-tools (only available on jessie, could probably just be copied to stretch, need to check if the java7-runtime-headless dep is satisfied by openjdk-8)

It's not satisfied, we'll need to rebuild the package for stretch. Should be straightforward, though.

It looks like openjdk-8-jre-headless does provide java7-runtime-headless:

gehel@maps-test2004:~$ apt-cache show openjdk-8-jre-headless=8u171-b11-1~deb9u1 | grep Provides
Provides: java2-runtime-headless, java5-runtime-headless, java6-runtime-headless, java7-runtime-headless, java8-runtime-headless

Is there something I don't understand here (well, I'm sure there are tons of things I don't understand...) ?

Anyway, I can rebuild the package, no big deal.

  • python-support >= 0.90.0 (required by Cassandra=2.2.6-wmf3, can't find it anywhere, seems to have disappeared from reprepro, looks like it should be a package-time dependency, but I'm a bit lost here...)

python-support was replaced by dh-python. We should not reintroduce it in stretch, but instead import cassandra 2.2.6 into a separate component for stretch and fix it to use dh-python (which I suppose was already done for cassandra 3 anyway).

I have only a very vague idea what "fix it to use dh-python" means. @MoritzMuehlenhoff / @elukey: help would be appreciated on this one.

Yeah, it's has binary dependencies against libstdc++ and that changed between jessie and stretch. It will need to be rebuild. Should be straightforward, let me know if I can help.

Rebuilding osmborder should be no issue, I'll try and let you know.

We may need to deploy different binaries for kartotherian and tilerator on Jessie and Stretch. The package contains at least one hard-coded reference to jessie

There is probably a conversation to have with @Mholloway to see if we want to create a "stretch" branch in the -package and -deploy repos during the transition.

gehel@maps-test2004:~$ apt-cache show openjdk-8-jre-headless=8u171-b11-1~deb9u1 | grep Provides
Provides: java2-runtime-headless, java5-runtime-headless, java6-runtime-headless, java7-runtime-headless, java8-runtime-headless

Is there something I don't understand here (well, I'm sure there are tons of things I don't understand...) ?

Ah, you're right. It missed the alternate dependency, this could also simply be copied over. Note that there's also WIP to move to a new upstream release in https://phabricator.wikimedia.org/T178839

  • python-support >= 0.90.0 (required by Cassandra=2.2.6-wmf3, can't find it anywhere, seems to have disappeared from reprepro, looks like it should be a package-time dependency, but I'm a bit lost here...)

python-support was replaced by dh-python. We should not reintroduce it in stretch, but instead import cassandra 2.2.6 into a separate component for stretch and fix it to use dh-python (which I suppose was already done for cassandra 3 anyway).

I have only a very vague idea what "fix it to use dh-python" means. @MoritzMuehlenhoff / @elukey: help would be appreciated on this one.

dh-python is now the standard way to build Python code for Debian. Historically there used to be several tools (with python-support being one of them), but eventually they all got coalesced to dh-python. It's quite likely that Eric already fixed that in the Cassandra 3 packages, I can have a look later on.

Gehel added a comment.May 29 2018, 8:10 AM

jvm-tools tested and copied from jessie-wikimedia to stretch-wikimedia

Gehel added a comment.May 29 2018, 9:25 AM

osmborder rebuilt and uploaded to reprepro. Only Cassandra left.

Gehel added a subscriber: Eevans.May 29 2018, 9:38 AM

I can't find a repo in gerrit for cassandra packaging. @elukey / @Eevans any idea where it could be?

tail-kartotherian shows a undefined symbol error. The best way to reproduce this is cd /srv/deployment/kartotherian/deploy-cache/current && nodejs -e "require('mapnik')"

module.js:597
  return process.dlopen(module, path._makeLong(filename));
                 ^

Error: /srv/deployment/kartotherian/deploy-cache/revs/9fc09ef37894cbcac437ba48209cf21a8c6829c7/node_modules/mapnik/lib/binding/node-v48-linux-x64/mapnik.node: undefined symbol: _ZN6mapnik14save_to_stringINS_5imageINS_7rgba8_tEEEEESsRKT_RKSsRKNS_12rgba_paletteE
    at Error (native)
    at Object.Module._extensions..node (module.js:597:18)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/srv/deployment/kartotherian/deploy-cache/revs/9fc09ef37894cbcac437ba48209cf21a8c6829c7/node_modules/mapnik/lib/mapnik.js:31:15)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)

This occurs because the kartotherian and tilerator deploy repos were built on jessie, while we're running on stretch. Specifically, https://gerrit.wikimedia.org/r/plugins/gitiles/maps/kartotherian/deploy/+/master/node_modules/mapnik/ has binaries which are built against stretch.

It was suggested we do a stretch branch of package and deploy repos, so I'm going to try that and see if I can get the service-runner build stuff, scap, and git-review to all cooperate and work on a branch that isn't named master.

I'm working on the scap3 stuff, but leaving the issue assigned to @Gehel since he's still got the Cassandra/etc stuff.

Just to note so it doesn't disappear into a long .bash_history, someone trying to follow the stretch work I'm doing will probably need to do git config deploy.srcbranch stretch && git config deploy.deploybranch stretch and switch that setting back and forth.

@Eevans packaged a new cassandra-2.2.6-wmf4 on https://people.wikimedia.org/~eevans/debian, we can test it and see how it works...

I can't find a repo in gerrit for cassandra packaging. @elukey / @Eevans any idea where it could be?

For posterity sake: https://github.com/wikimedia/cassandra/tree/wmf/2.2.6

Just to note so it doesn't disappear into a long .bash_history, someone trying to follow the stretch work I'm doing will probably need to do git config deploy.srcbranch stretch && git config deploy.deploybranch stretch and switch that setting back and forth.

Thanks, noted.

Change 436308 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: upgrade to cassandra 2.2.6-wmf4

https://gerrit.wikimedia.org/r/436308

Change 436308 merged by Gehel:
[operations/puppet@production] maps: upgrade to cassandra 2.2.6-wmf4

https://gerrit.wikimedia.org/r/436308

Gehel added a comment.May 30 2018, 6:53 PM

cassandra and cassandra-tools-wmf are manually installed from https://people.wikimedia.org/~eevans/debian/ on maps-test2004, after testing I'll upload them to reprepro.

I'm getting what looks to be https://issues.apache.org/jira/browse/CASSANDRA-11850 when I try to connect to Cassandra.

Yes, it's confirmed to be that issue. The workaround was @Gehel installing the python-cassandra package, then using CQLSH_NO_BUNDLED=TRUE cqlsh to connect

Pnorman moved this task from In progress to Done on the Maps-Sprint board.Jun 4 2018, 6:42 PM
Gehel closed this task as Resolved.Jun 12 2018, 4:06 PM
Vvjjkkii renamed this task from reimage maps-test2004 to stretch and cassandra 2.2 to 25baaaaaaa.Jul 1 2018, 1:07 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed Gehel as the assignee of this task.
Vvjjkkii raised the priority of this task from Normal to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed subscribers: gerritbot, Aklapper.
WhitePhosphorus renamed this task from 25baaaaaaa to reimage maps-test2004 to stretch and cassandra 2.2.Jul 1 2018, 2:53 AM
WhitePhosphorus closed this task as Resolved.
WhitePhosphorus assigned this task to Gehel.
WhitePhosphorus lowered the priority of this task from High to Normal.
WhitePhosphorus updated the task description. (Show Details)
WhitePhosphorus added subscribers: gerritbot, Aklapper.