Page MenuHomePhabricator

upgrade planet instances to stretch
Closed, ResolvedPublic

Description

upgrade both planet1001 and planet2001 from jessie to stretch.

This task is for the actual reinstall of the prod instances.

A requirement for that was replacing the planet-venus software with rawdog but this has moved to subtask T180498.


was:

make needed changes in puppet role

also.. replace the whole software since planet-venus doesn't exist in stretch anymore and is largely unmaintained

https://tracker.debian.org/pkg/planet-venus

one candidate is "rawdog" which would be preferred since it's a package in stable

http://offog.org/code/rawdog/

the other is "planet pluto" which is not a package and uses Ruby (gem install...)

https://github.com/feedreader

rawdog uses Python and Feedparser just like planet-venus did.

Event Timeline

Dzahn created this task.Jun 21 2017, 1:22 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 21 2017, 1:22 AM

Change 360596 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] planet: planet-venus not in stretch, try using "rawdog" instead

https://gerrit.wikimedia.org/r/360596

Dzahn updated the task description. (Show Details)Jun 21 2017, 1:24 AM
Dzahn triaged this task as Normal priority.Jun 21 2017, 1:26 AM

Change 360596 merged by Dzahn:
[operations/puppet@production] planet: planet-venus not in stretch, try using "rawdog" instead

https://gerrit.wikimedia.org/r/360596

Change 360597 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] planet: do not install libapache2-mod-php5

https://gerrit.wikimedia.org/r/360597

Change 360597 merged by Dzahn:
[operations/puppet@production] planet: do not install libapache2-mod-php5

https://gerrit.wikimedia.org/r/360597

Change 360598 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] planet: disable codfw backend temporarily

https://gerrit.wikimedia.org/r/360598

Change 360598 merged by Dzahn:
[operations/puppet@production] planet: disable codfw backend temporarily

https://gerrit.wikimedia.org/r/360598

Change 360599 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] planet: install python-tidylib for rawdog

https://gerrit.wikimedia.org/r/360599

Change 360599 merged by Dzahn:
[operations/puppet@production] planet: install python-tidylib for rawdog

https://gerrit.wikimedia.org/r/360599

Change 360600 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] planet: add support/compat for stretch and rawdog

https://gerrit.wikimedia.org/r/360600

Change 360600 merged by Dzahn:
[operations/puppet@production] planet: add support/compat for stretch and rawdog

https://gerrit.wikimedia.org/r/360600

Change 360601 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] planet: fix logo path and duplicate declare of config dir

https://gerrit.wikimedia.org/r/360601

Dzahn added subscribers: Dereckson, Paladox, Framawiki.
Dzahn added a subscriber: Nemo_bis.

Change 360601 merged by Dzahn:
[operations/puppet@production] planet: fix logo path and duplicate declare of config dir

https://gerrit.wikimedia.org/r/360601

Change 360603 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] planet: fix invalid relationship in cronjob, dirs on stretch

https://gerrit.wikimedia.org/r/360603

Change 360603 merged by Dzahn:
[operations/puppet@production] planet: fix invalid relationship in cronjob, dirs on stretch

https://gerrit.wikimedia.org/r/360603

Change 360910 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] planet: make config templates and crons flexible for rawdog

https://gerrit.wikimedia.org/r/360910

Change 360910 merged by Dzahn:
[operations/puppet@production] planet: make config templates and crons for rawdog

https://gerrit.wikimedia.org/r/360910

Change 360990 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] planet: fix "splitstate" option, config file name, icon

https://gerrit.wikimedia.org/r/360990

Change 360990 merged by Dzahn:
[operations/puppet@production] planet: fix "splitstate" option, config file name, icon

https://gerrit.wikimedia.org/r/360990

Change 361180 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] planet: add HTML page/item/feed templates for rawdog

https://gerrit.wikimedia.org/r/361180

Change 361180 merged by Dzahn:
[operations/puppet@production] planet: add HTML page/item/feed templates for rawdog

https://gerrit.wikimedia.org/r/361180

Paladox updated the task description. (Show Details)Nov 14 2017, 4:44 PM
Paladox updated the task description. (Show Details)Nov 14 2017, 5:03 PM
Dzahn updated the task description. (Show Details)Nov 14 2017, 5:28 PM
Dzahn updated the task description. (Show Details)

Change 392983 had a related patch set uploaded (by Paladox; owner: Paladox):
[operations/puppet@production] planet: Add support for http/2 on stretch

https://gerrit.wikimedia.org/r/392983

Dzahn added a comment.May 25 2018, 7:52 PM

15:33 <mutante> 15:31 <@James_F> mutante: Will the feed URL stay the same? If so, most won't notice the switchover.
15:33 <mutante> 15:33 < mutante> James_F: that's a good one. i will check that and if needed add some RewriteRule or so

Dzahn added a comment.May 25 2018, 8:00 PM

@Jdforrester-WMF I should have used T180498. That is the ticket for replacing the software. This here was just for the actual server upgrade .. then we ran into the blocker that planet-venus got dropped in stretch and then we found rawdog as the replacement. So i made it a subtask.

Change 435218 had a related patch set uploaded (by Paladox; owner: Paladox):
[operations/puppet@production] Planet: Replace rss20.xml with atom.xml (backwards compat filename)

https://gerrit.wikimedia.org/r/435218

Change 435218 abandoned by Paladox:
Planet: Replace rss20.xml with atom.xml (backwards compat filename)

Reason:
This is not atom format so we need to use rss20.

Which supports atom readers i think?

https://gerrit.wikimedia.org/r/435218

Change 436426 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] planet/misc-varnish: add codfw backend to director

https://gerrit.wikimedia.org/r/436426

Change 436427 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] planet/misc-varnish: comment out eqiad backend for reinstall

https://gerrit.wikimedia.org/r/436427

Change 436426 merged by Dzahn:
[operations/puppet@production] planet/misc-varnish: add codfw backend to director

https://gerrit.wikimedia.org/r/436426

Change 436427 merged by Dzahn:
[operations/puppet@production] planet/misc-varnish: comment out eqiad backend for reinstall

https://gerrit.wikimedia.org/r/436427

Dzahn added a comment.May 31 2018, 3:35 AM

Now planet2001 is on stretch, uses puppetized rawdog and is serving live traffic.

planet1001 is currently commented out in cache::misc and still has planet-venus on jessie.

For a short transitional period it will run like this to be able to revert easily if needed for any unexpected reason.

Next planet1001 will be reinstalled with stretch effectively removing planet-venus from production.

Finally the service can be active-active again in varnish which it hasn't been for a while since planet2001 has been upgraded to stretch for rawdog support.

Dzahn added a comment.May 31 2018, 3:38 AM

proper revert to go back to planet-venus would be to first revert

https://gerrit.wikimedia.org/r/#/c/436427/2/hieradata/role/common/cache/misc.yaml

and then, in a second change, comment out codfw and remove the comment from eqiad. for $reasons we should never switch a cache::misc backend in a single step from eqiad-only to codfw-only and vice versa.

There's quite a bit of crom spam by planet2001:

<planet@planet2001> http_proxy="http://url-downloader.codfw.wikimedia.org:8080" https_proxy=$http_proxy /usr/bin/rawdog -d /etc/rawdog/en/ -v -u -w > /var/log/planet/en-planet.log 2>&1

/bin/sh: 1: cannot create /var/log/planet/en-planet.log: Permission denied
Dzahn added a comment.Jun 1 2018, 12:52 PM

Fixed ownership of the "en" logfile. It was owned by root:root, all others by planet:planet. It was from manually running update command.

Change 437975 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] install_server: switch planet1001 from jessie to stretch

https://gerrit.wikimedia.org/r/437975

Change 437975 merged by Dzahn:
[operations/puppet@production] install_server: switch planet1001 from jessie to stretch

https://gerrit.wikimedia.org/r/437975

Mentioned in SAL (#wikimedia-operations) [2018-06-07T12:26:55Z] <mutante> planet1001 - schedule downtime, boot to PXE, reinstall with stretch (ganeti) (T168490)

Mentioned in SAL (#wikimedia-operations) [2018-06-07T12:46:08Z] <mutante> planet1001/puppetmaster: revoke old cert, sign new cert request, initial puppet run, reinstalled, will turn service active-active again once done (T168490)

Change 437987 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] planet: make active-active again, reactivate eqiad backend

https://gerrit.wikimedia.org/r/437987

Change 437987 merged by Dzahn:
[operations/puppet@production] planet: make active-active again, reactivate eqiad backend

https://gerrit.wikimedia.org/r/437987

Dzahn closed this task as Resolved.Jun 7 2018, 7:55 PM

done. planet is served again from both data centers at once. planet1001 and planet2001 are both on stretch and use rawdog. planet-venus has been decom'ed for being unmaintained upstream