Page MenuHomePhabricator
Create Task
Maniphest T341696

Zero traffic on bare metal ORES servers - COMPLETE
Open, Needs TriagePublic
Actions

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -1profile::trafficserver::backend: switch ores traffic to ores-legacy
operations/dnsmaster+1 -1Lower ores.wikimedia.org's TTL to 5M
Customize query in gerrit

Event Timeline

calbon created this task.Jul 12 2023, 2:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 12 2023, 2:14 PM
calbon renamed this task from proposed: completely done with ORES to proposed: Zero Traffic On Bare Metal ORES.Jul 12 2023, 2:15 PM
calbon renamed this task from proposed: Zero Traffic On Bare Metal ORES to Proposed Goal: Zero Traffic On Bare Metal ORES.
calbon renamed this task from Proposed Goal: Zero Traffic On Bare Metal ORES to Proposed Goal: Zero traffic on bare metal ORES servers.
calbon renamed this task from Proposed Goal: Zero traffic on bare metal ORES servers to Goal: Zero traffic on bare metal ORES servers.Jul 12 2023, 2:54 PM
calbon added a project: Epic.Jul 12 2023, 3:05 PM
calbon edited projects, added Goal; removed Epic.Jul 17 2023, 7:23 PM
calbon renamed this task from Goal: Zero traffic on bare metal ORES servers to Zero traffic on bare metal ORES servers.Jul 17 2023, 7:27 PM
elukey added a subscriber: elukey.Tue, Sep 5, 1:01 PM

Update:

  • ores-legacy is fully in production, scaled up and ready to take traffic.
  • The WME team is going to move their traffic to Lift Wing tomorrow (if everything goes as planned).
  • After an initial false start, the MediaWiki ORES extension is being configured to use Lift Wing. We have currently only two wikis with the feature enabled (itwiki and fiwiki), but we are planning to enable it for all the remaining ones during then next couple of weeks.
  • The WME team's migration will remove one of the biggest consumers of the revision-score stream, so we'll be able to hopefully announce its deprecation after that (removing the only blocker left to remove ChangeProp's traffic from the bare metal ORES nodes' traffic).
isarantopoulos added a subscriber: isarantopoulos.Tue, Sep 5, 2:47 PM

All wikis except enwiki and wikidata are now using Lift Wing for the recent changes filters.

elukey assigned this task to isarantopoulos.Tue, Sep 5, 3:04 PM
isarantopoulos added a comment.Wed, Sep 6, 2:53 PM

The plan is the following:

  • We get enwiki and wikidata (all mediawiki extension traffic) on LW next week (around 11-13 September).
  • Given all goes well, we communicate in wikitech-l of our intention to switch ores.wikimedia.org DNS entry to point to ores-legacy around 20-27th of September.
  • Do the DNS switch.
isarantopoulos added a comment.EditedTue, Sep 12, 2:15 PM

All MediaWiki traffic through the ORES extension has been redirected to use Lift Wing starting from 11th of September.

isarantopoulos added a comment.Tue, Sep 19, 8:20 AM

The changeprop stream has been stopped and ORES traffic has significantly decreased (4 times less traffic than when the stream was activated.)
Current requests are approx 600k/day of which approx 300k come from Twisted Pagegetter - a bot/crawler that only hits /v3/scores

gerritbot added a comment.Wed, Sep 20, 9:51 AM

Change 957689 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/dns@master] Lower ores.wikimedia.org's TTL to 5M

https://gerrit.wikimedia.org/r/957689

gerritbot added a project: Patch-For-Review.Wed, Sep 20, 9:51 AM
Stashbot added a comment.Wed, Sep 20, 10:02 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-20T10:02:23Z] <klausman> Merging change 957689 (T341696) to lower DNS TTL to 5m for ORES name.

gerritbot added a comment.Wed, Sep 20, 10:02 AM

Change 957689 merged by Klausman:

[operations/dns@master] Lower ores.wikimedia.org's TTL to 5M

https://gerrit.wikimedia.org/r/957689

MoritzMuehlenhoff added a subscriber: MoritzMuehlenhoff.Wed, Sep 20, 10:02 AM
Stashbot added a comment.Wed, Sep 20, 10:03 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-20T10:02:59Z] <klausman> RUnning authdns-update to activate change 957689 (T341696)

Maintenance_bot removed a project: Patch-For-Review.Wed, Sep 20, 10:11 AM
gerritbot added a comment.Thu, Sep 21, 3:04 PM

Change 959762 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::trafficserver::backend: switch ores traffic to ores-legacy

https://gerrit.wikimedia.org/r/959762

gerritbot added a project: Patch-For-Review.Thu, Sep 21, 3:04 PM
gerritbot added a comment.Mon, Sep 25, 8:58 AM

Change 959762 merged by Ladsgroup:

[operations/puppet@production] profile::trafficserver::backend: switch ores traffic to ores-legacy

https://gerrit.wikimedia.org/r/959762

Stashbot added a comment.Mon, Sep 25, 8:58 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-25T08:58:52Z] <elukey> migrate ores.wikimedia.org's ATS backend to ores-legacy.discovery.wmnet (k8s app) - This will drain traffic to ORES bare metal nodes - T341696

Maintenance_bot removed a project: Patch-For-Review.Mon, Sep 25, 9:11 AM
calbon renamed this task from Zero traffic on bare metal ORES servers to Zero traffic on bare metal ORES servers - COMPLETE.Tue, Sep 26, 2:30 PM
Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL