Page MenuHomePhabricator

Edits are made from internal cluster IPs: en.wiktionary, pl.wikipedia, and other sites
Closed, ResolvedPublic

Description

The Stewards received a report that WMF internal cluster IPs successfully save edits at pl.wikipedia, en.wiktionary and other sites. Example edits:

and other edits, see https://xtools.wmflabs.org/globalcontribs/ipr-2620:0:861::/48.

IPs are assigned to appservers:

urbanecm@notebook  ~
$ host 2620:0:861:10b:10:64:132:10 ns1.wikimedia.org
Using domain server:
Name: ns1.wikimedia.org
Address: 208.80.153.231#53
Aliases:

0.1.0.0.2.3.1.0.4.6.0.0.0.1.0.0.b.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa domain name pointer mw1492.eqiad.wmnet.
urbanecm@notebook  ~
$ host 2620:0:861:10a:10:64:131:5 ns1.wikimedia.org
Using domain server:
Name: ns1.wikimedia.org
Address: 208.80.153.231#53
Aliases:

5.0.0.0.1.3.1.0.4.6.0.0.0.1.0.0.a.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa domain name pointer mw1489.eqiad.wmnet.
urbanecm@notebook  ~
$

@taavi says wmf-config/reverse-proxy.php needs to be fixed to include new eqiad row e and f subnets. Assigning to them based on our IRC conversation.

Details

Event Timeline

Note I've blocked the IPv6 addresses to stop the vandalism. This has user impact as of now.

Urbanecm triaged this task as Unbreak Now! priority.Nov 29 2022, 1:41 PM

Has user impact => UBN.

Urbanecm lowered the priority of this task from Unbreak Now! to Needs Triage.Nov 29 2022, 1:43 PM

Relevant servers were depooled.

Change 861848 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/mediawiki-config@master] reverse-proxy: Add eqiad e/f[1-4] subnets

https://gerrit.wikimedia.org/r/861848

Change 861848 merged by jenkins-bot:

[operations/mediawiki-config@master] reverse-proxy: Add eqiad e/f[1-4] subnets

https://gerrit.wikimedia.org/r/861848

Mentioned in SAL (#wikimedia-operations) [2022-11-29T14:14:29Z] <taavi@deploy1002> Started scap: Backport for [[gerrit:861848|reverse-proxy: Add eqiad e/f[1-4] subnets (T324018)]]

Mentioned in SAL (#wikimedia-operations) [2022-11-29T14:16:02Z] <taavi@deploy1002> taavi and taavi: Backport for [[gerrit:861848|reverse-proxy: Add eqiad e/f[1-4] subnets (T324018)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2022-11-29T14:22:03Z] <taavi@deploy1002> Finished scap: Backport for [[gerrit:861848|reverse-proxy: Add eqiad e/f[1-4] subnets (T324018)]] (duration: 07m 33s)

Snrk, here's an event from a quick dig before being pointed at this, but everything looks like it's back to normal now