WDQS tests can no longer edit test.wikidata.org
Closed, ResolvedPublic

Description

CI jobs for WDQS now fail with:

18:12:20    > Throwable #1: org.wikidata.query.rdf.tool.exception.RetryableException: Error result from Mediawiki:  {code=permissiondenied, info=You do not have the permissions needed to carry out this action., messages=[{name=wikibase-api-permissiondenied, parameters=[], html={*=You do not have the permissions needed to carry out this action.}}, {name=systemblockedtext, parameters=[[[User:MediaWiki default|MediaWiki default]], Anonymous contributions are not allowed from your IP address (172.16.2.221). Please log in., 172.16.2.221, MediaWiki default, wgSoftBlockRanges, infinite, 172.16.2.221, 18:12, 7 November 2018], html={*=<p>Your username or IP address has been automatically blocked by MediaWiki.
18:12:20    > The reason given is:
18:12:20    > </p>
18:12:20    > <dl><dd><em>Anonymous contributions are not allowed from your IP address (172.16.2.221). Please log in.</em></dd></dl>
18:12:20    > <ul><li>Start of block: 18:12, 7 November 2018</li>
18:12:20    > <li>Expiration of block: infinite</li>
18:12:20    > <li>Intended blockee: 172.16.2.221</li></ul>
18:12:20    > <p>Your current IP address is 172.16.2.221.
18:12:20    > Please include all above details in any queries you make.
18:12:20    > </p>}}, {name=systemblockedtext, parameters=[[[User:MediaWiki

This worked before. Looks like the IPs are coming from new eqiad-r cluster, and unlike the old ones, which were 10.*, they are not allowed to edit.

Smalyshev created this task.Nov 7 2018, 7:08 PM
Restricted Application added a project: Wikidata. · View Herald TranscriptNov 7 2018, 7:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
bd808 added subscribers: faidon, bd808.

The 10.68.0.0./16 range used by the legacy nova-network powered side of Cloud VPS is explictly allowed for some wikis in CommonSettings.php. We have not yet added a similar exemption for the new private address space that the Neutron powered eqiad1-r side of things is using. We need to either a) change the network routing so that the wikis see traffic as coming from our public IP space, or b) add an exemption for the 172.16.0.0/16 range we are now exposing. I think the preferred long term fix is (a), but we can quickly setup (b) to keep CI jobs working until we can make deeper routing changes.

I bit more digging has exposed that this is not a wide spread problem for all eqiad1-r users. The part of CommonSettings.php that allows anon edits from the 10.68.0.0/16 is gated by the $wmgAllowLabsAnonEdits feature flag. That flag is only set to true for wikis in the wikidataclient-test dblist which is currently testwiki, test2wiki, and testwikidatawiki. This feature flag was introduced in rOMWCda44b986783c: Set $wgSoftBlockRanges for T154698: Prevent contributions attributed to private and WMF IP addresses.

Change 472243 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/mediawiki-config@master] Allow Cloud VPS 172.16.0.0/16 for $wmgAllowLabsAnonEdits wikis

https://gerrit.wikimedia.org/r/472243

Change 472243 merged by jenkins-bot:
[operations/mediawiki-config@master] Allow Cloud VPS 172.16.0.0/16 for $wmgAllowLabsAnonEdits wikis

https://gerrit.wikimedia.org/r/472243

Mentioned in SAL (#wikimedia-operations) [2018-11-07T22:02:12Z] <thcipriani@deploy1001> Synchronized wmf-config/CommonSettings.php: [[gerrit:472243|Allow Cloud VPS 172.16.0.0/16 for $wmgAllowLabsAnonEdits wikis]] T208986 (duration: 00m 54s)

Smalyshev triaged this task as High priority.Nov 8 2018, 6:58 AM
Addshore moved this task from incoming to monitoring on the Wikidata board.Nov 8 2018, 11:59 AM
Smalyshev closed this task as Resolved.Nov 8 2018, 3:30 PM
Smalyshev claimed this task.

Looks like it's fine now.