Maniphest T208986

WDQS tests can no longer edit test.wikidata.org
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Smalyshev
	Nov 7 2018, 7:08 PM

Description

CI jobs for WDQS now fail with:

18:12:20    > Throwable #1: org.wikidata.query.rdf.tool.exception.RetryableException: Error result from Mediawiki:  {code=permissiondenied, info=You do not have the permissions needed to carry out this action., messages=[{name=wikibase-api-permissiondenied, parameters=[], html={*=You do not have the permissions needed to carry out this action.}}, {name=systemblockedtext, parameters=[[[User:MediaWiki default|MediaWiki default]], Anonymous contributions are not allowed from your IP address (172.16.2.221). Please log in., 172.16.2.221, MediaWiki default, wgSoftBlockRanges, infinite, 172.16.2.221, 18:12, 7 November 2018], html={*=<p>Your username or IP address has been automatically blocked by MediaWiki.
18:12:20    > The reason given is:
18:12:20    > </p>
18:12:20    > <dl><dd><em>Anonymous contributions are not allowed from your IP address (172.16.2.221). Please log in.</em></dd></dl>
18:12:20    > <ul><li>Start of block: 18:12, 7 November 2018</li>
18:12:20    > <li>Expiration of block: infinite</li>
18:12:20    > <li>Intended blockee: 172.16.2.221</li></ul>
18:12:20    > <p>Your current IP address is 172.16.2.221.
18:12:20    > Please include all above details in any queries you make.
18:12:20    > </p>}}, {name=systemblockedtext, parameters=[[[User:MediaWiki

This worked before. Looks like the IPs are coming from new eqiad-r cluster, and unlike the old ones, which were 10.*, they are not allowed to edit.

Details

	Subject	Repo	Branch	Lines +/-
	Allow Cloud VPS 172.16.0.0/16 for $wmgAllowLabsAnonEdits wikis	operations/mediawiki-config	master	+12 -5

Customize query in gerrit

Related Objects

Mentioned In: T214313: Add new Tool Labs IPs to Varnish rate limit whitelist
T209011: Change routing to ensure that traffic originating from Cloud VPS is seen as non-private IPs by Wikimedia wikis
Mentioned Here: rOMWCda44b986783c: Set $wgSoftBlockRanges
T154698: Prevent contributions attributed to private and WMF IP addresses

Event Timeline

Smalyshev created this task.Nov 7 2018, 7:08 PM

Restricted Application added a project: Wikidata. · View Herald TranscriptNov 7 2018, 7:08 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Smalyshev added projects: SRE, Cloud-Services.Nov 7 2018, 7:10 PM

The 10.68.0.0./16 range used by the legacy nova-network powered side of Cloud VPS is explictly allowed for some wikis in CommonSettings.php. We have not yet added a similar exemption for the new private address space that the Neutron powered eqiad1-r side of things is using. We need to either a) change the network routing so that the wikis see traffic as coming from our public IP space, or b) add an exemption for the 172.16.0.0/16 range we are now exposing. I think the preferred long term fix is (a), but we can quickly setup (b) to keep CI jobs working until we can make deeper routing changes.

I bit more digging has exposed that this is not a wide spread problem for all eqiad1-r users. The part of CommonSettings.php that allows anon edits from the 10.68.0.0/16 is gated by the $wmgAllowLabsAnonEdits feature flag. That flag is only set to true for wikis in the wikidataclient-test dblist which is currently testwiki, test2wiki, and testwikidatawiki. This feature flag was introduced in rOMWCda44b986783c: Set $wgSoftBlockRanges for T154698: Prevent contributions attributed to private and WMF IP addresses.

Change 472243 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/mediawiki-config@master] Allow Cloud VPS 172.16.0.0/16 for $wmgAllowLabsAnonEdits wikis

https://gerrit.wikimedia.org/r/472243

gerritbot added a project: Patch-For-Review.Nov 7 2018, 8:08 PM

Change 472243 merged by jenkins-bot:
[operations/mediawiki-config@master] Allow Cloud VPS 172.16.0.0/16 for $wmgAllowLabsAnonEdits wikis

https://gerrit.wikimedia.org/r/472243

Mentioned in SAL (#wikimedia-operations) [2018-11-07T22:02:12Z] <thcipriani@deploy1001> Synchronized wmf-config/CommonSettings.php: [[gerrit:472243|Allow Cloud VPS 172.16.0.0/16 for $wmgAllowLabsAnonEdits wikis]] T208986 (duration: 00m 54s)

bd808 mentioned this in T209011: Change routing to ensure that traffic originating from Cloud VPS is seen as non-private IPs by Wikimedia wikis.Nov 7 2018, 11:46 PM

Smalyshev triaged this task as High priority.Nov 8 2018, 6:58 AM

Addshore moved this task from incoming to monitoring on the Wikidata board.Nov 8 2018, 11:59 AM

Looks like it's fine now.

faidon mentioned this in T214313: Add new Tool Labs IPs to Varnish rate limit whitelist.Jan 21 2019, 8:07 PM

WDQS tests can no longer edit test.wikidata.orgClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

WDQS tests can no longer edit test.wikidata.org
Closed, ResolvedPublic
Actions