Page MenuHomePhabricator

Requests to localhost spam the 'localhost' and 'xff' log buckets
Closed, ResolvedPublic

Description

POST requests having a X-Forwarded-For header are logged under the `xff' log bucket. That handling is done in wmf-config/CommonSettings.php as an entry in $wgExtensionFunctions.

The archived and compressed files are:

xff.log~ 2 GBytes
localhost.log600MBytes

The primary reason is the Jobrunner service enqueues jobs by hitting the web hosts directly on localhost which ends up causing a lot of spam such as:

bucketlog entry
localhost2016-03-15 09:40:37 mw1163 idwiktionary 1.27.0-wmf.16 localhost INFO: Tue, 15 Mar 2016 09:40:37 +0000 mw1163 , 127.0.0.1 http
xff2016-03-15 09:40:37 mw1163 idwiktionary 1.27.0-wmf.16 xff INFO: Tue, 15 Mar 2016 09:40:37 +0000 http://127.0.0.1:9005/rpc/RunJobs.php?wiki=idwiktionary&type=refreshLinksDynamic&maxtime=60&maxmem=300M , 127.0.0.1

A first cause of spam is the Nov 2013 commit 66af43f4483861f72ac2bca665bcbe31530d002c https://gerrit.wikimedia.org/r/#/c/93017/ . It logs WebRequest::detectProtocol() and I believe that was to debug / help with the HTTPS transition.

I dont think we need the localhost logbucket anymore.

For xff bucket, there is probably no need to log them when the request originate from 127.0.0.1.

Details

Related Gerrit Patches:
operations/mediawiki-config : masterStop logging xff from 127.0.0.1

Event Timeline

hashar created this task.Mar 15 2016, 9:48 AM
Restricted Application added subscribers: JEumerus, Matanya, Aklapper. · View Herald TranscriptMar 15 2016, 9:48 AM
fgiunchedi triaged this task as Medium priority.Apr 28 2016, 1:20 PM

Change 301339 had a related patch set uploaded (by Hashar):
Stop logging xff from 127.0.0.1

https://gerrit.wikimedia.org/r/301339

hashar claimed this task.Jul 27 2016, 8:53 AM

That came up again today with fluorine.eqiad.wmnet filling its disk (T141426).

From https://gerrit.wikimedia.org/r/301339 :

On fluorine, the xff.log archives are 5GBytes compressed and the localhost.log ones 3GBytes for a total of almost 1TBytes.

I have rebased the patch and added a few more reviewers.

hashar moved this task from Working on to To deploy on the Wikimedia-Site-requests board.

I am swamped in deployments this week, will revisit next week and probably formally announce the change then SWAT it.

From the Gerrit change:

I have added it to European SWAT window of Tuesday, September 13 at 13:00–14:00

Also sent a notification to the internal ops list.

Change 301339 merged by jenkins-bot:
Stop logging xff from 127.0.0.1

https://gerrit.wikimedia.org/r/301339

hashar closed this task as Resolved.Sep 13 2016, 1:33 PM

The spam traffic from the job runner is gone. There is still a lot of spam from internal box, but that was not the purpose of this task.

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:11 PM