Page MenuHomePhabricator

Requests to localhost spam the 'localhost' and 'xff' log buckets
Closed, ResolvedPublic

Description

POST requests having a X-Forwarded-For header are logged under the `xff' log bucket. That handling is done in wmf-config/CommonSettings.php as an entry in $wgExtensionFunctions.

The archived and compressed files are:

xff.log~ 2 GBytes
localhost.log600MBytes

The primary reason is the Jobrunner service enqueues jobs by hitting the web hosts directly on localhost which ends up causing a lot of spam such as:

bucketlog entry
localhost2016-03-15 09:40:37 mw1163 idwiktionary 1.27.0-wmf.16 localhost INFO: Tue, 15 Mar 2016 09:40:37 +0000 mw1163 , 127.0.0.1 http
xff2016-03-15 09:40:37 mw1163 idwiktionary 1.27.0-wmf.16 xff INFO: Tue, 15 Mar 2016 09:40:37 +0000 http://127.0.0.1:9005/rpc/RunJobs.php?wiki=idwiktionary&type=refreshLinksDynamic&maxtime=60&maxmem=300M , 127.0.0.1

A first cause of spam is the Nov 2013 commit 66af43f4483861f72ac2bca665bcbe31530d002c https://gerrit.wikimedia.org/r/#/c/93017/ . It logs WebRequest::detectProtocol() and I believe that was to debug / help with the HTTPS transition.

I dont think we need the localhost logbucket anymore.

For xff bucket, there is probably no need to log them when the request originate from 127.0.0.1.

Event Timeline

hashar created this task.Mar 15 2016, 9:48 AM
Restricted Application added subscribers: JEumerus, Matanya, Aklapper. · View Herald TranscriptMar 15 2016, 9:48 AM
fgiunchedi triaged this task as Medium priority.Apr 28 2016, 1:20 PM

Change 301339 had a related patch set uploaded (by Hashar):
Stop logging xff from 127.0.0.1

https://gerrit.wikimedia.org/r/301339

hashar claimed this task.Jul 27 2016, 8:53 AM

That came up again today with fluorine.eqiad.wmnet filling its disk (T141426).

From https://gerrit.wikimedia.org/r/301339 :

On fluorine, the xff.log archives are 5GBytes compressed and the localhost.log ones 3GBytes for a total of almost 1TBytes.

I have rebased the patch and added a few more reviewers.

hashar moved this task from Working on to To deploy on the Wikimedia-Site-requests board.

I am swamped in deployments this week, will revisit next week and probably formally announce the change then SWAT it.

From the Gerrit change:

I have added it to European SWAT window of Tuesday, September 13 at 13:00–14:00

Also sent a notification to the internal ops list.

Change 301339 merged by jenkins-bot:
Stop logging xff from 127.0.0.1

https://gerrit.wikimedia.org/r/301339

hashar closed this task as Resolved.Sep 13 2016, 1:33 PM

The spam traffic from the job runner is gone. There is still a lot of spam from internal box, but that was not the purpose of this task.

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:11 PM