Eliminate PHP backend call for Special:RecordImpression
Closed, ResolvedPublic2 Estimated Story Points
Actions

Assigned To

Authored By

	awight
	Jul 22 2015, 10:22 PM

Description

@ori tells me that we already have a rule in Varnish to return HTTP 204 (no response) for any URL beginning with "/beacon/" (http://git.wikimedia.org/blob/operations%2Fpuppet.git/e8c64e206ac573cd2f8f078029258bd60179189d/modules%2Fvarnish%2Ftemplates%2Fvcl%2Fwikimedia.vcl.erb#L402)

Let's hit an URL under that path rather than Special:RecordImpression. This is a cheap code fix, and is a huge deal for app servers and varnish memory.

Choose URL pattern. //${wgServer}/beacon/impression?params...
Create a new log filter for that pattern.
Verify that CentralNotice does not freak out if S:RI returns a 204.
Make sure impression log processor will be able to find new files, named "beaconImpressions*"
Point CentralNotice bannerController at this new endpoint, using the $wgCentralBannerRecorder config.

When finished, verify that impressions are still logged through to the database.

Details

Subject	Repo	Branch	Lines +/-
Read banner impressions from new file	wikimedia/fundraising/tools/DjangoBannerStats	master	+6 -6
Point to a no-op /beacon URL rather than Special:RecordImpression	operations/mediawiki-config	master	+2 -2
erbium: add a log for beacon/impression	operations/puppet	production	+7 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		awight	T45250 Redo /beacon/impression system (formerly Special:RecordImpression) to remove extra round trips on all FR impressions (title was: S:RI should pyroperish)
		Resolved		awight	T106624 Eliminate PHP backend call for Special:RecordImpression

Event Timeline

awight created this task.Jul 22 2015, 10:22 PM

awight raised the priority of this task from to High.

awight updated the task description. (Show Details)

awight added projects: Epic, Performance Issue, Wikimedia-Fundraising, Varnish, MediaWiki-extensions-CentralNotice.

awight added subscribers: Aklapper, He7d3r, KTC and 23 others.

awight updated the task description. (Show Details)Jul 22 2015, 10:31 PM

awight set Security to None.

awight added a project: Fundraising Sprint The Pogues.Jul 22 2015, 11:54 PM

Pcoombe subscribed.Jul 23 2015, 10:10 AM

awight claimed this task.Jul 23 2015, 8:53 PM

awight moved this task from Backlog to Doing on the Fundraising Sprint The Pogues board.

awight edited a custom field.

Documenting the current system, at:
https://wikitech.wikimedia.org/wiki/Fundraising/tech/Banner_impressions

Wow, I think our pipeline is robust enough that we can just change the URL, and everything else will work. It would have to look like,
https://meta.wikmedia.org/beacon/Special:RecordImpression?params...

Change 226642 had a related patch set uploaded (by Awight):
Try pointing to a /beacon URL rather than Special:RecordImpression

https://gerrit.wikimedia.org/r/226642

gerritbot added a project: Patch-For-Review.Jul 23 2015, 9:18 PM

awight moved this task from Doing to Review on the Fundraising Sprint The Pogues board.Jul 23 2015, 9:22 PM

awight moved this task from Review to Pending Deployment on the Fundraising Sprint The Pogues board.

awight updated the task description. (Show Details)

Change 226654 had a related patch set uploaded (by Ori.livneh):
erbium: add a log for beacon/impression

https://gerrit.wikimedia.org/r/226654

Change 226654 merged by Ori.livneh:
erbium: add a log for beacon/impression

https://gerrit.wikimedia.org/r/226654

ori mentioned this in rOPUP10d2a5f8e33d: erbium: add a log for beacon/impression.Jul 23 2015, 11:50 PM

How about /beacon/impression instead of //{$wmfHostnames['meta']}/beacon/impression? Can the log match any domain? It would be great to remove one extra domain.

@Nemo_bis: great idea!

Looking great!!

When would be a likely deploy time (just so we can give FR a heads-up to keep an eye on impression numbers...)?

awight updated the task description. (Show Details)Jul 24 2015, 5:42 PM

awight updated the task description. (Show Details)

awight updated the task description. (Show Details)Jul 24 2015, 5:46 PM

Change 226642 merged by jenkins-bot:
Point to a no-op /beacon URL rather than Special:RecordImpression

https://gerrit.wikimedia.org/r/226642

ori mentioned this in rOMWC5082b2da1ce5: Point to a no-op /beacon URL rather than Special:RecordImpression.Jul 24 2015, 6:38 PM

Change 226766 had a related patch set uploaded (by Awight):
Read banner impressions from new file

https://gerrit.wikimedia.org/r/226766

The CentralNotice config change was deployed at 20150724 18:39.

ori Synchronized wmf-config/CommonSettings.php: Ib7c7861e: Point to a no-op /beacon URL rather than Special:RecordImpression (duration: 00m 12s)

Change 226766 merged by Ejegg:
Read banner impressions from new file

https://gerrit.wikimedia.org/r/226766

We'll need to occasionally trawl the bannerImpressions files to import the long tail of clients using the old URL. Not sure how we can do that, actually. The parsing code is really twitchy about filenames.