Page MenuHomePhabricator

Eliminate PHP backend call for Special:RecordImpression
Closed, ResolvedPublic2 Estimate Story Points

Description

@ori tells me that we already have a rule in Varnish to return HTTP 204 (no response) for any URL beginning with "/beacon/" (http://git.wikimedia.org/blob/operations%2Fpuppet.git/e8c64e206ac573cd2f8f078029258bd60179189d/modules%2Fvarnish%2Ftemplates%2Fvcl%2Fwikimedia.vcl.erb#L402)

Let's hit an URL under that path rather than Special:RecordImpression. This is a cheap code fix, and is a huge deal for app servers and varnish memory.

  • Choose URL pattern. //${wgServer}/beacon/impression?params...
  • Create a new log filter for that pattern.
  • Verify that CentralNotice does not freak out if S:RI returns a 204.
  • Make sure impression log processor will be able to find new files, named "beaconImpressions*"
  • Point CentralNotice bannerController at this new endpoint, using the $wgCentralBannerRecorder config.
  • When finished, verify that impressions are still logged through to the database.

Details

Related Gerrit Patches:
wikimedia/fundraising/tools/DjangoBannerStats : masterRead banner impressions from new file
operations/mediawiki-config : masterPoint to a no-op /beacon URL rather than Special:RecordImpression
operations/puppet : productionerbium: add a log for beacon/impression

Event Timeline

awight created this task.Jul 22 2015, 10:22 PM
awight raised the priority of this task from to High.
awight updated the task description. (Show Details)
awight added subscribers: Aklapper, He7d3r, KTC and 23 others.
awight updated the task description. (Show Details)Jul 22 2015, 10:31 PM
awight set Security to None.
awight claimed this task.Jul 23 2015, 8:53 PM
awight moved this task from Backlog to Doing on the Fundraising Sprint The Pogues board.
awight edited a custom field.

Wow, I think our pipeline is robust enough that we can just change the URL, and everything else will work. It would have to look like,
https://meta.wikmedia.org/beacon/Special:RecordImpression?params...

Change 226642 had a related patch set uploaded (by Awight):
Try pointing to a /beacon URL rather than Special:RecordImpression

https://gerrit.wikimedia.org/r/226642

awight moved this task from Review to Pending Deployment on the Fundraising Sprint The Pogues board.
awight updated the task description. (Show Details)

Change 226654 had a related patch set uploaded (by Ori.livneh):
erbium: add a log for beacon/impression

https://gerrit.wikimedia.org/r/226654

Change 226654 merged by Ori.livneh:
erbium: add a log for beacon/impression

https://gerrit.wikimedia.org/r/226654

How about /beacon/impression instead of //{$wmfHostnames['meta']}/beacon/impression? Can the log match any domain? It would be great to remove one extra domain.

Looking great!!

When would be a likely deploy time (just so we can give FR a heads-up to keep an eye on impression numbers...)?

awight updated the task description. (Show Details)Jul 24 2015, 5:42 PM
awight updated the task description. (Show Details)
awight updated the task description. (Show Details)Jul 24 2015, 5:46 PM

Change 226642 merged by jenkins-bot:
Point to a no-op /beacon URL rather than Special:RecordImpression

https://gerrit.wikimedia.org/r/226642

Change 226766 had a related patch set uploaded (by Awight):
Read banner impressions from new file

https://gerrit.wikimedia.org/r/226766

The CentralNotice config change was deployed at 20150724 18:39.

ori Synchronized wmf-config/CommonSettings.php: Ib7c7861e: Point to a no-op /beacon URL rather than Special:RecordImpression (duration: 00m 12s)

Change 226766 merged by Ejegg:
Read banner impressions from new file

https://gerrit.wikimedia.org/r/226766

We'll need to occasionally trawl the bannerImpressions files to import the long tail of clients using the old URL. Not sure how we can do that, actually. The parsing code is really twitchy about filenames.

Confirmed that the database is populated with counts from the new files.

awight closed this task as Resolved.Jul 24 2015, 8:32 PM
awight updated the task description. (Show Details)
Nemo_bis awarded a token.
Restricted Application added a project: Operations. · View Herald TranscriptMay 4 2016, 9:14 AM
mmodell removed a subscriber: awight.Jun 22 2017, 9:52 PM