Page MenuHomePhabricator

No maps tile requests in webrequests table as of 1 June 2017
Closed, ResolvedPublic

Description

Our counts of map tiles served disappeared as of 1 June 2017 (query we use). In fact, running the following simple query shows that there are no tile requests after 31 May 2017:

SELECT
  TO_DATE(ts) AS `date`,
  COUNT(1) AS requests
FROM wmf.webrequest
WHERE
  webrequest_source = 'maps'
  AND year = 2017 AND (
    (month = 5 AND day = 31)
    OR (month = 6 AND day = 1)
  )
  AND http_status = '200'
  AND uri_path RLIKE '^/([^/]+)/([0-9]{1,2})/(-?[0-9]+)/(-?[0-9]+)(@([0-9]\.?[0-9]?)x)?\.([a-z]+)$'
  AND uri_query <> '?loadtesting'
GROUP BY TO_DATE(ts);

Event Timeline

mpopov created this task.Jun 6 2017, 1:35 AM
Restricted Application added subscribers: Jay8g, TerraCodes, Aklapper. · View Herald TranscriptJun 6 2017, 1:35 AM
Gehel added a comment.Jun 6 2017, 12:44 PM

This looks like it might be related to T164608#3248269. It looks like the webrequest_source has changed to "upload" and that some differentiation has to be done on the hostname (maps.wikimedia.org). I'm not familiar with how the data is stored, but hopefully, this will point you in the correct direction. If not, we can dig into this together...

Nuria added a comment.Jun 6 2017, 2:15 PM

Correct, webrequest_source="maps" no longer exists.

debt added a comment.Jun 6 2017, 3:18 PM

@Nuria - how can we get this fixed? Is the data for the last 6-ish days lost? @MaxSem - is this something you can help us with?

Gehel added a comment.Jun 6 2017, 3:49 PM

My guess is that @mpopov probably knows enough to fix that request (now that he knows what has changed). If it is not the case, I'm pretty sure that with a few minutes between @mpopov and myself, we should find a way...

Change 357440 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/golden@master] Update partition info for maps tiles

https://gerrit.wikimedia.org/r/357440

Change 357440 merged by Bearloga:
[wikimedia/discovery/golden@master] Update partition info for maps tiles

https://gerrit.wikimedia.org/r/357440

mpopov closed this task as Resolved.Jun 6 2017, 7:22 PM
mpopov claimed this task.

Thanks for finding the source of the problem, @Gehel!

@debt: the tile counts for june will be backfilled

debt moved this task from Backlog to Done on the Maps-Sprint board.Jun 6 2017, 7:34 PM

Woohoo! Thanks, @mpopov and @Gehel !

Change 373628 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/golden@master] metrics::maps::users_by_country: Fix wmf.webrequest partition

https://gerrit.wikimedia.org/r/373628

Change 373628 merged by Bearloga:
[wikimedia/discovery/golden@master] metrics::maps::users_by_country: Fix wmf.webrequest partition

https://gerrit.wikimedia.org/r/373628