Page MenuHomePhabricator

Increase maxUrlSize from 1000 to 1500
Closed, ResolvedPublic

Description

The NavigationTiming schema is currently producing urls of around 930-930 characters. I would not be surprised if we're already dropping a significant number of events on the client because of the limit.

Although at the moment mw.track entries for 'eventlogging.error' are not logged so we have no insight into this.

We recently added some properties to the schema (5d4824f41e) and this caused the metrics to go haywire and drop hitcount of the schema by 30%. The change has been reverted for now, but we'd like to be able to try again in the future. And as mentioned, we are most likely already dropping a fair number of events. Thus causing the data to be skewed towards users with faster devices/connections where the integers are smaller.

It seems we need to deal with several layers of infrastructure:

  • User browsers. The conservative limit there 2000 (http://stackoverflow.com/a/417184/319266).
  • Proxies (e.g. Varnish).
  • Server log tailers (eg. varnishlog). It seems shm_reclen is currently set to 1024 in production (09cdeb8b86) but it supports upto 65,535 so we should be able to increase it.
  • varnishncsa / ZMQ
  • varnishkafka / Kafka / Hadoop

Event Timeline

Krinkle created this task.Sep 9 2015, 8:28 PM
Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle added a subscriber: Krinkle.
Restricted Application added subscribers: Matanya, Aklapper. · View Herald TranscriptSep 9 2015, 8:28 PM

Change 237010 had a related patch set uploaded (by Krinkle):
Increase maxUrlSize to 1500

https://gerrit.wikimedia.org/r/237010

Krinkle triaged this task as High priority.Sep 9 2015, 8:29 PM
Krinkle removed a project: Patch-For-Review.
Krinkle set Security to None.
Krinkle updated the task description. (Show Details)
Krinkle added subscribers: BBlack, ori, Nuria.
BBlack added a comment.Sep 9 2015, 9:35 PM

Varnish is the primary issue here. Raising shm_reclen is non-trivial, especially to much-larger values. We've talked about it another ticket in the past (T91347#1249819 and onwards). There's a lot of scary low-level stuff going on with varnish shm log and memory limits, and we've crashed varnish before from getting sizes a little off here and there. I think we can work out how to raise it to 2048 safely pretty easily. 8192, a little scarier, but doable. Anything bigger like maxing it out at 64K and I worry the necessary resulting changes to other related buffer/workspace sizes could have serious performance or memory consumption consequences, too.

ori added a comment.Sep 9 2015, 9:40 PM

I think we can work out how to raise it to 2048 safely pretty easily.

2048 is plenty, especially since URLs longer than 2000 characters are liable to be problematic at other layers.

BBlack added a comment.EditedSep 21 2015, 10:09 PM

Note the Varnish change itself is merged and deployed to config files on the caches, but it will be a while before this is in full effect across the clusters, as I need to very slowly restart all of the varnish instances globally for it. Will update here once that process is complete.

The varnish change should be live on all production and beta varnishes now, raising the limit to ~2K there.

Change 237010 merged by Nuria:
Increase maxUrlSize to 2000

https://gerrit.wikimedia.org/r/237010

Change 240922 had a related patch set uploaded (by Krinkle):
Increase maxUrlSize to 2000

https://gerrit.wikimedia.org/r/240922

Change 240922 merged by jenkins-bot:
Increase maxUrlSize to 2000

https://gerrit.wikimedia.org/r/240922

ori closed this task as Resolved.Sep 25 2015, 1:48 AM
ori assigned this task to Krinkle.
Restricted Application added a project: Analytics. · View Herald TranscriptOct 30 2018, 12:26 AM