The NavigationTiming schema is currently producing urls of around 930-930 characters. I would not be surprised if we're already dropping a significant number of events on the client because of the limit.
Although at the moment mw.track entries for 'eventlogging.error' are not logged so we have no insight into this.
We recently added some properties to the schema (5d4824f41e) and this caused the metrics to go haywire and drop hitcount of the schema by 30%. The change has been reverted for now, but we'd like to be able to try again in the future. And as mentioned, we are most likely already dropping a fair number of events. Thus causing the data to be skewed towards users with faster devices/connections where the integers are smaller.
It seems we need to deal with several layers of infrastructure:
- User browsers. The conservative limit there 2000 (http://stackoverflow.com/a/417184/319266).
- Proxies (e.g. Varnish).
- Server log tailers (eg. varnishlog). It seems shm_reclen is currently set to 1024 in production (09cdeb8b86) but it supports upto 65,535 so we should be able to increase it.
- varnishncsa / ZMQ
- varnishkafka / Kafka / Hadoop