Page MenuHomePhabricator

ServerTiming schema value for duration is 0
Closed, ResolvedPublic

Description

Our current process for refining data from EventLogging into Hadoop infers the type for each field instead of looking at the schema. As we discussed in the issue with NavigationTiming data, we're working on a longer term fix for this. But in the meantime we audited where JSON schemas differ from Hive schemas and found that the ServerTiming schema wants duration to be a number, but usually sends a value of 0. Is this field supposed to be a double, is it always an integer, is there a bug in instrumentation? Let us know, we can alter the Hive table and re-import if necessary to recover any data truncated.

Event Timeline

Milimetric triaged this task as Normal priority.
Milimetric moved this task from Incoming to Data Quality on the Analytics board.
Milimetric removed a project: Analytics-Kanban.
Krinkle reassigned this task from Krinkle to Gilles.Mar 3 2019, 2:04 AM
Krinkle added subscribers: Gilles, Krinkle.

This is presumably similar to the issue with NavigationTiming/deviceMemory from T214384 (which has since been resolved). And we need to decide whether Analytics will apply the same fix to ServerTiming/duration.

I suppose it's harmless to do, but I don't know whether fractions can/should actually be used for this as well. The unit is in milliseconds, and the spec (and MDN doc) says its parsed as a double from the corresponding HTTP header, so it certainly can have a fraction.

But, depending on whether it does in our case, we may not need to repair/backfill any old data like we did for T214384. Re-assigning to @Gilles to confirm/decide.

Gilles added a comment.Mar 4 2019, 9:06 AM

Currently we're only using Server-Timing to pass Varnish caching information, which doesn't use the duration field.

Server-Timing is a freeform thing we can use. The spec that it's a double: https://w3c.github.io/server-timing/#duration-attribute but I doesn't state that the unit is milliseconds. It can be whatever we want, and doesn't have to be a "duration" per se. Server-Timing is designed to allow us to pass anything we want.

We can certainly multiply and round what's passed, for the sake of consistency with other performance APIs and to turn a double into an integer. It's a good time to make that change, since the "duration" field hasn't been used yet.

As for Hive storage, let's consider that this is always an integer. I'll make the necessary change in NavigationTiming for future use (multiply by 1000 and round).

Change 494179 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/extensions/NavigationTiming@master] Assume that Server Timing duration is expressed in ms

https://gerrit.wikimedia.org/r/494179

Change 494179 merged by jenkins-bot:
[mediawiki/extensions/NavigationTiming@master] Assume that Server Timing duration is expressed in ms

https://gerrit.wikimedia.org/r/494179

kchapman moved this task from Inbox to Doing on the Performance-Team board.Mar 4 2019, 9:08 PM
Gilles added a comment.Mar 5 2019, 8:56 AM

@Milimetric is the data for "duration" in Hive currently stored as integer?

@Gilles yes, bigint:

pageviewToken: string
description: string
duration: bigint
name: string
Gilles closed this task as Resolved.Mar 5 2019, 2:13 PM