Page MenuHomePhabricator

[MEP] [BUG] Timestamp format changed in migrated server-side EventLogging schemas
Closed, ResolvedPublic

Description

From @nettrom_WMF:

Head’s up that the timestamp format in some schemas appears to have changed from e.g. 2021-02-24T00:57:39Z to e.g. 2021-02-25T00:54:21+0000
I’ve so far identified that this has happened with ServerSideAccountCreation, which is a server-side schema that was migrated last week (the format switch happens in the 14th hour of 2021-02-24, it looks the 15th partitions has the new format).

Run on https://superset.wikimedia.org/superset/sqllab:

WITH ranked_events AS (
  SELECT
    dt, client_dt, schema, meta.dt AS serverside_dt, meta.id, uuid,
    RANK() OVER (PARTITION BY hour ORDER BY dt DESC) AS rnk
  FROM serversideaccountcreation 
  WHERE year = 2021 AND month = 2 AND day = 24 AND hour IN(14, 15)
)
SELECT dt, client_dt, schema, serverside_dt, id, uuid
FROM ranked_events
WHERE rnk <= 5
dtclient_dtschemaserverside_dtiduuid
2021-02-24T14:59:53ZNAServerSideAccountCreationNANA9df7d97be5ee5d348dd67fa9e7f2be80
2021-02-24T14:59:52ZNAServerSideAccountCreationNANA6a29bf176edd58c99384f76e37c2ad03
2021-02-24T14:59:43ZNAServerSideAccountCreationNANA9b63480e51615ff9823cafcf8d74f5e9
2021-02-24T14:59:43ZNAServerSideAccountCreationNANA1fd61277dd505e91840a5351a5abeb65
2021-02-24T14:59:36ZNAServerSideAccountCreationNANA4e2f3ab655995fa6abe34726292d8178
2021-02-25T00:26:57+09002021-02-25T00:26:52+0900ServerSideAccountCreation2021-02-24T15:26:57.027Z6c0707b5-477d-458d-895b-f83ad059740bNA
2021-02-24T22:11:01+07002021-02-24T22:10:55+0700ServerSideAccountCreation2021-02-24T15:11:01.734Z2f0dbaa8-437b-40aa-a278-a4305f8155fdNA
2021-02-24T21:48:16+06002021-02-24T21:48:08+0600ServerSideAccountCreation2021-02-24T15:48:16.087Ze5dd5671-8c35-4fc1-bcfc-6e988d3ec075NA
2021-02-24T17:59:57+02002021-02-24T17:59:53+0200ServerSideAccountCreation2021-02-24T15:59:57.603Z556819ec-74ed-4bcc-a8a1-b0065a1a6bf6NA
2021-02-24T17:56:15+02002021-02-24T17:56:09+0200ServerSideAccountCreation2021-02-24T15:56:15.780Z35c8f4eb-fb34-4b3a-b761-244561225ee7NA

Event Timeline

mpopov moved this task from Triage to Tracking on the Product-Analytics board.

Oh that is a problem, I saw your original message Morten and thought ok so timezone is +0000 i guess that's ok, but if it is using varying timezones, that is not great!

Oh that is a problem, I saw your original message Morten and thought ok so timezone is +0000 i guess that's ok, but if it is using varying timezones, that is not great!

For the record, a format change from Z to +0000 is still not great. One of the biggest pain points when querying all our varying data sources is dealing with timestamps, whether formatting or data types or converting from VARCHAR to date/time.

Change 667888 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/extensions/EventLogging@master] Fix timestamp format for migrated events

https://gerrit.wikimedia.org/r/667888

Change 667813 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/extensions/EventLogging@wmf/1.36.0-wmf.33] Fix timestamp format for migrated events

https://gerrit.wikimedia.org/r/667813

Change 667814 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/extensions/EventLogging@wmf/1.36.0-wmf.32] Fix timestamp format for migrated events

https://gerrit.wikimedia.org/r/667814

Change 667888 merged by jenkins-bot:
[mediawiki/extensions/EventLogging@master] Fix timestamp format for migrated events

https://gerrit.wikimedia.org/r/667888

Change 667813 merged by jenkins-bot:
[mediawiki/extensions/EventLogging@wmf/1.36.0-wmf.33] Fix timestamp format for migrated events

https://gerrit.wikimedia.org/r/667813

Mentioned in SAL (#wikimedia-operations) [2021-03-02T18:21:27Z] <mholloway-shell@deploy1002> Synchronized php-1.36.0-wmf.33/extensions/EventLogging: Fix timestamp format for migrated events (T276235) (duration: 01m 09s)

Change 667814 merged by jenkins-bot:
[mediawiki/extensions/EventLogging@wmf/1.36.0-wmf.32] Fix timestamp format for migrated events

https://gerrit.wikimedia.org/r/667814

Mentioned in SAL (#wikimedia-operations) [2021-03-02T18:53:08Z] <mholloway-shell@deploy1002> Synchronized php-1.36.0-wmf.32/extensions/EventLogging: Fix timestamp format for migrated events (T276235) (duration: 01m 10s)

Verified that timestamps are now coming in with expected format and time zone.

hive (event)> select dt, client_dt, meta.dt as meta_dt from serversideaccountcreation where year = 2021 and month = 3 and day = 2 order by meta_dt desc limit 10;
<snip>
OK
dt	client_dt	meta_dt
2021-03-02T18:59:52Z	2021-03-02T18:59:48Z	2021-03-02T18:59:52.534Z
2021-03-02T18:59:44Z	2021-03-02T18:59:40Z	2021-03-02T18:59:44.410Z
2021-03-02T18:59:43Z	2021-03-02T18:59:41Z	2021-03-02T18:59:43.909Z
2021-03-02T18:59:25Z	2021-03-02T18:59:19Z	2021-03-02T18:59:25.190Z
2021-03-02T18:59:23Z	2021-03-02T18:59:20Z	2021-03-02T18:59:23.166Z
2021-03-02T18:59:21Z	2021-03-02T18:59:17Z	2021-03-02T18:59:21.948Z
2021-03-02T18:59:12Z	2021-03-02T18:59:08Z	2021-03-02T18:59:12.697Z
2021-03-02T18:59:10Z	2021-03-02T18:59:07Z	2021-03-02T18:59:10.122Z
2021-03-02T18:59:05Z	2021-03-02T18:59:03Z	2021-03-02T18:59:05.412Z
2021-03-02T18:58:54Z	2021-03-02T18:58:49Z	2021-03-02T18:58:54.788Z
Time taken: 44.371 seconds, Fetched: 10 row(s)
mpopov claimed this task.

Thank you, @Mholloway!