Page MenuHomePhabricator

Camus should look for multiple possible timestamp fields to use for hourly partitioining
Closed, ResolvedPublic

Description

Hm, am trying to solve the dt. vs meta.dt client timestamp during migration issue with https://gerrit.wikimedia.org/r/c/operations/puppet/+/593610/7/modules/profile/manifests/analytics/refinery/job/camus.pp#b119 by making a new Camus job just for EventLogging that we will switch each legacy event stream to after it is fully migrated to EventGate. On more thought, this would work, but would require a manual and careful Refine during the hour that the job is switched to the new Camus job, as there would probably be data in two locations that would need to end up Refined into the table.

A smoother way would be to make Camus take a list of possible timestamp fields from which to partition. If the first is not found, the second would be used. We can then use the same Camus job configured to look for first meta.dt and then dt if meta.dt is not found. This would allow the same Camus job and raw directory to be used throughout the migration.

Event Timeline

Ottomata created this task.Thu, Jun 25, 1:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptThu, Jun 25, 1:29 PM

Change 607796 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/camus@wmf] Make JsonStringMessageDecoder search for list of possible camus.message.timestamp.field

https://gerrit.wikimedia.org/r/607796

Ottomata moved this task from Incoming to Event Platform on the Analytics board.Mon, Jun 29, 3:48 PM

Change 607796 merged by Ottomata:
[analytics/camus@wmf] Make JsonStringMessageDecoder search for list of possible camus.message.timestamp.field

https://gerrit.wikimedia.org/r/c/analytics/camus/ /607796

Change 608460 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery@master] Deploy camus-wmf-0.1.0-wmf10.jar

https://gerrit.wikimedia.org/r/c/analytics/refinery/ /608460

Change 608622 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Camus eventlogging - consider meta.dt and dt for event partition time

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608622

Change 608460 merged by Ottomata:
[analytics/refinery@master] Deploy camus-wmf-0.1.0-wmf10.jar

https://gerrit.wikimedia.org/r/c/analytics/refinery/ /608460

Mentioned in SAL (#wikimedia-operations) [2020-06-30T14:08:57Z] <otto@deploy1001> Started deploy [analytics/refinery@d63944e]: Deploying new camus wmf10 jar to an-launcher1002 for T256370

Mentioned in SAL (#wikimedia-operations) [2020-06-30T14:10:54Z] <otto@deploy1001> Finished deploy [analytics/refinery@d63944e]: Deploying new camus wmf10 jar to an-launcher1002 for T256370 (duration: 01m 56s)

Mentioned in SAL (#wikimedia-operations) [2020-06-30T14:47:36Z] <otto@deploy1001> Started deploy [analytics/refinery@d63944e]: Deploying new camus wmf10 jar to an-launcher1002 for T256370 - take 2

Mentioned in SAL (#wikimedia-operations) [2020-06-30T14:47:40Z] <otto@deploy1001> Finished deploy [analytics/refinery@d63944e]: Deploying new camus wmf10 jar to an-launcher1002 for T256370 - take 2 (duration: 00m 03s)

Mentioned in SAL (#wikimedia-operations) [2020-06-30T14:49:33Z] <otto@deploy1001> Started deploy [analytics/refinery@d63944e]: Deploying new camus wmf10 jar to an-launcher1002 for T256370 - take 3

Mentioned in SAL (#wikimedia-operations) [2020-06-30T14:49:37Z] <otto@deploy1001> Finished deploy [analytics/refinery@d63944e]: Deploying new camus wmf10 jar to an-launcher1002 for T256370 - take 3 (duration: 00m 03s)

Mentioned in SAL (#wikimedia-operations) [2020-06-30T15:54:01Z] <otto@deploy1001> Started deploy [analytics/refinery@d63944e]: Deploying new camus wmf10 jar to an-launcher1002 for T256370 - take 3

Mentioned in SAL (#wikimedia-operations) [2020-06-30T15:54:05Z] <otto@deploy1001> Finished deploy [analytics/refinery@d63944e]: Deploying new camus wmf10 jar to an-launcher1002 for T256370 - take 3 (duration: 00m 03s)

Change 608622 merged by Ottomata:
[operations/puppet@production] Camus eventlogging - consider meta.dt and dt for event partition time

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608622

Ottomata moved this task from Next Up to Done on the Analytics-Kanban board.Tue, Jun 30, 7:17 PM
Nuria closed this task as Resolved.Mon, Jul 6, 10:42 PM
Nuria set Final Story Points to 8.