Page MenuHomePhabricator

Pageview hourly data in Pivot is not showing up correctly
Closed, ResolvedPublic5 Story Points

Description

After restarting Pivot to pick up Marcel's banner_activity_minutely_sanitization_test cleanup we noticed that the pageviews hourly data is not showing up. Pivot started to log a weird patter of auto-discovery from May 2nd:

May  2 22:03:44 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-02T22:00:00.000Z)
May  2 22:04:39 thorium pivot[57590]: Cluster 'druid' could not introspect 'pageviews-hourly' because: null exception
May  2 22:04:45 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-02T22:02:00.000Z)
May  2 22:04:45 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-02T22:02:00.000Z)
May  2 22:05:45 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-02T22:04:00.000Z)
May  2 22:05:45 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-02T22:04:00.000Z)
May  2 22:06:39 thorium pivot[57590]: Cluster 'druid' could not introspect 'pageviews-hourly' because: null exception
May  2 22:06:45 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-02T22:06:00.000Z)
May  2 22:06:45 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-02T22:06:00.000Z)
May  2 22:07:45 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-02T22:07:00.000Z)
May  2 22:07:45 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-02T22:07:00.000Z)
[..]

After the last pivot restart we got also another error message:

May  5 13:07:35 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-05T13:07:00.000Z)
May  5 13:07:35 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-05T13:07:00.000Z)
May  5 13:07:54 thorium pivot[17209]: Cluster 'druid' could not introspect 'pageviews-hourly' because: null exception
May  5 13:07:54 thorium pivot[17209]: Cluster 'druid' has never seen 'pageviews-hourly' and will introspect 'pageviews-hourly'
[..]

Event Timeline

elukey created this task.May 5 2017, 2:03 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 5 2017, 2:03 PM

Mentioned in SAL (#wikimedia-operations) [2017-05-05T15:13:37Z] <elukey> increase nginx error log verbosity on mw2146 as test for T164586

Milimetric edited projects, added Analytics-Kanban; removed Analytics.
Milimetric set the point value for this task to 5.
Milimetric moved this task from Next Up to Done on the Analytics-Kanban board.
Nuria added a subscriber: Nuria.May 10 2017, 2:07 PM

This resolved by turning off the real time job, there were two granularities being written to the same place and druid did not like it

Nuria closed this task as Resolved.May 10 2017, 3:57 PM