Page MenuHomePhabricator

Pageview hourly data in Pivot is not showing up correctly
Closed, ResolvedPublic5 Estimated Story Points

Description

After restarting Pivot to pick up Marcel's banner_activity_minutely_sanitization_test cleanup we noticed that the pageviews hourly data is not showing up. Pivot started to log a weird patter of auto-discovery from May 2nd:

May  2 22:03:44 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-02T22:00:00.000Z)
May  2 22:04:39 thorium pivot[57590]: Cluster 'druid' could not introspect 'pageviews-hourly' because: null exception
May  2 22:04:45 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-02T22:02:00.000Z)
May  2 22:04:45 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-02T22:02:00.000Z)
May  2 22:05:45 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-02T22:04:00.000Z)
May  2 22:05:45 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-02T22:04:00.000Z)
May  2 22:06:39 thorium pivot[57590]: Cluster 'druid' could not introspect 'pageviews-hourly' because: null exception
May  2 22:06:45 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-02T22:06:00.000Z)
May  2 22:06:45 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-02T22:06:00.000Z)
May  2 22:07:45 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-02T22:07:00.000Z)
May  2 22:07:45 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-02T22:07:00.000Z)
[..]

After the last pivot restart we got also another error message:

May  5 13:07:35 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-05T13:07:00.000Z)
May  5 13:07:35 thorium pivot[57590]: Got the latest time for 'pageviews-hourly' (2017-05-05T13:07:00.000Z)
May  5 13:07:54 thorium pivot[17209]: Cluster 'druid' could not introspect 'pageviews-hourly' because: null exception
May  5 13:07:54 thorium pivot[17209]: Cluster 'druid' has never seen 'pageviews-hourly' and will introspect 'pageviews-hourly'
[..]

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2017-05-05T15:13:37Z] <elukey> increase nginx error log verbosity on mw2146 as test for T164586

Milimetric edited projects, added Analytics-Kanban; removed Analytics.
Milimetric set the point value for this task to 5.
Milimetric moved this task from Next Up to Done on the Analytics-Kanban board.

This resolved by turning off the real time job, there were two granularities being written to the same place and druid did not like it