Page MenuHomePhabricator

No banner impression data since 4 Feb
Closed, ResolvedPublic

Description

Same issue as T125676, don't know if it's the same cause again?

select * from pgehres.bannerimpressions order by timestamp desc limit 10; shows the last entries at 2016-02-04 20:47

Event Timeline

Pcoombe raised the priority of this task from to Unbreak Now!.
Pcoombe updated the task description. (Show Details)
Pcoombe added a project: Fundraising-Backlog.
Pcoombe added subscribers: Pcoombe, Jgreen, jrobell.
Pcoombe renamed this task from No banner impressions since 4 Feb to No banner impression data since 4 Feb.Feb 8 2016, 3:51 PM
Pcoombe set Security to None.

Amongst all the alert emails over the weekend I did see several that looked like failing to connections to silicon, Jeff probably knows more.

The banner history problem is the exact same cause as before, somehow I managed to revert the fix. I will fix this today and re-do the backfill.

Amongst all the alert emails over the weekend I did see several that looked like failing to connections to silicon, Jeff probably knows more.

This is unrelated, and from what I saw in email appeared to be caused by problems on worldpay's end.

re. backfilling, last logged line was: 2016-02-04T20:47:39

Hi @Jgreen, any ETA on this? The backfilling can wait, but we would like to have new impressions coming in before we start the Sweden campaign.

The configuration is fixed and the backfilling process has already
started, I think we'll be caught up in 8-12 hours.

To clarify, backfilling was a bad description on my part. What we do is
reset the log collector to the point in time right when the outage
started, so we re-pull logs from the analytics kafka log store. I don't
think we will lose any logs this time because I've got a better handle on
how to recover right were we broke, but it will take time to process the
backlog.

jg

Okay, thanks for the clarification.

Pcoombe claimed this task.

It seems to have caught up now. Thanks again!