Page MenuHomePhabricator

Android debug builds event logging not recorded {oryx} [3 pts]
Closed, ResolvedPublic

Description

This is a request for troubleshooting help. When testing event logging, we use the following URL:

http://deployment.wikimedia.beta.wmflabs.org/beacon/event

And normally see the logs drop out here:

ssh deployment-eventlogging02.eqiad.wmflabs tail -f /var/log/eventlogging/all-events.log

Today, I don't see any logs being recorded. As I'm a newish developer, it's likely an issue ion my end. I also wondered if the recent HTTPS changes would have impacted this path.

Event Timeline

Niedzielski assigned this task to Milimetric.
Niedzielski raised the priority of this task from to Needs Triage.
Niedzielski updated the task description. (Show Details)
Niedzielski added a subscriber: Niedzielski.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 17 2015, 9:28 PM
Milimetric renamed this task from Android debug builds event logging not recorded to Android debug builds event logging not recorded {oryx} [3 pts].Jun 18 2015, 4:38 PM
Milimetric set Security to None.

@Niedzielski, this is a bug. The logs are not showing up because we've been actively playing with a different setup for EL in the beta cluster and things are unstable right now. I'm going to work on this and get it normal again. Until then, events seem to be making it through to the database. If you need help looking in there, let me know. Basically you ssh and do this:

mysql -ueventlogging -p68QrOq220717816UycU1 -Dlog

and then you can show tables to find your table and select from it. Use the index on the "timestamp" column as it's pretty fast.

@Milimetric, thanks! Any idea how I would know which tables we're dumping into? There's quite a few in the database.

@Milimetric, @Dbrant claried. He says the tables correspond to the schema revision ID.

@Milimetric, actually the database appears to not be getting events as confirmed by @Dbrant and myself.

@Niedzielski - if that's true even accounting for any sampling you might have, then it looks like there's still a problem. Other events seem to be getting through but only partially.

bearND added a subscriber: bearND.Jun 18 2015, 8:39 PM

I've updated Stephen's code to use the latest schema revision of MobileWikiAppArticleSuggestions (12443791), ran the app to make sure it produces some logs. There is no sampling for this funnel. I also see that's it's sending event from SuggestedPagesFunnel in logcat. But I don't see the new table created. I've waited at least 30 minutes already.

mysql> select * from MobileWikiAppArticleSuggestions_12443791;
ERROR 1146 (42S02): Table 'log.MobileWikiAppArticleSuggestions_12443791' doesn't exist

It could be that there's a schema validation issue but not sure if the system is in a state where we could see that.

I then went back to master, and created a few more log entries with the old schema, but still nothing. The last entry I see in the table MobileWikiAppArticleSuggestions_11448426 is 20150520194221, almost a month old.

yep, that's what we see too, we're actively debugging this now, we don't know what's going on.

@bearND & @Niedzielski - how do you generate events exactly? Andrew got the logs working again and I'd like to troubleshoot further.

@Milimetric, thanks for the update! I changed the event logging URL to "http://deployment.wikimedia.beta.wmflabs.org/beacon/event" and just poke around the app. I've attached an Android APK (

) for a build with this change. I don't see any new events in the log file or database (I only checked one table). I'm happy to guinea pig on my end, just ping me on IRC if there's something I can do to help.

All - Andrew figured this out. varnisncsa was not running in beta, so none of the client events were making it through. He restarted it and I definitely see events coming in now. Please try again and let us know if there are any problems.

Milimetric moved this task from Ready to Deploy to Done on the Analytics-Kanban board.

@Milimetric, great news! I confirm that some events appear to be logging properly! I'm not sure this is yet working for new table events. I added a new event revision a couple days ago, MobileWikiAppArticleSuggestions_12443791, and do not see that table being added. Any idea if this is on the backend?

ggellerman moved this task from Done to In Progress on the Analytics-Kanban board.Jun 22 2015, 3:31 PM

@Niedzielski: I'm having a hard time generating proper events that then don't go through the pipeline. Do you think you could set up a meeting with me and we can try debugging together?

Milimetric closed this task as Resolved.Jun 23 2015, 7:35 PM

Thanks for the debug session, it turned out events were invalid but I was looking in the wrong logs. FYI in case someone's learning from this:

/var/log/upstart/eventlogging_processor-client-side-events.log has logs of invalid events along with the error that caused the validation to fail.

Milimetric moved this task from Paused to Done on the Analytics-Kanban board.Jun 23 2015, 7:36 PM