| • Niedzielski | |
| Jun 17 2015, 9:28 PM |
| F181178: wikipedia-dev-debug.apk | |
| Jun 19 2015, 8:21 PM |
This is a request for troubleshooting help. When testing event logging, we use the following URL:
And normally see the logs drop out here:
ssh deployment-eventlogging02.eqiad.wmflabs tail -f /var/log/eventlogging/all-events.log
Today, I don't see any logs being recorded. As I'm a newish developer, it's likely an issue ion my end. I also wondered if the recent HTTPS changes would have impacted this path.
@Niedzielski, this is a bug. The logs are not showing up because we've been actively playing with a different setup for EL in the beta cluster and things are unstable right now. I'm going to work on this and get it normal again. Until then, events seem to be making it through to the database. If you need help looking in there, let me know. Basically you ssh and do this:
mysql -ueventlogging -p68QrOq220717816UycU1 -Dlog
and then you can show tables to find your table and select from it. Use the index on the "timestamp" column as it's pretty fast.
@Milimetric, thanks! Any idea how I would know which tables we're dumping into? There's quite a few in the database.
@Milimetric, @Dbrant claried. He says the tables correspond to the schema revision ID.
@Milimetric, actually the database appears to not be getting events as confirmed by @Dbrant and myself.
@Niedzielski - if that's true even accounting for any sampling you might have, then it looks like there's still a problem. Other events seem to be getting through but only partially.
I've updated Stephen's code to use the latest schema revision of MobileWikiAppArticleSuggestions (12443791), ran the app to make sure it produces some logs. There is no sampling for this funnel. I also see that's it's sending event from SuggestedPagesFunnel in logcat. But I don't see the new table created. I've waited at least 30 minutes already.
mysql> select * from MobileWikiAppArticleSuggestions_12443791;
ERROR 1146 (42S02): Table 'log.MobileWikiAppArticleSuggestions_12443791' doesn't exist
It could be that there's a schema validation issue but not sure if the system is in a state where we could see that.
I then went back to master, and created a few more log entries with the old schema, but still nothing. The last entry I see in the table MobileWikiAppArticleSuggestions_11448426 is 20150520194221, almost a month old.
yep, that's what we see too, we're actively debugging this now, we don't know what's going on.
@bearND & @Niedzielski - how do you generate events exactly? Andrew got the logs working again and I'd like to troubleshoot further.
@Milimetric, thanks for the update! I changed the event logging URL to "http://deployment.wikimedia.beta.wmflabs.org/beacon/event" and just poke around the app. I've attached an Android APK (
) for a build with this change. I don't see any new events in the log file or database (I only checked one table). I'm happy to guinea pig on my end, just ping me on IRC if there's something I can do to help.All - Andrew figured this out. varnisncsa was not running in beta, so none of the client events were making it through. He restarted it and I definitely see events coming in now. Please try again and let us know if there are any problems.
@Milimetric, great news! I confirm that some events appear to be logging properly! I'm not sure this is yet working for new table events. I added a new event revision a couple days ago, MobileWikiAppArticleSuggestions_12443791, and do not see that table being added. Any idea if this is on the backend?
@Niedzielski: I'm having a hard time generating proper events that then don't go through the pipeline. Do you think you could set up a meeting with me and we can try debugging together?
Thanks for the debug session, it turned out events were invalid but I was looking in the wrong logs. FYI in case someone's learning from this:
/var/log/upstart/eventlogging_processor-client-side-events.log has logs of invalid events along with the error that caused the validation to fail.