Ok, I'm looking into this. Will document my debug steps.

1. Looked at raw data: kafkacat -C -b kafka-jumbo1001.eqiad.wmnet:9092 -t eventlogging-client-side | grep -i kaios | grep InukaPageView
2. Applied UA parser to some of the user agent strings found:

ADD JAR hdfs://analytics-hadoop/wmf/refinery/current/artifacts/org/wikimedia/analytics/refinery/refinery-hive-0.0.119.jar;
CREATE TEMPORARY FUNCTION ua_parser as 'org.wikimedia.analytics.refinery.hive.UAParserUDF';
select ua_parser('...');
-> "os_family":"KaiOS"

3. Applied the python version of UA parser to the same strings:

from ua_parser import user_agent_parser
user_agent_parser.Parse('...')
-> ... { os { family: u'KaiOS'

That's with latest code. But indeed if I do the query you do or look at eventlogging-valid-mixed or eventlogging_InukaPageView, I don't see the right os family in the user agent map. So maybe the code running somewhere is not the latest?

It looks like our python-ua-parser debian package deployed on eventlog1002 is old. I just had a quick go at repackaging it, but failed. It looks like some upstream stuff has changed enough that our debian/patches don't apply anymore.

But, overall, it kinda sucks that we have 2 separate sets of user agent parsing logic. In T238230: Decommission EventLogging backend components by migrating to MEP I'm currently working on getting Refine to parse EventLogging data the same way other data in Hadoop is parsed. The plan there is to slowly switch each topic over to the new system.

The right thing to do here would be to accelerate the migration of InukaPageView over to EventGate. We are waiting for some patches to be reviewed and merged and deployed before we can do this.

Or, we could take some time and rebuild the python-ua-parser package. This probably isn't THAT hard to do, I just wasn't able to do it in 5 minutes :p

What Andrew said. Also, as for the data, we keep the raw events as Varnish sees them for 90 days, so we have as much data as you have in event.inukapageview. Let us know if you need to re-parse past data, and I can backfill it for you (it's a bit of a pain, but if you need it, it's fine).

In T248560#6063901, @Milimetric wrote:

What Andrew said. Also, as for the data, we keep the raw events as Varnish sees them for 90 days, so we have as much data as you have in event.inukapageview. Let us know if you need to re-parse past data, and I can backfill it for you (it's a bit of a pain, but if you need it, it's fine).

Thanks for figuring this out, y'all! Don't worry, there's no need to backfill the old data 😊

@Ottomata let's please try updating the debian package

Done. Tested on eventlog1002 with

from ua_parser import user_agent_parser
user_agent_parser.Parse("Mozilla/5.0 (Mobile; ALCATEL4044T; rv:37.0) Gecko/37.0 Firefox/37.0 KaiOS/1.0")

Before:

{'user_agent': {'minor': '0', 'family': 'Firefox Mobile', 'major': '37', 'patch': None}, 'string': 'Mozilla/5.0 (Mobile; ALCATEL4044T; rv:37.0) Gecko/37.0 Firefox/37.0 KaiOS/1.0', 'device': {'family': 'Generic Smartphone', 'brand': 'Generic', 'model': 'Smartphone'}, 'os': {'minor': None, 'family': 'Firefox OS', 'major': None, 'patch_minor': None, 'patch': None}}

After:

{'user_agent': {'major': '37', 'minor': '0', 'patch': None, 'family': 'Firefox Mobile'}, 'string': 'Mozilla/5.0 (Mobile; ALCATEL4044T; rv:37.0) Gecko/37.0 Firefox/37.0 KaiOS/1.0', 'device': {'model': '4044T', 'brand': 'Alcatel', 'family': 'Alcatel 4044T'}, 'os': {'patch_minor': None, 'major': '1', 'minor': '0', 'patch': None, 'family': 'KaiOS'}}