Page MenuHomePhabricator

KaiOS / Inuka Event Platform client
Closed, ResolvedPublic

Description

In order to complete T259163: Migrate legacy metawiki schemas to Event Platform, all clients that currently send EventLogging events need to be modified to send to Event Gate instead, using Event Platform schemas.

There are 3 schemas to migrate:

We've got client support for posting Event Platform events from MediaWiki in EventLogging, and from Android and iOS.

We'll need this support in KaiOS app in order to migrate these event schemas to Event Platform.

Event Timeline

Ottomata renamed this task from KaiOS / Inuka Event Platform client migration to KaiOS / Inuka Event Platform client.Jan 28 2021, 7:38 PM
Ottomata created this task.

Hm, perhaps I am slightly misinformed about how this works. ext:WikimediaEvents has code to support InukaPageView that uses EventLogging. Does the KaiOS app also submit events?

Hm, perhaps I am slightly misinformed about how this works. ext:WikimediaEvents has code to support InukaPageView that uses EventLogging. Does the KaiOS app also submit events?

Sending InukaPageView events from WikimediaEvents will be removed in T265921 but I don't know when that will happen.

The KaiOS app will continue sending InukaPageView events so yes it needs to be migrated. If I'm not mistaken, the app is already sending client-side errors via the new system (code) so it can be generalized for other schemas.

Nice! Interesting, the KaiOS app is JS? Cool.

@jlinehan maybe we should one day consider having language specific client libraries (JS, PHP, Java, etc.) , rather than app specific (MW, Android, iOS, KaiOS).

@jlinehan and I discussed this last quarter when I was getting started on productionizing the Java/Android library. I think our consensus was that orienting the libraries toward language and making them client-agnostic would be better in principle, but in practice the point was kind of moot since there was essentially a 1:1 relationship between languages and target clients for the libraries anyway. That's certainly the case for the Swift/iOS client library (and by that point the decision had already been made to embed that client into the app code itself, so by writing a generic Java client we'd have been introducing new inconsistency). Are there any other Java apps or services around here that might realistically want to make use of a generic Java event platform client library? I guess there is some analytics pipeline stuff, and WDQS, and maybe some search stuff? The JS client would probably be the primary candidiate for breaking out into a standalone library, though, I think (at least from a Product-centric viewpoint).

Are there any other Java apps or services around here that might realistically want to make use of a generic Java event platform client library?

Yes, but they would more likely be used to produce to Kafka directly, rather than POSTing to EventGate.

The JS client would probably be the primary candidiate for breaking out into a standalone library, though, I think (at least from a Product-centric viewpoint)

Heh, I guess someone could make an API request to ResourceLoader to just embed the EventLogging JS code. :p (JK)

Nice! Interesting, the KaiOS app is JS? Cool.

@jlinehan maybe we should one day consider having language specific client libraries (JS, PHP, Java, etc.) , rather than app specific (MW, Android, iOS, KaiOS).

Ya, the way we had broken down the architecture was to have a language-specific core and a runtime-specific integration layer, but (as @Mholloway points out) we didn't impose that up-front at least while the apps are in beta. I think there would be benefits to this approach. The main goal is to be able to have behavior conform to a common spec, to make maintenance and documentation easier, plus behavioral guarantees that are important for statistics and performance and compatibility and stuff. I think those things are helped by not having an app-specific approach. Right now we could go to a language-specific approach if we wanted, we just haven't, because it could have been overkill to start (while there was a 1:1 relationship b/t language:apps) Perhaps we should though.

the way we had broken down the architecture was to have a language-specific core and a runtime-specific integration layer

Ah, yes, good point, I forgot to mention this. I maintained this pattern in my Android integration work last Q. I don't think it survived the iOS integration in that version (or didn't after I was finished with the requested refinements last Q, in any event), but again the Swift library is highly unlikely ever to be shared anyway.

@SBisson any tips for getting this work scheduled? :)

@Ottomata from the app's perspective, changing the destination URL and the shape of the message for all 3 schema is almost trivial so we can squeeze that in any time. Before we do that, is there something that needs to be done in the backend so it's ready to receive those messages?

Yes, we need to migrate the schemas and declare the streams. Basically Steps 1 - 6 in the Migration Plan in T259163: Migrate legacy metawiki schemas to Event Platform.

We can do that now, but we'll want to edit protect the schemas on metawiki so there aren't new changes while we migrate. If you aren't planning on making any schema changes, we can do that anytime I guess. Thanks!

There's one last change @nshahquinn-wmf would like to make to the InukaPageView schema before it gets protected but you can go ahead with the migration for the other 2 and let us know when it's time to change the app.

Ok, awesome. If alright with you we'll just wait for all the schemas to be settled before proceeding (so we can try do them all together). We are also a little backlogged with dealing with the small fallouts from Hadoop upgrade last week, so are behind on EventLogging migration anyway.

@Ottomata Sure, will let you know when all 3 are settled.

@Ottomata Sure, will let you know when all 3 are settled.

@Ottomata I've made the one schema change we had in mind (T276475), so feel free to protect the schemas and go ahead with the migration. I'll update this ticket if anything changes on our end.

Thanks, if I don't get to this this week, will do the next.

Ok @SBisson we are ready to go. Steps 1-6 done. You should be able to produce events to https://intake-analytics.wikimedia.org/v1/events?hasty=true. Let me know if you need any assistance!

Hm, Q. Does the KaiOS app work like other mobile apps where users have to upgrade the app to get the new code? If so, how long do we have to wait before we can disable accepting events via the deprecated /beacon/event endpoint?

@Ottomata the vast majority of users are getting auto updates. For the others, we'll see how their numbers decline once we have the migrated version out and we should be able to forecast when they'll be down to virtually zero.

@Ottomata what is the hasty=true param? Should we always have it in the intake URL?

@SBisson since the new logging code is already working well in production (T283768), can we close this?