@jlinehan, is there a way to fashion the client error logging from MediaWiki in a standalone fashion for applications like the KaiOS (HTML5 web app) app? I had noted to @SBisson we have a client error logging component rolling out and it dawned on me that it might be useful for this app!

Tangentially, this makes me think about the client event logging stuff as well. The KaiOS app does not have a MediaWiki PHP dependency as I understand it (although it indirectly and directly depends on web APIs powered by such tech).

In T248615#6003253, @dr0ptp4kt wrote:

@jlinehan, is there a way to fashion the client error logging from MediaWiki in a standalone fashion for applications like the KaiOS (HTML5 web app) app? I had noted to @SBisson we have a client error logging component rolling out and it dawned on me that it might be useful for this app!

Tangentially, this makes me think about the client event logging stuff as well. The KaiOS app does not have a MediaWiki PHP dependency as I understand it (although it indirectly and directly depends on web APIs powered by such tech).

Yes, right now the KaiOS app is an unknown quantity for our BUOD work. I think ultimately we will be working with them to create a client solution for this platform, for both analytics events and error logging events. Error logging events are the easier problem here I think. We can potentially set up a spike next quarter to produce error events to the endpoint.

@jlinehan the window for us to introduce error logging before the release is rather short. Could you share an example of an error logging request? It is an event logging call to an Error schema or a different system? I'm trying to understand if there's a shortcut for us to log 1 or 2 types of error in the short term.

In T248615#6004633, @SBisson wrote:

@jlinehan the window for us to introduce error logging before the release is rather short. Could you share an example of an error logging request? It is an event logging call to an Error schema or a different system? I'm trying to understand if there's a shortcut for us to log 1 or 2 types of error in the short term.

Hey @SBisson! How short is the window? We may be able to do something but we might also need to create a new stream dedicated to KaiOS. We just deployed error logging for the MediaWiki web client and I don't want to cause too much short-term open questions for you if you're approaching a release. If you think it would be beneficial and we have a few days before launch, I think we can make it happen.

I'm going to list some things here for you to reference and I'll let you decide if it's something that's possible in the app. The basics are you need to POST some JSON to a public URL, and the errors will show up in logstash and you can browse them using Kibana. Here is an example dashboard for the web clients: mw-client-errors

Here is code the produces the request on the MediaWiki web client:
https://github.com/wikimedia/mediawiki-extensions-WikimediaEvents/blob/master/modules/ext.wikimediaEvents/clientError.js#L34

• You need to send an HTTP POST to https://eventgate-logging.wmflabs.org/v1/events?hasty=true, but this URL may change in the future.
• You'll need to set HTTP header Content-Type: application/json or Content-Type: text/plain.
• The POST body must be a JSON string that conforms to the /mediawiki/client/error JSONSchema
  • See example below, don't worry about deciphering this right now
• You'll need to send:

{
    $schema: "/mediawiki/client/error/1.0.0",  // (required) this literal string
    meta: {
        stream: "mediawiki.client.error",      // (required) this literal string
    }, 
    error_class: <your error class here>,      // (optional) the class of the error constructor
    message: <your message here>,              // (required) A message explaining the error
    url: <your URL here>,                      // (optional) URL of the resource where the error happened
    file_url: <your file URL here>,            // (optional) URL of the file that threw the error
}

If you just did this exact thing then the KaiOS error events would be mixed in with the web stuff so we should either mark it as KaiOS so it can be filtered, or create another stream that uses the same schema. I can help you figure that out. If you think it's possible to do this from inside the KaiOS app, then we can probably get something done early next week.

@jlinehan thanks for your prompt and detailed response. We have more than a few days so I think we can do it and we should.

In T248615#6005614, @jlinehan wrote:

[...]

Here is code the produces the request on the MediaWiki web client:
https://github.com/wikimedia/mediawiki-extensions-WikimediaEvents/blob/master/modules/ext.wikimediaEvents/clientError.js#L34

This code works pretty much as-is in KaiOS.

[...]
If you just did this exact thing then the KaiOS error events would be mixed in with the web stuff so we should either mark it as KaiOS so it can be filtered, or create another stream that uses the same schema. I can help you figure that out. If you think it's possible to do this from inside the KaiOS app, then we can probably get something done early next week.

While it's a web app, the KaiOS app will be installed locally on devices so we have to be careful with things that may change like the stream or the URL since users may keep running a specific version of the app for longer than we wish. We can publish new versions but we can't force users to update.

Creating a new stream looks like a good idea if it's not too much work.

The wmflabs URL worries me though. Is there any plan to move to a more opaque dns (like logging.wikimedia.org`)? Do you know if sendBeacon follows redirects?

In T248615#6014436, @SBisson wrote:

Creating a new stream looks like a good idea if it's not too much work.

Should be fine. @Ottomata, what do you think?

The wmflabs URL worries me though. Is there any plan to move to a more opaque dns (like logging.wikimedia.org`)? Do you know if sendBeacon follows redirects?

I think we may already be using another one. @Ottomata can clarify. I think sendBeacon is supposed to follow them.

Is there any plan to move to a more opaque dns (like logging.wikimedia.org`)?

https://intake-logging.wikimedia.org :)
See: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/master/wmf-config/InitialiseSettings.php#209

Creating a new stream looks like a good idea if it's not too much work.

It isn't too much work. You are coming into a newish system so docs aren't great yet. I can help with this when you are ready. We would need to configure logstash to ingest this new stream though, we should check with @fgiunchedi on that, but I expect it to be fine.

Documentation task here: T248884, we'll track this as well to help with docs.

we are in the middle of developing this, let us know when the new stream is available so we can add it to our source.

What should the stream be named? This will map to Kafka topic names as well as other downstream datastores, e.g. Hive.

• mediawiki.kaios.error
• kaios.client.error
• kaios.mediawiki.client.error

@jlinehan thoughts? We should choose a naming scheme that would make sense for other apps too. Theses are wikipedia specific apps, right? Maybe:

• wikipedia.client.kaios.error
• wikipedia.client.ios.error
• wikipedia.client.android.error

Other suggestions welcome please!

Chatted with @Ottomata and we agreed on keeping it simple with a stream called kaios_app.error. Will update this ticket when the stream is available.

Done!

In T248615#6073135, @Ottomata wrote:

Done!

Thanks a lot!

Yay! So @hueitan the stream name is kaios_app.error, so the meta.stream field in the JSON that gets sent will need to be set to kaios_app.error. Let us know what you need next or if you want code review.

Also, setting meta.dt to the ISO-8601 UTC date timestamp of the error is good too! If you don't set it, eventgate will set it for you, but setting it yourself gets the timestamp closer to the actual error event time.

Thanks @jlinehan @Ottomata

When I tried to send event to the page https://intake-logging.wikimedia.org, it currently returns 404, therefore I can't check the result in the dashboard, is this expected?

I guess the task doesn't say this explicitly, but the full URL is 'https://intake-logging.wikimedia.org/v1/events?hasty=true'

https://intake-logging.wikimedia.org/?doc

Thanks @Ottomata one more question, I tried to hit the error and the response returns success 201, but I can't see it on the dashboard, does the request header matter, for example User-Agent and Referer? or do I miss something?

Change 593225 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Collect kaios_app.error stream into logstash clienterror input

https://gerrit.wikimedia.org/r/593225

I see your events in Kafka!

Logstash is not yet ingesting this topic. Here's a patch for @fgiunchedi to review: https://gerrit.wikimedia.org/r/c/operations/puppet/+/593225

Change 593225 merged by Ottomata:
[operations/puppet@production] Collect kaios_app.error stream into logstash clienterror input

https://gerrit.wikimedia.org/r/593225

Merged, let me know if it works now.

Thanks @Ottomata, I can now see some data on the dashboard! 🎉

@jlinehan @Ottomata we would need a place to put the app version. I tried adding meta.appVersion and it works (it shows up in logstash) but I don't want to do something we're not supposed to, and see it break later. Any suggestions? Thanks!

Can you put it in the tags map? https://schema.wikimedia.org/repositories//primary/jsonschema/mediawiki/client/error/latest.yaml

so

{
  ...,
  "tags": {
    "app_version": "my.app.version.X.Y.Z"
  }
}

I'd recommend not using camelCase for any keys. It will be allowed in map type keys, since those are free form strings, but might bite you (and us) later if you try to e.g. use your data in a case insensitive SQL system.

See also https://wikitech.wikimedia.org/wiki/Event_Platform/Schemas/Guidelines#map_types

https://wikitech.wikimedia.org/wiki/Event_Platform/Schemas/Guidelines#No_Capital_letters_-_Use_snake_case

@Ottomata thanks, it works well.

@SBisson can you add your code to the list of clients here? https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate#eventgate-logging-external

In T248615#6137559, @Ottomata wrote:

@SBisson can you add your code to the list of clients here? https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate#eventgate-logging-external

Voila. https://wikitech.wikimedia.org/w/index.php?title=Event_Platform/EventGate&diff=1866295&oldid=1866281

Thank you!