Thank you a lot @hashar . I guess this ticket can be closed, however I have a last question - do you think we could somehow improve quibble to output this info without the need to place *cough* console.log()?

I'm not sure how Flow works internally, which hooks are called and why doesn't it set a user, better to ask Community-Tech ?

Parsoid was in a bit of trouble again today. At 02:44 XioNoX: depool eqsin, so all the mobile traffic for Chinese wiki started hitting RESTBase. Since Chinese has variants, RB started requesting Parsoid to transform HTML into correct variant. As the transformation request rate reached roughly 40 r/s, Parsoid started experiencing troubles, alerting and timing out.

I have tried to resubmit the change: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/EventBus/+/491014/ with no luck.

This is a bug in EventBus. Serializing the events for sending is protected with a try-catch, it logs an error and drops the job if it's not serializable. However, in this case, it fails in a different place - each job is signed via JWT so that it could be checked in a later point and verified that Mediawiki is the entity actually sending the job. That signing procedure internally requires serialization as well apparently and it's not protected with try-catch.

Rejecting events that haven't lowercased the headers is another option, albeit not as user-friendly.

EventGate, so that all parts of the system have a standard representation of the headers.

It's still undecided what to do with package-lock (T179229), so maybe let's just freeze the verison?

KafkaSSE requires ^2.3.4.

This is getting in our way since we can't merge an otherwise ready patchset

I don't think complicating Varnish logic worths here, I don't see how UA is better than a query parameter regarding DDOs. We can mitigate the DDOS possibility by setting Varnish max-age grace_ttl smaller then Cassandra stashing TTL for the VE requests and using 'must-revalidate'.

Indeed, haven't thought about that. On page deletion no LinksUpdate is scheduled, LinksDeletionUpdate is used instead which doesn't call a hook we use to construct the event.

@ssastry ye, it's a set of integration tests running against life Parsoid in the beta cluster. I think using
'nock' is a good approach, with an option to quickly update the nocks once a new major version of Parsoid format is deployed and quickly updating the expected/supported format versions.

In T216191#4955874, @Mholloway wrote:

Removed the linked patch next to mobileapps, which was unrelated unless I'm missing something.

MW train has been deployed, so the events are available for all wikis. The final piece is adding the stream to the documentation, but apparently, it's blocked now by T216184.

So, in kafka we have:

{"database": "enwiki", "mediawiki_signature": "97c895ef8e934b335f74b4d8d57d5b7bebd1de16dd68d88f76479c826f4a9bf9", "meta": {"domain": "en.wikipedia.org", "dt": "2019-02-12T22:12:43+00:00", "id": "51840dae-2f13-11e9-881a-aa00006e80dc", "request_id": "XGNE2gpAIHsAABGqlmYAAAAV", "schema_uri": "mediawiki/job/2", "topic": "mediawiki.job.clearUserWatchlist", "uri": "https://en.wikipedia.org/wiki/Special:EditWatchlist/clear"}, "page_namespace": -1, "page_title": "Special:EditWatchlist/clear", "params": {"maxWatchlistId": 214224212, "requestId": "XGNE2gpAIHsAABGqlmYAAAAV", "userId": 10774019}, "sha1": "32624f411d5070eacbfe3d0aab95b55a60748a6f", "type": "clearUserWatchlist"}

Hmm.... Just brainstorming here...

Seems like the things has calmed down. Even though we were not able to figure out the core reason, it seems by now we never will. Closing the ticket.

I'm trying to get alternative solutions because I would honestly hate emitting a heartbeat event.

If one of the topics in a stream has no data for a given hour, we can't tell if that is because of an upstream (producer related) error/delay, or if there is simply no data in that topic for that hour.

So in this case we have all the events in eqiad, and some rogue events in codfw. Is it possible to consume from 2 topics in the job and rely on mirror maker to deliver codfw events? We could quite safely ignore the codfw events, there should be a negligible number of them.

There's one more issue that's not being addressed here. The precaching config does not have all the models available for the wiki, so the revision-score event only contains a subset of everything ORES can calculate. There are actually 3 separate sets of models ORES calculates for different purposes T201868#4802296 . I guess you'd want to archive all the models? Can we in that case finally merge all the configs and merge the changeprop and job queue ores pre-calculations?

ORES change propagation is configured to mostly go through one datacenter

Sorry, got out of my radar somehow. After the latest deploy ^^ this has to work properly.

Maps will go on k8s eventually afaik, but it's an independent issue. Node 6 support ends in April even for security updates and that is the primary reason for the move.

Since we will (probably? for now?) only be producing new schemas to EventGate, I think it would be fine to add this to our meta field. Thoughts?

That I already have, just need to revert a couple of commits that went too far.

Although, I'm not sure if we should bother using CI to enforce JSONSchema restraints. If we can, that's awesome, but I wouldn't make it a priority. Good guidelines and schema review should be enough I think.

Hm... After hacking this around a little bit more, I think creating something general-purpose was actually a bad idea. JSON schema is much too powerful for us and what we really need is something very specific for us making sure nobody uses features of the schema we do not want people to use. As an example, json schema supports declarations like type: [ 'string', 'number' ] - which means the property could be either a string or a number. That would clearly break hadoop refinery, so we want to disallow this feature.

To achieve point 1, we need to

• • Refactor PDF module in RESTBase only to proxy traffic to Proton and remove the capability of split traffic, plus simplify the config.
• See whether we would still need a js module for it, or it's possible to achieve the same result with a template-based module with no JS.

Ouch... Somehow missed that one.

So, %u10DE is 'GEORGIAN LETTER PAR' and it's supported from Unicode v1.1.0. The %u1C9E is GEORGIAN MTAVRULI CAPITAL LETTER PAR and it's only supported from Unicode v11.0 which was released in June 2018.

We're getting closer. The last part of the puzzle would be to emit the event publically via event streams. In order to do that, we need to add the mediawiki.page-links-change topic to the list of exposed topics in puppet for production and for deployment-prep

If we were to use Monolog, we'd likely want to do it with the aim of converting all existent EventBus events to use it as well.

Interesting.. I will have to read this first before voicing an opinion. Not sure how likely this would become a standard.

In node we have an implementation of bynuan -> syslog adapter and it has a bit of a different mapping. It's not used yet, but this seems to conform to most of the mappings other node packages do. (Not saying we need to do what node does, just adding 2 cents of info)

With JSON, consumers don't really 'read data' with the schema. The schema allows us to ensure that the data the consumers get will be backwards compatible; meaning old consumer code will be able to read data produced with the new schema.

cc @Joe

If the endpoint logic has changed in the summary endpoint wouldn't the summary be outdated?

(unless Core Platform wants it for their own purposes). Something to chat about in the biweekly sync meeting.

The blip and ORES deploy seem to be unrelated and just coincided in time.

We have analytics by job type but not by wiki

We have discovered that we would need to update LinkUpdates class in the core to support this functionality, so it will take a little bit more time then we anticipated, but we're making progress.

I'm not particularly sure what do you mean and want to achieve. Putting ORES behind RESTBase without using storage (as a simple proxy) will not really free up any resourses on ORES side. Do you mean you want reads of pre-computed scores to not touch ORES at all?

Given that all the code is actually called from hooks, which are static entry points, this is not actually a real issue.

We need to locally test whether the bynuan-syslog-udp actually works and modify it to add the @cee annotation. Then we can proceed with testing in beta cluster.

How should we handle the situation with long list of links? Is that really a problem?

Are links url encoded? (we probably want them to be so).

@Pchelolo yes, replacing "properties" with "links" should do it. I don't think we need the full list of links, just the changes.

@bmansurov look at the event schema for properties change I've linked, it contains all of the metadata you need and it's generated from the same hook. Will such a schema work for you if we replace removed_properties with removed_links and added_properties with added_links? We can also emit the full list of links before the edit and after the edit and let clients do the diff. In general, we can get any metadata you can imagine.

Moving to services-blocked until Research comes up with the schema for the event.

The task at hand is very easy. There's LinksUpdateComplete hook in MW core, it gets the LinksUpdate which contains the list of external links and methods to get links insertions and removals. Writing code in EventBus extension to use the hook is simple. The important part of the work is for you to come up with the schema for the event you want to get. We already have an event for page properties change. I would assume your event will be quite similar. If you create the schema, I can write the producer code or point you where to write it.

Hmmm you seem to assume we control all the producers, but I'm not sure that will be true for this new system. Since it will also replace EventLogging, this might not be the case.

The topic appears to be deeper than I thought. Please read https://docs.confluent.io/current/schema-registry/docs/avro.html for the context.

Does that also go for errors? (Especially when rendering takes too long and gets aborted by the service, or abandoned by Varnish.)

There's already json-schema-compatibility lib on npm, but there's always some libs with a similar name to anything we could imagine, so jsonschema-compatibility is fine by me.

Before we begin, some bike-shedding needs to be done.

Change-Prop now has a separate rule for this job, so it doesn't interfere with low-traffic jobs. The concurrency is set to 30, which should be enough, however after deploying 100% we need to monitor the lag and increase concurrency if we see the job is steadily lagging.

Default low volume queue concurrency is 50 per https://github.com/wikimedia/mediawiki-services-change-propagation-jobqueue-deploy/blob/master/scap/vars.yaml#L101

So getting this to 100% without hitting the limit should be no problem.

Fixed and deployed.

Should we create a new schema repo now for analytics purposes, or should we just use mediawiki/event-schemas for now and create a new repo later when it is time?

Regarding stronger CSP, we might need a bit of an investigation. We definitely do not want the CSP to vary by user agent, so we need to look how narrow we can make the CSP without breaking anything. I think wikidomain + *.wikimedia.org + wikidata (just in case) should work well. I'll look into it.