From the RESTBase perspective, it means deleting a huge chunk of complicated code and a lot of simplification. If there are no plans to use these in the future, I would love to get rid of these.

We have had to deal with this in EventBus as well, thes page properties are not JSON-encodable. We ended up with a hack like this.

The one that I've posted I think is not easily replaceable with MW API, however, I think it's possible with a bit of code and might be faster than going to WDQS.

The exact query that's being used:

I think we have a consensus here. @Smalyshev would you agree with my proposal?

In T211397#4806827, @bmansurov wrote:

We use WDQS when we need to get article titles in a set of languages give a Wikidata item ID.

First things first, article 'Libro' does not exist in the beta Spanish wiki.

There's not that much logging happening in beta services, I would guess we should start big, so deployment-restbase0[1,2]?

It is setup on deployment-sca, but I could not make it work. Needs more investigation why.

I can just remove the expectation of specific itemType, but let's see what @mobrovac has in mind.

Probably related to https://phabricator.wikimedia.org/T211088

Production is correct. @mobrovac disabled zotero in beta cluster. The problem only manifests in beta cluster.

As we're finally emitting the revision-score events and will make them publically accessible soon, I would like to bring this up again.

In T206889#4801877, @Nuria wrote:

Ping @Ottomata @Pchelolo is this work we are iaming to do for next quarter (january onwards) as we are also working on dockerization?

Oh.... Apparently, apple have broken syslog in newer OSX versions, so I'm most certainly not the best persong to tackle this task. Supposedly, if we support receiving logs over UDP everything should work out of the box. If you're not too opposed to UDP transport, we could test it in beta almost immediately.

So, currently we only support sending to syslog via UDP using the node-bunyan-syslog-upd package created by @mobrovac It's pure-js implementation since we want to avoid native bindings.

We're currently not using CP in the dev cluster since we have no new major RESTBase features in work, so the CP was stopped there and thus the priority of this became lower.

Seems like after this has been done the citation alerts started flapping much more then they used to. Also, the mean latency for citations endpoint went up from seconds to minutes.

Everything looks perfect from CP side. Please close whenever you feel like it.

@bearND Currently MCS (PCS) only talks to MW and RB. Adding Parsoid into the mix would not be my preferred way to go. Adding an endpoint to MCS (PCS) where we can POST the HTML and get all the transformations done would feel more natural. We eventually can even change everything in MCS(PCS) to do POST based API - it will be more efficient then what we're doing now. One less round trip, one less load up of HTML from storage => less latency.

@Pchelolo Yeah, that's why I think this new endpoint would probably be similar to /transform/wikitext/to/html{/title}{/revision}.

Also, it's gonna be interesting to implement this. MCS output is based on Parsoid HTML, and the HTML is not POSTed to MCS, it GETs it from storage. So in order to implement the transform endpoint, we will need MCS to support a completely different request flow. Not that it is impossible, it just might be a bit more work than expected.

Mm.. I will not be that certain the deserialization of JSON is the issue here. We deserialize much much bigger messages in the job queue and have no problems with OOMs. I'm not opposed to the proposed patch, just saying I have a gut feeling it will not help to solve the problem. Since EventStreams is powered by service-runner, we can take a heap dump from prod with kill -SIGUSR2 and analyze the heap. I just don't have the sudo powers for that.

The alert has been created and tested.

my 2 cents: EventGate is awesome, GH is the best place for this piece of code.

Do you need the events for the last month to be replayed?

Not to be worried. We have all the failed events stored since 2018-04-18. If needed, I will fetch all the missing page deletes tomorrow.

Hm... Tried testing this in beta and apparently Varnish purging is broken in beta cluster. Needs investigation.

So here's the plan: tomorrow I will test some alerts manually, if all goes well - we will add analytics into the contactgroup for eventbus dashboard.

The ones we have for RESTBase doesn't go to IRC, but I think for analytics it should appear on IRC judging by https://github.com/wikimedia/puppet/blob/ce74f598509d2b448fc1252a3cd7c8241cc7b2c2/modules/nagios_common/files/contactgroups.cfg#L7

@Nuria yes.

Currently, if the alerts will be enabled, the emails will go to only to services mailing list should we add Analytics there as well?

If I understand the proposal correctly, it's easier to do it using grafana alerts which I'm proposing in T210031

Yes, the fix has not been deployed yet.

The fast-json-stable-stringify library is indeed 20% faster than the one we're using and it does not have any dependencies. Even though we do not use this lib in critical for performance code paths, I think replacing is still a good idea.

Ok. The events with the new schema are being emitted, there are no rejections on the EventBus service side, no errors in ChangeProp since the last patch was deployed. I believe we can finally resolve this task 🎉

Relying on the order of things in JSON was clearly a bug and it's nice that it's been fixed. However, I do not think json-stable-stringify was the one to blame - it's not used anywhere on the actual response data - it's used more on various metadata to compile hashes of, for example, table schemas. I do not quite understand which exact change has made the JSON properties reverse order, but since it's not specifyed, it could be literally anything. I'm inclined to close the task as Invalid.

This is by design. REST API is supposed to be accessed when the whole {title} parameter is URI-encoded since that's how they're referenced by Varnish. We redirect to URI-encoded paths as well.

The argument is for some reason not documented

Ok, I understand what's happening here. The BounceHandler extension sends the job cross-wiki, in this particular case to the wikitech wiki queue. However, JobQueueGroup::singleton()->get() uses global $wgJobTypeConf, which is different for all wikis except wikitech. That's how the event ends up in kafka and obviously fails cause wikitech doesn't support kafka job queue.

Actually, I believe there's an easier way to find out the events that cause it: https://logstash.wikimedia.org/goto/2519c6383201ddf651de7e3effed92b9

There actually is something wrong here. The average TTL for redis_misc in codfw is 0, which, I believe, indicate there's no TTL set at all in codfw.

Yay!!! Since the keyspaces have been actually deleted, we can close the task now. Congrats @Clarakosi

@Ottomata Let's try again? https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/EventBus/+/472189/

A quick update here.

So, some search results. I will not include only things that are remotely applicable to us:

LGTM

There should be a way to set basePath somewhere in config.*.yaml and get it merged into the Swagger spec.

Oh, I've missed the use via config->get! I guess that means we need to support this in the multi-master session store if it's really used.

A link to Slack? I don't have it, could you please copy-paste what you meant @Jdlrobson ?

Right now we're deciding on the interface for the multi-master session store and we're trying to decide how exactly to configure the TTL for sessions, and coming across this setting I started looking into the current codebase and failed to find any use of the config variable. It seems like in MW 1.27 change

I've proposed the env variable solution, but after sleeping on it I do not think it's a great one, it's more like a workaround. Env variables are supposed to be a configuration and in this case, cancellable promises are a requirement for the software to even work.

Since T207817 got closed I guess we can make one more attempt to fix this.

The code has been deployed. This could be applied in production now.

Fix has been deployed

fix has been deployed.

@Ottomata is a much better person to ping

heh, so it seems our beautiful idea for the API have just crashed into the rocks of reality @Ottomata

Some related conversations I was able to find https://github.com/json-schema/json-schema/issues/220#issuecomment-210152816 and I agree with the idea in that comment - by including the schema in any property within the data we already assume a certain format for the data (the fact it's JSON, the fact that $schema property exist, the fact it's pointing to the schema). It's really easy to verify, but logically it's wrong.

I think by now we've all reached the agreement to use references.

@Etonkovidova sorry I didn't update this ticket.. We've had an outage caused by the fix to it so it was reverted this morning. We will continue working on this - we know what the problem is, just need more time

We can keep theorizing about this, but we need more information before we could make any real theory.

Things I've checked so far:

• There are no logs anywhere associated with these events.
• All the events are legitimate edits, they exist in DB, they show up in the page history.
• I did not find anything common between the events, it's just a completely random sample.
• Events related to these edits are not present in eqiad.

Nope. The events are all legitimate edits, just random portion of them. Here's an example: