As a part of the WMF job execution overhaul under T244826 we're planning to kill both /rpc endpoints in mw-config and replace the /rpc/RunSingleJob with an internal REST endpoint.

Oh, I was somehow not aware of this task existence.

All uses have been eliminated.

This RESTBase endpoint is pretty much wrapping Action API generator:search endpoint with a morelike query and returns the result. The logs indicate the problem is either in the Action API or in Cirrus Search. Here's the code for the request RESTbase generates.

Question: wouldn't it be better if we just had a configuration swtich that either allows or disallows access to this endpoint (set to false by default)?

I haven't realized the store we have is a CachedBagOStuff thus a check I wanted to eliminate only goes to the backend store once.

In light of T244826 we are not really going to do this, instead remove the JobExecutor entirely.

I've poked around mediawiki-config, and I don't really see a none-hacky way of setting a variable depending on MW cluster. set-time-limit.php has an implementation, but it seems a bit hacky. Is there a better way of getting a name of the MW cluster within MW config?

Per conversation on IRC this is not the case.

In T244586#5861273, @Jdforrester-WMF wrote:

deployment-restbase01.deployment-prep.eqiad.wmflabs reports The last Puppet run was at Mon Jan 20 10:54:08 UTC 2020 (26669 minutes ago). but as it's still working I guess it's fine?

RESTBase itself seem to be working correctly. Something is wrong with routing before RESTBase. If you look at https://en.wikipedia.beta.wmflabs.org/api/rest_v1/ the default error page my mediawiki is returned.

Let's hold on this one until we finish T220127 cause we will change how the job execution is called in production.

@Clarakosi has fixed the underlying issue and a htcp-purge@0.3.1 was published and will be deployed on the next change-prop deploy.

Per discussion on ops-services-sync, this will not really be needed for a k8s switch, but it's still a good idea to do it.

In T244069#5855237, @Joe wrote:

Maybe change it to use a 64 bit integer instead?

Oh! The core reason is that a counter has overflowed int32. Our software is so stable, we can overflow int32 now!

Yes

Since @Clarakosi is now a RESTBase deployer, we can let her do it for the sake of learning. Please coordinate with me when/what will be happening, I'm not entirely sure the steps below are a complete list of steps.

I haven't really thought about it before, but how're we gonna handle redirect pages?
This is going to be a valid question for all the HTML endpoints.

The job now has it's separate rule/queue, so you can feel free to start producing these jobs, it at least shouldn't break anything else.

The result that @SBisson is seeing is coming from Varnish:

age: 74298
etag: W/"932739206/80397760-3471-11ea-aa69-9fd8dec7efad"
x-cache: cp1089 hit, cp1083 hit/41

I see a 200 in both Safari and Chrome, but the result is Carnivore - the target of the redirect. I believe the browser just follows the redirect by default and not shows it. Or am I supposed to see something else there?

In T241072#5808916, @Cparle wrote:

Looks like we're getting ~500 File page creations a day, so we're talking at least 1000 delayed items being in the queue at all times. Is that a lot?

After a bunch of fighting with corrupted change-prop repo on gerrit and with help from RelEng, we now have a working image built via the pipeline.

In T241072#5797155, @Mholloway wrote:

My recommendation would be to go ahead with splitting into a separate queue/changeprop rule, if that's acceptable from a service ops perspective. Relatively speaking, this is a pretty low-volume job type.

Since (long-term) we aim to replace all of this, is abandoning it entirely an option?

LGTM.

Now you can add integration tests to the endpoint.

I like the idea of separating partial and full schemas very much.

In T240068#5789897, @MSantos wrote:

@Pchelolo and @Clarakosi, I wonder if this issue has anything to do with parsoid changes and if the recent restbase deploy could fix the language conversion problem.

I've fixed up the last few broken metrics. It seems that the job queue dashboard is in good condition now.

It worked but took ca. 6 minutes to complete.

Perhaps we could make it log a warning and error-out if JobQueueEventBus is used.

Seems like the core reason is the overrides introduced in T219279 and mediawiki-config Php72ToUpper.php file that overrides the upper-casing of certain characters in php for backwards compatibility.

Some of the links in the task description do provide the content.

I've relinked my Phabricator account (@Pchelolo) to WMF account on mediawiki.org User:PPchelko (WMF) - would that be enough?

Ok, I still like the idea of EventGate being able to partition by configured key, but it sounds like that doesn't really help us here, eh?

In event data: database_shard: s1

If key_field is set, EventGate will produce the message partitioned by the value of that field in the message.

Seems like this task is only about RESTBase tests, as it should be the only one accessing JS parsoid in beta directly. Updated the task description to accommodate that.