What makes you think that Terbium is to unstable for this? Terbium seems to always have more than enough spare resources, and the dispatch problems we saw recently seem to correlate with a high number of edits, but not with spikes on Terbium.

In T193733#4178239, @hoo wrote:

What makes you think that Terbium is to unstable for this? Terbium seems to always have more than enough spare resources, and the dispatch problems we saw recently seem to correlate with a high number of edits, but not with spikes on Terbium.

That is a very valid reason and I failed to demonstrate my intention fully. One big reason is to be able to scale better and utilize more resources, because if we have a dedicated node, we know the limits and have a pretty stable view of things but in terbium it's not okay to take over the whole node because the number of edits is just too high. Also having a node means more resources which seems what we need here too.

Ok, in that case this sounds like a valid request to get an own VM (or even bare metal server).

Depending on how fast this can be done, this is a good short term solution for the performance problems. Long-term this sounds like a good idea as this moves the "real time" service away from terbium, which is probably not the right place for this.

We shouldn keep track of things like T110528: [Task] Use JOIN to find changes relevant for a given wiki though (which should make this use way less resources, as most of the comparison logic will be done with a simple JOIN).

In T193733#5276330, @Addshore wrote:

Going to mark this as stalled.

Also we havn't had performance issues with dispatching for quite some time now after fixing something with the db tables that we spotted at the last DC switch over, so maybe these can stay where they are for now.

In general, change dispatching is rather big technical debt that needs to be dropped. For these reasons:

• It's a SPOF, if mwmaint1002 node goes down for HW issues, we can't dispatch at all. If there's a need to restart the node, dispatching has to stop until it's done.
• "Noisy neighbor" effect, people run maintenance scripts in the mwmaint node, it can be choked to death by other scripts and it can make running maintenance scripts impossible by having bugs that eats all of the resources.
• The distributed system we designed for this (pulling the wikis using three cronjobs, dispatching and picking up basically random + most stalled ones). This can use the great infrastructure for jobqueues we have.
• Cronjobs are hard to debug, moving them to jobqueue makes it easier to debug in logstash.

So I like to just drop the whole thing but first we need to address T220696: [Story] Create better edit summaries for wbeditentity API endpoint which enables us to make all the edits done in UI by just one edit instead of several back-to-back edits. Once that's done, we can switch to each edit triggering job to trigger jobs to wikis that are subscribed to the entity and then those tirgger parser cache invalidation and other needed jobs for pages in the client. Does it sound good to you @Addshore? (Maybe there's other historical reasons to still use cronjob-based dispatching I'm missing, @daniel Do you know anything else?)

In T193733#5276659, @Ladsgroup wrote:

So I like to just drop the whole thing but first we need to address T220696: [Story] Create better edit summaries for wbeditentity API endpoint which enables us to make all the edits done in UI by just one edit instead of several back-to-back edits. Once that's done, we can switch to each edit triggering job to trigger jobs to wikis that are subscribed to the entity and then those tirgger parser cache invalidation and other needed jobs for pages in the client.

Why do we need to wait for the edit summaries?
The job queue already has good de duplication, so triggering a job after every edit already should not really be an issue.

Does it sound good to you @Addshore? (Maybe there's other historical reasons to still use cronjob-based dispatching I'm missing, @daniel Do you know anything else?)

The main benefit of the current system vs the way a job based system would probably look is the batching, and there are a few different way we could think about doing the batching.

There are some details in T48643 from back in 2015, but they should probably have another wave of thinking about now we are 4 years down the line.

Eg. Do we:

1. Schedule a job post edit to push out the change for the 1 entity to all subscribed clients? Do we do this all in 1 job, or spin off a job per client?
   • less batching, not very similar to how we do things now
   • Means we can run multiple jobs for a single wiki at once
2. Schedule a job post edit to spawn new jobs for updating each client subscribed to the entity? But then in the branch jobs pull in all available changes?
   • More similar to our current batching process
   • Probably only want to run 1 branch job per client wiki at a time
3. Some other odd smush of the above

Some other small things to consider with the job queue, is determining how lagged (in terms of minutes and seconds) these jobs are / dispatching is becomes slightly harder, as we dont just have a column in a db table with timestamps we can look at (or maybe we do if we decide to keep the table?)

Still some more thinking to be done regarding jobs, but I'd rather us focus on jobs than try to move the dispatching cron job to somewhere else.

So perhaps we leave this ticket alone for now and get discussing on T48643: [Story] Dispatching via job queue (instead of cron script) ?

is this task still valid?