Wikidata has potential need for delayed and/or periodic jobs. This would replace maintenance/dispatchChanges.php, which we currently run via cron on terbium. Ask @hoo about the details.

@hoo, @daniel let's talk about it tomorrow during the hackathon day 2?

@mobrovac sounds good!

Sounds good to me too.

Some relevant use cases from today's eventbus sync discussion:

• Batching & rate limiting for Echo notifications
• CirrusSearch for rate limiting / batching
• Wikidata for rate limiting, partly work-around for lack of dependency tracking
• Delayed HTCP purging (by ~10s) to account for replication lag

There are three considerations relevant to Echo:

1. Delayed notifications (T156808: Back-end infrastructure for timed notifications in Echo)

1a. Article reminder notifications (T2582: Remind me of this article in X days)
1b. User group expiry notifications (T153817: Notify users when their user group membership is about to expire, or has expired)

2. "Batching & rate limiting for Echo notifications".

There is a in-progress patch for T156808 (https://gerrit.wikimedia.org/r/#/c/370502/). This is being developed for T2582, but it's general infrastructure meant to work for all use cases, including T153817 (which I specifically considered).

The current patch for T156808: Back-end infrastructure for timed notifications in Echo does not use Jobs in the sense of JobQueue (it has a maintenance script, and if this is merged, it will use OS cron).

However, the patch for T153817: Notify users when their user group membership is about to expire, or has expired does use getReleaseTimestamp. However, I've asked them to build it on top of our infrastructure instead (I think they're waiting until it's merged).

So that leaves #2. I think this is a reference to ProcessEchoEmailBatch. We currently use a OS cron job for that. If there's another use case you're thinking of, please let us know.

(I also checked the existing Echo jobs, and they do not use getReleaseTimestamp/jobReleaseTimestamp)

@dr0ptp4kt @phuedx @Fjalapeno @kaldari @TrevorParscal adding you to make you aware of this request from Services team.

IIRC Readers Web doesn't actively maintain any extensions or skins that make use of delayed job executions.

We had considered them for fetching related pages after a successful edit in the RelatedArticles extension so that we could remove the API request from the client-side code. However, this is mostly mitigated by deferring loading of related pages until the user has very-nearly-almost scrolled them into view.

@MarkTraceur, adding for your review as well in case anything pertinent for Multimedia.

Multimedia probably needs this for thumbnail/convert/rotate/other media manipulation queues, though we haven't really worked with those systems as a team in the past. Something to keep in mind (unless I've misunderstood what's being discussed here)

In T172832#3540031, @Mattflaschen-WMF wrote:

There are three considerations relevant to Echo:

1. Delayed notifications (T156808: Back-end infrastructure for timed notifications in Echo)

1a. Article reminder notifications (T2582: Remind me of this article in X days)
1b. User group expiry notifications (T153817: Notify users when their user group membership is about to expire, or has expired)

2. "Batching & rate limiting for Echo notifications".

So that leaves #2. I think this is a reference to ProcessEchoEmailBatch. We currently use a OS cron job for that. If there's another use case you're thinking of, please let us know.

(I also checked the existing Echo jobs, and they do not use getReleaseTimestamp/jobReleaseTimestamp)

@Mattflaschen-WMF, thank you for the background on echo! Regarding use case #2, can this be fairly summarized as "wait for a limited time before sending out notifications, in order to reduce the volume with batching"?

I wrote a little script to run through a sample of events in the job topics we have in prod right now and here's.a list of job types that had the releaseTimestamp set:

• mediawiki.job.cdnPurge
• mediawiki.job.cirrusSearchCheckerJob
• mediawiki.job.cirrusSearchElasticaWrite
• mediawiki.job.cirrusSearchIncomingLinkCount
• mediawiki.job.UpdateRepoOnDelete
• mediawiki.job.UpdateRepoOnMove

The list might not be complete.

In T172832#3559604, @GWicke wrote:

@Mattflaschen-WMF, thank you for the background on echo! Regarding use case #2, can this be fairly summarized as "wait for a limited time before sending out notifications, in order to reduce the volume with batching"?

I was quoting "Batching & rate limiting for Echo notifications" from the notes you posted above. The original Etherpad notes say "Batching & rate limiting for Echo notifications (might actually just have been a patch, not merged)", and I don't know what that was originally referring to.

ProcessEchoEmailBatch implements optional email batching for user experience (i.e. so emails you get from Echo don't feel spammy). This implements the preference at https://en.wikipedia.org/wiki/Special:Preferences#mw-prefsection-echo ("Send me:"). Choices are:

• Individual notifications as they come in
• A daily summary of notifications
• A weekly summary of notifications

As mentioned, we use a maintenance script and OS cron job, which seems to work fine.