Page MenuHomePhabricator

Provide the ability to have time-delayed or time-offset jobs in the job queue
Open, Needs TriagePublic

Description

aezell: Noting here at the top that it's been made clear that the "Watchlist Expiry" wish does not need this technology specifically. I would still be interested in having it as a use case to ensure a flexible solution/architecture is derived from this discussion. I'm leaving the description as is so the context of the discussion isn't lost.


The CommTech team received two wishes for 2019 which may require an ability to have jobs in a job queue that will be executed at a specific time. This tasks exists as a place to start the conversation about how we might implement this feature and who should implement it. We'll use this to identify specific feature needs and approach some consensus on implementation details.

The two wishes that would benefit from this capability are Watchlist Expiry and Article Reminders.

Specifically, here's how it might work for each:

  • If a user adds an expiring item to their watchlist, when the expiration date/time is reached, the item is removed from their Watchlist.
  • If a user wants to be reminded to edit/read/etc. a specific page, they can set a reminder that alerts them (via Echo and/or email) to go to that page for whatever reason.

A naive thought about this would be to have the ability to have jobs that can fire at a specific date/time. That could be achieved by allowing the developer to provide an "execution timestamp" when the job is added to the queue. The queue then would have code to manage checking these timestamps and determining what to do. Or, the developer could provide an offset (in seconds?) from the job creation timestamp. Again, the job queue would have the code to determine if offset has been reached and the job should be fired.

Based on my reading of the docs for the existing job queue, this use case might be slightly different. It's different in that it the current job queue seems to have grown out of the need to handle delayed processing based on something happening in the page or the web request. A generic delayed job queue could potentially be used completely outside the context of page requests. I don't think it must be different but it could be different.

There are issues with a system like this. One issue is having lots of delayed jobs clogging up the queue for things that need to happen more immediately. That could be solved by having two queues with different listeners. I'm thinking of something more like a job broadcast or pubsub kind of model. It might be as straightforward as having different job runners looking at the DB for jobs with different flags. Another issue can be dealing with failed jobs. In many short-lived jobs, it's clearer about what to do should the job fail. For something that has been in a queue for a year, the user's expectations become less clear.

I've previously used Celery to accomplish this kind of delayed job queue. I'm not recommending that specific technology but I think the approach documented here might be helpful for discussion.

I think we have the opportunity to do one of three things here:

  • Find a work-around. That is, design a very small way to do this without an actual queue-based implementation. The Watchlist Expiry is something we could certainly work around but Article Reminders might be more challenging.
  • Add a new capability to our existing job queue. Add this delayed job capability for all users of the job queue.
  • Create a new job queue divorced from the existing job workflow that provides this delayed job capability (and maybe other features).

Event Timeline

aezell created this task.Mar 20 2019, 6:06 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 20 2019, 6:06 PM
aezell added a comment.EditedMar 20 2019, 6:24 PM

@Aklapper This is a bit of a draft. I'll tag appropriate teams and projects (or ask for help in doing so) when it's ready. I didn't want you to waste your time wondering where this should go. :)

aezell updated the task description. (Show Details)Mar 20 2019, 6:51 PM
aezell added subscribers: mobrovac, kaldari, aaron, Joe.

Adding TechCom to get some visibility. Pinging @mobrovac and @Joe as we've previously discussed this in an Interlock meeting. Also pinging @aaron on @kaldari's suggestion.

Project tags for the mentioned projects can be added when they are created.

Restricted Application added a project: Analytics. · View Herald TranscriptMar 20 2019, 8:56 PM
Restricted Application added a project: Analytics. · View Herald TranscriptMar 20 2019, 8:57 PM
kaldari added a comment.EditedMar 20 2019, 11:43 PM

@aezell - There's a whole slew of related (but more specific) tasks in Phabricator. Not sure how these should best be organized...

  • T153817 Notify users when their user group membership is about to expire, or has expired
  • T189391 New block option: Notify me when this block is about to expire or has expired
  • T2582 Remind me of this article in X days (from 2004!)
  • T156808 Back-end infrastructure for timed notifications in Echo
  • T172542 Maintenance script to look for scheduled events and fire them
  • T172541 Method API for scheduled events
  • T172543 WMF production cron job configuration for scheduled events
  • T172544 Vagrant and WMF Beta Cluster cron job configuration for scheduled events
  • T88781 Create a Timer based reminder for workflows

I don't think any of these have been worked on recently though.

Joe added a comment.Mar 21 2019, 7:31 AM

I'm a bit conflicted about this, and let me clarify why:

in all the use-cases referenced above the use of a queue seems like adapting the actual need to our preexisting schema of how things work.

What is needed, at least at a superficial look at the list of tickets, is either:

  • TTL-based expiry of records
  • A mechanism to schedule events at specific times

I'm not sure either of those things are best implemented using the jobqueue.

The queue is in its nature not going to be deterministic about when a job will be executed, we need some other mechanism for this.

The most obvious way to do it is to have some lightweight cronjob run every X hours and do the necessary cleanups. If we want to build something more scalable/sustainable, that's a whole other project.

@aezell is there something I'm missing? Wouldn't a scheduled periodic job do what you need as well? Or you need precision down to the minute?
in that case, I'm not really sure it can be done with our queue either.

There is already an ability to execute jobs after a delay or at more-or-less specific time, but it's really not something we want to build on.

I the Kafka job queue it works by attaching a delay_until timestamp to a job, and then reading in jobs, delaying them for a little in memory and re-enqueuing them back to Kafka. For low-volume jobs this mechanism works ok, but it's not scalable - the more jobs we have and the longer the delays are, the more re-enqueueing will be happening and creating more and more unnecessary load on Kafka and change-prop.

TLDR, implementing this as a JobQueue feature is doable (and actually already exists in all the queue implementations), but it is a workaround and not a scalable solution.

@aezell is there something I'm missing? Wouldn't a scheduled periodic job do what you need as well? Or you need precision down to the minute?
in that case, I'm not really sure it can be done with our queue either.

In my view, there are two things here that we could separate:

First, I don't think it's a must for us to use the current job queue at all. @aezell can correct me if I'm wrong here, but I think it was given as an option out of multiple, just to explore available alternatives. In that respect, it might be a good exercise to try and do this the other way around (like you're suggesting, @Joe) and think first about what we'd need this for, and then see what the best implementation is (and whether it fits something that currently exists or if it requires a completely new thing)

Second, I think that while the resolution should probably not be "per minute", I am not sure if a strictly daily (in any one timezone) would work best either. The product would be adjusted to accommodate if this is technically the best way to go, but if we go by "what we'd like to see" I think that it should probably be resolution of hours or, at the very least, a couple of times a day.

For example, it could be perfectly fine to say that your watchlist item will expire at some date (resolution of a full day) at midnight UTC.

But for echo notifications, block reminders, and workflow reminders, we would probably want resolution of a few hours. I'm thinking of two reasons of looking into *at least* a half-day or quarter-day resolution here:

  1. If a user wants to be reminded to check into an Article for Deletion process, then they might want a reminder at the end of the day as opposed to the beginning of the day for their timezone.
  2. If we go by "day" resolution, we'll have to settle for what a day means (probably UTC midnight) which might not be the best for all timezones. Splitting the resolution for a couple of times a day can help us make this feel slightly more tailored to other timezones.

That said, I also don't think any of that is entirely crucial. I can see and guess a few performance and technical limitations that might make it harder to go into higher resolutions, but will still make the product viable, and we should hash those out.

My only point is that if we're thinking about building a system now that will hold us for future features, then we should look into whether we can accommodate a resolution that will give us support for more products, and only roll back to daily or "what we have" if that turns out to be technically terrible.

@Joe I'm not sure if that answers the question, or if I did a good job explaining myself fully, please let me know if it makes sense. I'm trying to think outside the box a bit, since we will all definitely bring ourselves back into reality with technical constraints and then adjust back to those. Does that make sense?

  • TTL-based expiry of records

I agree. This is the "work around" I alluded to regarding the Watchlist Expiry. This one does seem like the most straightforward option given the feature criteria. As you say, I saw the queue and thought we could use it for this feature as well but I wouldn't argue that it's the best solution.

  • A mechanism to schedule events at specific times

The most obvious way to do it is to have some lightweight cronjob run every X hours and do the necessary cleanups. If we want to build something more scalable/sustainable, that's a whole other project.

I think the scalability of this is where I thought it might fail and that a queue system or something that can handle backpressure might be a better idea. It's this I had in mind when I mentioned building something different than our current jobqueue. In my mind, it's still a queue in some sense but maybe it doesn't need to contain all the bits that a true message queue would contain.

I've seen and built systems like the one you describe with just a cronjob reading from a table in the past. I've seen them start to fail because of wrap around. Imagine you have a table full of reminders to send out. New reminders are added constantly. Your cron job runs every minute. But, because of the number of reminders to be sent or because of the failure of a downstream system, that process takes 3 minutes to complete. Now, you have wraparound and it's not clear which reminders are being dealt with and which are left to work on.

What kinds of protections might we have to guard against this? A message queue implementation generally handles backpressure like this quite well which is why that pattern came to min.

Joe added a comment.Mar 22 2019, 6:52 AM

@Mooeypoo thanks for clarifying requirements of the current project better, it's much clearer now.

@aezell I am aware of the limitations, that's why I said it seems like we need a scheduler, which is a different thing than a jobqueue. A cronjob (even if correctly guarding against overlaps, which is doable) is not a real solution, just a temporary patch.

The point I was trying to make (and now I can make with more confidence) is that:

  • If you need to build a one-off project and likely won't ever need something similar ever again, a cronjob with some data store (a db table) might be "good enough for now"
  • If being able to schedule specific jobs at given times is a need, then we need to create / utilize a distributed job scheduler. This is a larger project and it would require proper resourcing imho.

From all the links I saw above I would infer we should probably look at building such a scheduler. To be clear I don't think it should fall on your team's shoulders, but we need to find some resourcing and a time slot to work on it. Luckily, I'm not a manager so I can let that part (the truly difficult one) to someone else.

If you want to bring the feature early to users, creating a cronjob might be enough as a stopgap, but I wouldn't go in that direction without a clear commitment from others to build such a system.

Pchelolo moved this task from next to watching on the Services board.Mar 22 2019, 11:28 AM
Pchelolo edited projects, added Services (watching); removed Services (next).

Thanks for the insight @Joe. I appreciate your giving some thought to this.

I'll share some of this feedback with the team and see how we think we want to proceed.

fdans moved this task from Incoming to Radar on the Analytics board.Mar 25 2019, 4:00 PM

@EvanProdromou I think that a solution like that is something closer to what @Joe was proposing than the job queue thing I described. I really like the idea of a capability like this. I'm glad that Giuseppe helped make it clear in his comments.

It does seem that a generic system like this available to all Wikimedia projects could be really helpful for many kinds of tasks. Having a single place for this sort of work to happen would seem to me easier to maintain and administrate and scale.

FWIW, Analytics will (eventually?) be considering replacing our scheduler (Oozie) with Apache Airflow: https://airflow.apache.org/, which has k8s integration.

daniel added a subscriber: daniel.Mar 26 2019, 3:13 PM

Wikidata also has a need for delayed and/or periodic tasks for change dispatching. That is currently based on cron jobs, and it's pretty terrible, see T48643. I'll block that ticket on this discussion here.

Overall, having a facility for delayed and/or periodic jobs in core would be nice. However, we'd have to provide a way for this to work in a minimal shared hosting environment - probably based on cron jobs. So in ordedr to "do this right", we have to figure out a number of things:

  1. what should the in interface for scheduling delayed and periodic jobs look like from inside MediaWiki? What are the needs/requirements?
  2. how do we provide a reasonable baseline implementation that works in a basic schared hosting environment?
  3. what should the implementation based on a distributed job scheduler look like?
  4. how do we deploy and run such a distributed job scheduler?
  5. who will do the work needed?

Overall, this looks like RFC material if it's supposed to be more than a one-off.

@EvanProdromou I think that a solution like that is something closer to what @Joe was proposing than the job queue thing I described. I really like the idea of a capability like this. I'm glad that Giuseppe helped make it clear in his comments.
It does seem that a generic system like this available to all Wikimedia projects could be really helpful for many kinds of tasks. Having a single place for this sort of work to happen would seem to me easier to maintain and administrate and scale.

Sounds like we should rename the task.

I also think that making sure MediaWiki is installable and usable in shared hosting is a worthy goal, and there are definitely some instances where we should be more mindful of it.

That said, I also think that there are a lot of cases where we can solve this by taking a different approach on these type of features, where we can answer both the need to give a powerful new tool to use *alongside* the ability to continue enabling 3rd party users to install and use the software.

Bear with me here, I'm hoping my point makes sense: I think that in the case of a scheduler/timed-queue, we can consider it an "extra" feature; the wiki can work with it (and then you get the benefits of the products that can use it) but it doesn't require it for general operation.
There are multiple ways to approach building this tool, as this ticket shows, so we could also consider creating this as a replaceable module (or extension).

Putting aside consideration of manpower for just a minute here (I'll get back to that) and considering this idea requires more fleshing out, we could do something like this:

  • Create a process queue interface in core that defines the operations of what's needed.
  • Create an extension that implements a timed queue for wikis that are on a server/farm they control (and hence, something that fits *our* use case, as well as some other bigger 3rd parties, but not shared hosting)
  • All features that require this queue will be conditional on whether this system is enabled; we enable it in production. Self-hosted wikis enable if they can. Shared hosting disable it (for now).
  • We document the requirements of this queue, and either build outright or at least help guide a less-perfect but shared-hosting-ready service as either another extension, or as a fallback to the module/extension we created.

The advantage of this approach is that we can then create something very powerful for bigger wikis as well as Wikipedias and our production wiki family, as well as give this powerful tool to bigger 3rd parties, but also not go against our principles of having the wiki be easily self hosted.

A second advantage is that as shared hosting grows and as things like droplets and vps servers become cheaper and more widely available, the tool we build today -- a robust modern "The Right Way To Do This" tool -- will, eventually (and not too far in the future) be also available for smaller 3rd parties.

It's an investment for the future, with a fallback for smaller wikis.

All of the desired features that depend on this tool are not "be all end all" features that the wiki will be unable to function without. Giving those as optional enhancements with a way to do it "small" scale vs "bigger" scale is, in my opinion, a great way to support both causes.

Coming back to reality a bit, that would mean that we might need to create two queues like this, which impacts realistic planning about manpower and prioritization and timing, and that's fair. However, as I pointed out above, we can start with a solid structure that allows for implementing somethng for self-hosted, and then prioritize building something for our stack first, and either go back to the smaller/fallback product later, or encourage and allow our volunteers to implement that smaller-scale tool. If we create a good basis, it should be doable and straight forward.

I hope this makes sense and that I'm not too much of an optimist here... ;)

Overall, this looks like RFC material if it's supposed to be more than a one-off.

Yes, I think that there's agreement on that in this ticket. We're sharing our needs and concerns first so that an RFC can be created that's based on the best solution and approach, we're hoping to be able to get something going that's valid before a more practical/written RFC is presented to the community.

Task description:

If a user adds an expiring item to their watchlist, when the expiration date/time is reached, the item is removed from their Watchlist.

This does not require scheduled execution of jobs. We have various features in MediaWiki already that have an expiry of sorts, which are enforced at run-time. One may be concerned with internal storage allocation, but that is entirely unrelated to any user story or product requirement, and we have existing mechanisms in place to address those. Unless and until there is a problem with that, I do not think we should invest in another way to solve the same problem.

Note also that execution of jobs is by no means guarantee to happen at a certain time. We could maybe ensure it won't execute before a certain time, but not that it won't be a day or a week later. This is precisely why you'll still need run-time enforcement. Which we do already for user rights, page protection, blocks and other features.

Task description:

If a user adds an expiring item to their watchlist, when the expiration date/time is reached, the item is removed from their Watchlist.

This does not require scheduled execution of jobs. We have various features in MediaWiki already that have an expiry of sorts, which are enforced at run-time. One may be concerned with internal storage allocation, but that is entirely unrelated to any user story or product requirement, and we have existing mechanisms in place to address those. Unless and until there is a problem with that, I do not think we should invest in another way to solve the same problem.
Note also that execution of jobs is by no means guarantee to happen at a certain time. We could maybe ensure it won't execute before a certain time, but not that it won't be a day or a week later. This is precisely why you'll still need run-time enforcement. Which we do already for user rights, page protection, blocks and other features.

Yes, in the case of the expiry, we were discussing alternative ways to implement on runtime. I would say that it does have some complexities, since watched pages impact some other products, but that specifically can be worked outside the timed queue.

That said, there are quite a number of other products (scheduled notifications and more from the list at the top of the ticket) that require a scheduler. The scheduler doesn't have to use the job queue; if we agree that scheduling should be consistent *and* trustworthy, then we should find a way to implement a system that allows for guaranteed operation. As as pointed out in the ticket, the two cases we brought forth aren't the only use cases for this, so it is a good idea to come up with something that can answer the need from now on.

Joe added a comment.Mar 28 2019, 6:17 AM

FTR, I don't think kubernetes is an option for running mediawiki jobs for now. Because we're still deploying mediawiki with scap, which is not exactly a cloud-native tool, making mediawiki execute on k8s is very difficult.

Also, for now I'd concentrate on gathering requirements for such a new system.

[..] there are quite a number of other products [..] that require a scheduler.

Yes! I didn't meant to suggest we won't need a scheduler, sorry about that.

I merely think it would help focus this task by omitting the expiring watchlist use case, focussing on these products instead.

daniel added a comment.Apr 3 2019, 7:25 PM

A quick note on process.

Mooeypoo wrote:

Yes, I think that there's agreement on that in this ticket. We're sharing our needs and concerns first so that an RFC can be created that's based on the best solution and approach, we're hoping to be able to get something going that's valid before a more practical/written RFC is presented to the community.

That early/preparatory stage and and perhaps should already be part of the RFC process. An RFC doesn't have to be approval for a proposed technical solution. It can also explore a technical problem, or request guidance in the search for a solution.

We have in the past not always been very clear on this. Perhaps we should more often be explicit about kind of RFC something is, or in what stage or iteration it is. Such as "RFC looking for guidance" as opposed to "RFC requesting approval".

aezell updated the task description. (Show Details)Apr 8 2019, 10:45 PM
Krinkle edited projects, added TechCom-RFC; removed TechCom.Apr 9 2019, 12:31 AM
Krinkle moved this task from Inbox to Backlog on the TechCom-RFC board.EditedApr 9 2019, 12:35 AM

Tracking on the RFC board. As Daniel mentioned, it's not yet in the stage where it's seeking input or consensus for a specific implementation. Instead, I've moved it in the backlog pending a point in time where there's consensus on direction, acceptance criteria, and resourcing.

Then once you'd like input from others to help form one or more proposals and/or present your own, feel free to move it to the Inbox for the next step.

It is possible at any stage, including this early stage, to request an IRC meeting.

MaxSem added a subscriber: MaxSem.May 14 2019, 5:44 PM
fsero moved this task from Backlog to Incoming on the serviceops board.Thu, Jun 20, 2:24 PM
Joe edited projects, added serviceops-radar; removed serviceops.Fri, Jun 21, 8:44 AM
Joe moved this task from Backlog to Watched on the serviceops-radar board.Fri, Jun 21, 8:52 AM