Hi SREs! I'm not sure if these are the right tag, please feel free to re-tag this as appropriate. The CampaignEvents extension now has a maintenance script that should be run periodically to mitigate the side effects of timezone rule changes. We would like to schedule this as a cronjob both in beta and in prod. I was wondering if you could provide some help/guidance with that; I think we will have to add something in puppet under maintenance but I'm not familiar with puppet and I don't know exactly what to do (beyond copying existing code).

I also don't know how to do that for beta (assuming that it's separate from the section for production). And even for production, it should only be run for the wikis where the CampaignEvents extension is enabled.

Finally, I wanted to hear from you what would be an acceptable schedule. Would you be OK with once per day?

Thanks in advance!

Thank you for reaching out @Daimona, I'll move this to serviceops for their opinion

@Daimona Do you have a timeline as to when you need this completed? As far as schedule, I believe it depends on our needs as well as how long it takes for the job to run ?

In T320403#8393213, @jijiki wrote:

@Daimona Do you have a timeline as to when you need this completed?

Our planned release date is the last week of November, see schedule (CC @ifried). On the one hand, this is not blocking the release because outdated entries are not a serious issue, and we can also run the script manually if the timezone rules change. However, I think ideally we wouldn't want to wait too long for this, so that we can close the epic as resolved and get it out of our radar.

As far as schedule, I believe it depends on our needs as well as how long it takes for the job to run ?

This is not easy to estimate. The script always scans the whole table, so the time scales linearly with the number of records. I don't expect the table to become very large (especially not soon), so we're talking seconds for now.

Change 858346 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] P:mediawiki::maintenance: CampaignEvents periodic

https://gerrit.wikimedia.org/r/858346

@Daimona I can deploy it on prod whenever, but deploying it on beta will take some unbreaking first. Are you ok with it just being deployed on prod for now?

In T320403#8405586, @Clement_Goubert wrote:

@Daimona I can deploy it on prod whenever, but deploying it on beta will take some unbreaking first. Are you ok with it just being deployed on prod for now?

Sure, I think that's fine! Also, I was thinking about it, and wanted to ask if the following ideas can be implemented:

• We're planning to deploy the extension on more wikis, and having to update the puppet class every time would not be ideal. Can the list of wikis be automated somehow?
• Running the script is only needed when the tzdata package is upgraded. Would there be a way to trigger the script when that happens, rather than it being cronjob-based?

Not a big deal if these aren't possible, I think the cronjob approach is sufficient for now, but I thought it was worth asking.

In T320403#8405639, @Daimona wrote:

• We're planning to deploy the extension on more wikis, and having to update the puppet class every time would not be ideal. Can the list of wikis be automated somehow?

There are two other ways of doing it that I can see:

• Maintain a dblist in mediawiki-config and run a cronjob with foreachwikiindblist
• Run for all wikis and check in your script if your extension is enabled, if not return immediately

• Running the script is only needed when the tzdata package is upgraded. Would there be a way to trigger the script when that happens, rather than it being cronjob-based?

While technically possible, the mechanism we use for updates makes that difficult. However, if it is that unfrequent that the script needs to be ran, can I move to a weekly rather than daily schedule?

Not a big deal if these aren't possible, I think the cronjob approach is sufficient for now, but I thought it was worth asking.

I agree, especially since the cronjob patch is essentialy done, and all other solutions would require much longer than just deploying the periodic job.

In T320403#8405891, @Clement_Goubert wrote:

• Run for all wikis and check in your script if your extension is enabled, if not return immediately

Right, the script kind of does that, but it exists with exit code 1. I guess that could be a problem, right?

• Running the script is only needed when the tzdata package is upgraded. Would there be a way to trigger the script when that happens, rather than it being cronjob-based?

While technically possible, the mechanism we use for updates makes that difficult. However, if it is that unfrequent that the script needs to be ran, can I move to a weekly rather than daily schedule?

Thanks, that makes sense. As for the schedule, you're right in that the script won't actually do anything useful basically every time it runs (time zone rules are updated roughly once a month). The point is, if we were to run it once a week, the times shown in the interface might remain wrong for the whole week. That's why I was suggesting daily, especially because for now the script is really fast.

Not a big deal if these aren't possible, I think the cronjob approach is sufficient for now, but I thought it was worth asking.

I agree, especially since the cronjob patch is essentialy done, and all other solutions would require much longer than just deploying the periodic job.

Yup yup, that's OK. Just one note: we're planning to deploy the extension on meta in a couple weeks (target date is 28-30 Nov), and we would need the script to run there as well. Enabling the cronjob for the current 3 wikis is not super-urgent, so we can wait until them to do all 4 in the same patch, or start with 3 now and add meta in two weeks, whatever works best for you.

In T320403#8407057, @Daimona wrote:

In T320403#8405891, @Clement_Goubert wrote:

• Run for all wikis and check in your script if your extension is enabled, if not return immediately

Right, the script kind of does that, but it exists with exit code 1. I guess that could be a problem, right?

If I run the script through foreachwiki, it's a bash for loop that will not break if the inner script exits in error. My only possible concern would be logging, but if I understand your script correctly it would at most log and error line per wiki not using CampaignEvents, wouldn't it?

However, I have to ask, how many wikis do you intend to deploy it to? If it's to a subset of wiki, having a dblist is a must, and then we can do a foreachwikiindblist referencing it. If it is to be deployed to all wikis, foreachwiki is probably a better solution.

• Running the script is only needed when the tzdata package is upgraded. Would there be a way to trigger the script when that happens, rather than it being cronjob-based?

While technically possible, the mechanism we use for updates makes that difficult. However, if it is that unfrequent that the script needs to be ran, can I move to a weekly rather than daily schedule?

Thanks, that makes sense. As for the schedule, you're right in that the script won't actually do anything useful basically every time it runs (time zone rules are updated roughly once a month). The point is, if we were to run it once a week, the times shown in the interface might remain wrong for the whole week. That's why I was suggesting daily, especially because for now the script is really fast.

Ah, right, I see how that could be problematic. I'll keep it daily then.

Not a big deal if these aren't possible, I think the cronjob approach is sufficient for now, but I thought it was worth asking.

I agree, especially since the cronjob patch is essentialy done, and all other solutions would require much longer than just deploying the periodic job.

Yup yup, that's OK. Just one note: we're planning to deploy the extension on meta in a couple weeks (target date is 28-30 Nov), and we would need the script to run there as well. Enabling the cronjob for the current 3 wikis is not super-urgent, so we can wait until them to do all 4 in the same patch, or start with 3 now and add meta in two weeks, whatever works best for you.

If you use a dblist, then I'll change the cronjob to reflect that, and then there's nothing more to do than adding the new wiki in the file at the same time as your deployment.

In T320403#8405586, @Clement_Goubert wrote:

@Daimona I can deploy it on prod whenever, but deploying it on beta will take some unbreaking first. Are you ok with it just being deployed on prod for now?

I've unbroken the wikimedia::maintenance host on beta, so it'll be deployed there too when merged.

In T320403#8408889, @Clement_Goubert wrote:

In T320403#8407057, @Daimona wrote:

In T320403#8405891, @Clement_Goubert wrote:

• Run for all wikis and check in your script if your extension is enabled, if not return immediately

Right, the script kind of does that, but it exists with exit code 1. I guess that could be a problem, right?

If I run the script through foreachwiki, it's a bash for loop that will not break if the inner script exits in error. My only possible concern would be logging, but if I understand your script correctly it would at most log and error line per wiki not using CampaignEvents, wouldn't it?

Yep, I thin so.

However, I have to ask, how many wikis do you intend to deploy it to? If it's to a subset of wiki, having a dblist is a must, and then we can do a foreachwikiindblist referencing it. If it is to be deployed to all wikis, foreachwiki is probably a better solution.

That's a very good question; I'm not 100% sure about our future plans, so CC'ing @ifried. Also, the plan is probably to use a shared database for most wikis, so on second thought it might be sufficient to run the script on meta.

If you use a dblist, then I'll change the cronjob to reflect that, and then there's nothing more to do than adding the new wiki in the file at the same time as your deployment.

I think for now it might be better not to use a dblist; either way, I guess we can just add meta to the puppet class when we deploy to meta, right?

In T320403#8409022, @Clement_Goubert wrote:

In T320403#8405586, @Clement_Goubert wrote:

@Daimona I can deploy it on prod whenever, but deploying it on beta will take some unbreaking first. Are you ok with it just being deployed on prod for now?

I've unbroken the wikimedia::maintenance host on beta, so it'll be deployed there too when merged.

Amazing, thank you! And just to confirm: the extension is installed on all beta wikis, but it uses a single shared database. So we only need the script to run on beta-meta.

In T320403#8409372, @Daimona wrote:

That's a very good question; I'm not 100% sure about our future plans, so CC'ing @ifried. Also, the plan is probably to use a shared database for most wikis, so on second thought it might be sufficient to run the script on meta.

Just to clarify, that is not the case at the moment in production, and it does need to run on the three original wikis testwiki, test2wiki and officewiki?

I think for now it might be better not to use a dblist; either way, I guess we can just add meta to the puppet class when we deploy to meta, right?

Yes, no problem, just ping me when you do and I'll add the puppet config.

I've unbroken the wikimedia::maintenance host on beta, so it'll be deployed there too when merged.

Amazing, thank you! And just to confirm: the extension is installed on all beta wikis, but it uses a single shared database. So we only need the script to run on beta-meta.

I amended the CR so it pushes the job for metawiki on beta.

In T320403#8409477, @Clement_Goubert wrote:

In T320403#8409372, @Daimona wrote:

That's a very good question; I'm not 100% sure about our future plans, so CC'ing @ifried. Also, the plan is probably to use a shared database for most wikis, so on second thought it might be sufficient to run the script on meta.

Just to clarify, that is not the case at the moment in production, and it does need to run on the three original wikis testwiki, test2wiki and officewiki?

Yep, because those wikis use their local databases.

I think for now it might be better not to use a dblist; either way, I guess we can just add meta to the puppet class when we deploy to meta, right?

Yes, no problem, just ping me when you do and I'll add the puppet config.

Thank you!

Change 858346 merged by Clément Goubert:

[operations/puppet@production] P:mediawiki::maintenance: CampaignEvents periodic

https://gerrit.wikimedia.org/r/858346

Sorry for the delay, it's been deployed.

Change 861813 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] mediawiki::maintenance::campaignevents: meta

https://gerrit.wikimedia.org/r/861813

I've prepared the change. Tell me when you prod on meta and I'll merge it.

@Clement_Goubert Hi, and thank you for the deployment :) The extension is now enabled on meta, so you can proceed with that one as well.

@Daimona The change is in review, I will deploy it when it's been +1'd. Thanks for the heads up.

Change 861813 merged by Clément Goubert:

[operations/puppet@production] mediawiki::maintenance::campaignevents: meta

https://gerrit.wikimedia.org/r/861813

Amazing, thank you!