In T422544#11815046, @HouseBlaster wrote:

"Starting on November 2, 2023" appears to be a typo, unless you own a time machine ;)

How about:

@Trizek-WMF as discussed, we can use the same announcements referenced in T420748#11779097. Thank you for your help!

Declining this in favour of using T422544 directly.

In the end, we decided to use the old process for all wikis this time (see T419049#11796113 if you're curious about the reasoning). So we can use the exact same messaging as last time, including the same time frame, for all wikis.

Upon further discussion with DBA, we will fall back to the old process this time around.

Upon further discussion with DBA, we will fall back to the old process (i.e. just running the maintenance script in place) this time around.

Upon further discussion with DBA, we will fall back to the old process this time around.

Clarifying that on s6, only frwiki and ruwiki are affected by the ICU upgrade, so we are only going to do the procedure for those two.

In T419980#11793374, @Marostegui wrote:

In T419980#11793076, @Raine wrote:

1. I'd strongly suggest we do this gradually, it is dangerous enough to warrant it. As mentioned somewhere, probably wiki by wiki isn't doable, but the smaller the better, so the list of s3 wikis on the task description (minus ruwikinews) sounds good to me.

Yes, it needs to be (roughly) all at once. We can potentially wait a few minutes between the first table swap and the rest if that helps, but the time from the start (when ServiceOps deploy the new image) to the table swap is potentially user-visible, so we don't want to stretch it too much.

To be super double sure: we can do one section at the time and it doesn't matter if it takes days in between sections?

No, we would have to do all sections roughly at once. This is because we have to do all wikis at once (constraint on our side), and the time when wikis are done and tables are not is potentially user-visible, so we want to minimize it. That is why we are now only doing s3, and we are not using this process at all for the other sections. (We will upgrade the other sections via a long-running maintenance script that has more user impact but doesn't need DB changes.)

In T419980#11777371, @Marostegui wrote:

Sorry for the delay, I've been out for almost 3 weeks and I am catching up now.

No worries, and thank you for the below!

Many thanks @Scott_French <3

The finalized list of affected wikis and the updated timeline is on the task.

@Marostegui or @Ladsgroup, when can we do the table swap? Would one of you be available for the swap (coordinated with serviceops) on Apr 7 or 8?

Oh, and many thanks to @taavi and @Scott_French for connecting the problem to my patch so quickly <3

I would like to invoke The Oops Defense, and request a sticker :D

In T419980#11763489, @JAllemandou wrote:

@JAllemandou We will postpone the upgrade (i.e. the table hot swap). However, the new tables are already created, though they are not in use. Is that a problem for you?

I don't see how just having the new tables created (and even loaded for the sake of it) could be an issue. Thanks for warning us :)

In T419980#11736923, @JAllemandou wrote:

The sqoop jobs fleet starts at midnight UTC on the first of the month, and usually lasts 2 days and a half if everything goes well.
Thank you for considering scheduling your operation at a different time :)

@kostajh Could you (or someone from your team) update the parts of HCaptcha wikitech marked for PSI? The scope is incident mitigation and making sure oncallers have enough info to identify culprits and know what to do, so it can stay high level. I believe some of this is already documented in your design docs, but it would be good to have it in a place where oncallers can easily find it.

It's not clear what the dead link pointed to originally, but my best guess is that it was a dashboard for the MW deployment serving the API traffic. During an incident, I would want to see where the external REST and action API traffic lands. So I changed the dead dashboard link to point to the mw-api-ext dashboard.

I have reorganized https://wikitech.wikimedia.org/wiki/HCaptcha and added placeholders for missing/outdated info.

In T420748#11738120, @MLechvien-WMF wrote:

@Raine Based on past iteration, do we have more details about the expected disruption, what do we mean by out of order display, is it expected for all users the same way, during the entire duration?

In T419980#11737958, @Raine wrote:

In T419980#11735868, @AtUkr wrote:

A mass deletion of categories has just been launched on ruwikinews, with ~1.5M categories (out of ~2M) slated for deletion. It's going to take more than a few weeks to complete. I'm not sure how exactly it could affect this task, but a warning probably couldn't hurt. See the ruwikinews forum for more info.

@Ladsgroup or @tstarling, you have a better understanding of the MW side than me, do you know if this affects the new process, and if yes, what to do about it? Thank you!

In T419980#11735868, @AtUkr wrote:

A mass deletion of categories has just been launched on ruwikinews, with ~1.5M categories (out of ~2M) slated for deletion. It's going to take more than a few weeks to complete. I'm not sure how exactly it could affect this task, but a warning probably couldn't hurt. See the ruwikinews forum for more info.

In T419980#11732573, @Ladsgroup wrote:

In T419980#11732368, @Raine wrote:

In T419980#11732313, @Ladsgroup wrote:

My suggestion has been to do it gradually across different wikis.

This is not really feasible, as we will need to deploy the image with the ICU72 libraries just prior to swapping the table, and currently there is no mapping of image to wiki (we're serving all wikis from the same image), so we're not able to roll out the new ICU libraries wiki by wiki.

The only option I can think of is to swap the table _before_ we upgrade the MW libraries, but I do not know what would happen in that case -- that would mean serving ICU72 keys with ICU67 libraries, and I have no idea whether it's forward-compatible and what would happen if it isn't.

Then maybe this time, we do swap mode on small wikis and then old method for large wikis

In T419980#11732313, @Ladsgroup wrote:

[...] the script needs to populate the new table [...], I'm not sure enwiki would be done by then.

In T419980#11732313, @Ladsgroup wrote:

My suggestion has been to do it gradually across different wikis.

Looks like it's working! I'm getting a grumpy response from shellbox when I curl at it from another pod.

kamila@deploy2002:~/icu72$ kubectl exec -it mw-experimental.codfw.pinkllama-8578d75f87-jrkhp -c mediawiki-pinkllama-app -- php -r "`cat test-shellbox-conn.php`"
Connection successful, HTTP status: 500
{
    "__": "Shellbox server error",
    "class": "Shellbox\\ShellboxError",
    "message": "No action was specified",
    "log": [
        {
            "level": "ERROR",
            "message": "Exception of class Shellbox\\ShellboxError: No action was specified",
            "context": {
                "trace": "#0 /srv/app/src/Server.php(77): Shellbox\\Server->guardedExecute('/srv/app/config...')\n#1 /srv/app/src/Server.php(66): Shellbox\\Server->execute('/srv/app/config...')\n#2 /srv/app/index.php(3): Shellbox\\Server::main('/srv/app/config...')\n#3 {main}"
            }
        }
    ]
}

The new images look happy:

sh
kamila@deploy2002:~/icu72$ kubectl exec -it shellbox-icu72-c4f45f566-p2srn -c shellbox-icu72-app -- sh -c 'apt list | grep libicu'
libicu72/now 72.1-3+deb12u1~wmf11u1 amd64 [installed,local]

In T419980#11718195, @Ladsgroup wrote:

I won't be but Manuel will. OTOH, it'll be around the time of dc switchover, can it be moved to a bit later? e.g. first week of April.

I will discuss with Manuel (and schedule around data lake imports), thank you!

In T419980#11723874, @Ottomata wrote:

Hi! FYI it would be probably be good to avoid early in a calendar month, as this is when data is sqooped into the Data Lake. I think things would break if a table rename happens while big selects are running on the analytics replicas.

@daniel I was bold and edited the wikitech page. Ping me (and potentially reopen this) if you dislike anything, otherwise you can consider it reviewed.

In T419980#11713167, @Ladsgroup wrote:

• I think this needs to be done gradually. We have never done this in production and it could be broken since it was introduced. So please do it on testwiki for example, then swap and if it's working fine. Move to mediawikiwiki (s3) and then rest of s3 and then wikis in s5 and so on.

Yes, definitely going to do that. Thanks!