cc @Ragesoss: I think Mod Tools can handle this migration, specifically @jsn.sherman who did two similar upgrades recently. He and the others here on Mod Tools (apart from me!) are quite good at Python and may tackle a Django upgrade as well.

Let us know if there are any questions or concerns.

I spent some time looking over the code and what's on the box. It mostly makes sense, but I do wonder: from the code, it looks like the chobs stuff all expects elasticsearch, but I didn't immediately find any service that's providing it. Is the elasticsearch stuff vestigal or did I miss something? looks like the api handler just degrades gracefully when it's not there.

Also, how do we feel about bringing this into our gitlab? There's no linting or CI currently, and I'd feel a lot better with that stuff up and running. I figure we might as well start where we know we can continue since there's nothing in github to port over.

Moving to gitlab makes sense to me @jsn.sherman . I'm not aware of how it would use elasticsearch. The codebase we inherited had a lot of features that don't need for the applications we support with it, and that might be one of them.

jsn opened https://gitlab.wikimedia.org/repos/modtools/wikiwho_api/-/merge_requests/1

Containerize WikiWho for rootless docker compose deployment

In T432147#12121845, @Ragesoss wrote:

Moving to gitlab makes sense to me @jsn.sherman . I'm not aware of how it would use elasticsearch. The codebase we inherited had a lot of features that don't need for the applications we support with it, and that might be one of them.

In T432147#12122112, @CodeReviewBot wrote:

jsn opened https://gitlab.wikimedia.org/repos/modtools/wikiwho_api/-/merge_requests/1

Containerize WikiWho for rootless docker compose deployment

Okay, I'm proceeding in gitlab for now. The only part of the project that still touches github is the WikiWho algorithm and the inert wikiwho_chobj, which was the piece that wanted elasticsearch.

I rejiggered WikiWho into a regular ole dependency instead of a submodule. We can talk about what to do with wikiwho_chobj (prune it and relevant code, vendor it, make it an optional add on that's not built by default, etc) after the vm migration.

Okay, I managed to revive the test suite and get it plugged into CI; I also pulled a couple of empty pickled objects off of prod and stuck them in tree to run a binary compatibility smoke test as part of the unit suite. This should help protect us from accidentally invalidating several TiB of cached objects as we move forward.

I realized @MusikAnimal wasn't in the modtools gitlab group yet, so invitation sent.

musikanimal merged https://gitlab.wikimedia.org/repos/modtools/wikiwho_api/-/merge_requests/1

Containerize WikiWho for rootless docker compose deployment

jsn opened https://gitlab.wikimedia.org/repos/modtools/wikiwho_api/-/merge_requests/2

deploy: filter docker prune to spare fresh builds

jsn merged https://gitlab.wikimedia.org/repos/modtools/wikiwho_api/-/merge_requests/2

deploy: filter docker prune to spare fresh builds

Planning on doing this Wednesday, July 22. I'm setting aside 2 hours for us to either cutover or to call it a failure and defer cutover.

• 1:00 – 3:00pm America/Chicago
• 6:00 - 8:00pm UTC

things are moving, but slowly; I'm doing a chown over those pickle volumes which is taking some time. We may go over our window, but it's kind of a fixed cost because of storage performance.

Note that the django database moved and we validated (on accident) that pickles were resolving, but we're still waiting on chown to finish fixing up permission across 2/3 stores. Web proxies are moved over, services are configured, but we have them all shutdown pending the chown. I'll check in periodically and update here when things are back up and running.

chown finished on another pickle store; that's pickles-02 and pickles-03 done. We're just waiting on pickles now. (note that I kicked them off in parallel since it's 3 separate volumes)

We're seeing CORS errors today on Wiki Education Dashboard, like this:

https://wikiwho-api.wmcloud.org/en/whocolor/v1.0.0-beta/KIR2DL1/0/?origin=*' from origin 'https://dashboard.wikiedu.org' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Related to the rebuild?

The web proxies are connected to the new host, but I have all of the services down on it, so I find that not surprising. Let's recheck when we're back up.

Okay, we're back up and running; expect things to be shaky as the box warms up and tries to handle requests. @Ragesoss can you check your dashboard?

jsn opened https://gitlab.wikimedia.org/repos/modtools/wikiwho_api/-/merge_requests/4

deploy: post-cutover cleanup

I'm seeing some jobs succeed and a lot fail with the smell of binary incompatibility. Initially, I was worried that I did this, but I don't think so:
AttributeError("'Wikiwho' object has no attribute 'spam_hashes_set'")

{F95340514}
looks to have been introduced a while back:
https://github.com/wikimedia/WikiWho/commit/359337c68754945343f87794f702cab4722235de

that means lots cache from before may 6 is invalid. I should be able to patch in backwards compatibility for the old binaries by rebuilding the value from spam_hashes

jsn opened https://gitlab.wikimedia.org/repos/modtools/wikiwho_api/-/merge_requests/5

pickle: backfill spam_hashes_set on legacy loads

jsn merged https://gitlab.wikimedia.org/repos/modtools/wikiwho_api/-/merge_requests/5

pickle: backfill spam_hashes_set on legacy loads

okay, looking better with the patch. before:

note that I initially set the env to "production" during vm setup, but I should have set it to "main" for this project. It didn't do any harm, but deploys weren't going to get through that way. I updated the instance metadata and the patches are auto deploying as expected.

In T432147#12148263, @jsn.sherman wrote:

Okay, we're back up and running; expect things to be shaky as the box warms up and tries to handle requests. @Ragesoss can you check your dashboard?

It's working again. Thanks!

Thank you so much for this, Jason! I learned a lot.

note that I initially set the env to "production" during vm setup, but I should have set it to "main" for this project. It didn't do any harm, but deploys weren't going to get through that way. I updated the instance metadata and the patches are auto deploying as expected.

I thought you were referring to a Django env var akin to Symfony's APP_ENV, in which case something like production made sense. I should have asked! Oh well, no harm no foul.

If you want, we can create a production branch so that it matches other Mod Tools repos. That might prevent accidental deploys and such.

In T432147#12148394, @MusikAnimal wrote:

Thank you so much for this, Jason! I learned a lot.

note that I initially set the env to "production" during vm setup, but I should have set it to "main" for this project. It didn't do any harm, but deploys weren't going to get through that way. I updated the instance metadata and the patches are auto deploying as expected.

I thought you were referring to a Django env var akin to Symfony's APP_ENV, in which case something like production made sense. I should have asked! Oh well, no harm no foul.

If you want, we can create a production branch so that it matches other Mod Tools repos. That might prevent accidental deploys and such.

Nah, we only have one project that works that way, and that's basically for historical CI reasons. We'll need to exercise discipline with merging code. Before merge, any patch should:

• include test coverage for the change
• be attached to a phab task that's tagged with moderator tools team and ideally gone through our code review process

When we do need to hotfix something, still create an MR. Maintainers can set an MR to automerge after CI passes. Direct push to main is disabled.

After merging, QA should happen. Patch-specific stuff should be validated, and I'm also working up a baseline health punchlist of things to check for any deploy.

We have pretty much everything except secrets in version control now, so even adjusting the cron schedule is done through an MR. We're shrinking the need to shell in and touch anything on the server.

Also, I added @Ragesoss as a maintainer on the project.

Before marking as resolved, file a ticket to reduce quota again, after deleting the old instance.

musikanimal merged https://gitlab.wikimedia.org/repos/modtools/wikiwho_api/-/merge_requests/4

deploy: post-cutover cleanup