Come hear useful lies about MediaWiki deploys (what we are doing today), our plans for a new "Pretrain" deployment group and workflow that has been in the works for a year or two, and even wilder ideas that we are thinking about for July 2026-June 2027 ("FY27") work.

Audience: MediaWiki developers, folks who like to think about deployment tooling, folks who use testwiki, folks who use Beta Cluster

Currently scheduled for Saturday 2026-05-02 at 16:45-17:15 in Space 5.

---

Raw session notes

Pretrain (How MediaWiki code gets deployed and things we are changing)

Speaker: Bryan Davis / User: bd808 - Principal SW, Developer Experiences Cloud Services Technical Docs Release Engineering QUality Services and Test Platform WMF.
Epic: T369112: Pretrain (née Group -1) QTE validation environment

A story about your MW Code
16956 patches merged and deployed to production in 2025 across Core extensions and WMF config repo. Making about 46 patches/day. We got to this point through a lot of years of iterations, to get deployments better and faster. Out of the 16956 about 2k+ were backports.

Lies about mediawiki deploys

• We deploy once a week on Thursday

To a lot of people this is all they see, we take 203 repositories, 8 skins MW core, branch them into 1062 wikis done 50 times a week. This is True but we deploy starting on Tuesday, ending on Thursday.
On Tuesday, 140+ wikis get the codes, Wednesday about 740, and the rest fo out on Thursday. Most times we have multiple versions of wikis running at the same time.

• We deploy every day except Friday, Saturday & Sunday.

We do backport deploys on any day during the Week. This is done about 3 times a week. We deploy all the time because sometimes there's emergencies and has to be fixed immediately for example security or data loss issues.

We tried to figure out how to deploy the code closer to the time the developer did it to ensure the memory is still clear.

Train blocking were about 58 in 2025.

Group 0 - Text wikis, MW.org, office wiki, wikitechs, with 25 blockers
Group 1 - More attention are on this but it, had the most blockers in 2025 with 31 blockers
Group 2 - All Wikipedias with 2 blockers

How we are breaking everything to make it better

Today when a patch is merged on Tuesday after the patch is put on the release branch, it waits a week to the next Thursday to be put on the train. train-stats.toolforge.org...

Average of 4 days between +2s and deployment to production. From patch submission to gerrit to deployment is about 6 days.

We are going trying to figure out how to deploy often, hence the inception of Pretrain.

The first pretrain is only going to be the test wiki which is theoretically going to be like the English Wikipedia which has a lot of extensions, gadgets,etc. A wide community uses it.

We want to run a micro train that just touches test wiki, picks up code that was written up until this morning is pulled and put into testwiki daily. The 3 backport windows would have the opportunity to affect the testwiki as well. The patch that is cut would be deployed by the next day UTC 2am. The hope is to get the deployment days closer to 1.

Completely automated using a system d timer that runs the scalp program, the deployment tooling responsible for gathering the code and pushing it to where they should be.

We hope to incrementally reduce the deployments to as often as possible.

As we learn to deploy faster we need to figure out how to decouple release from deployment. Deployment is moving source files from one place to another, release is the usage of the functionality of the source code for example with the use of feature flags.

WP: Thursday & Pretrain

What we don't want to do from a product perspective is not to make people sad often, we need to think about the feature flags once they get released how can we fix the application to ensure it does.

Today
Image builds nightly
Image updated on each scap
Design for routing at the CDN edge
Design for internal routing

Next
"Shadow" deployment
Testwiki is going to be the first wiki. This includes all the testwikis in there, probably invent an additional testwiki for example RTL and then put officewiki, wikitech, and mediawiki.org. The last 3 are the least impactful on the community and most used by people with technical experience to raise bug reports if anything goes wrong.

No changes to how long it takes code to English Wikipedia. We need to figure out how to make it safer to deploy often. The developer experience group and SRE are currently planning over this year and next on how to improve this.

Trainsperiment week ran 4 complete trains in one week. code. When that was done manually with humans, everyone who ran all the trains said they don't want to do it again. We need to work a lot on making the computers take the toil off the humans to make the deployment train easier.

Keep calm, keep deploying.
Questions

• Statistics on how many deployments to beta would have caused deployment blockers in the beta cluster

A: No specific value as the process of the code soaking in the beta cluster isn't well tracked. Data we have are pretty noisy, so its difficult to tell if its a causal change. Part of the reason for doing the deploys in pretrain is get production like deployments without deploying to production.

• Distribution of the 50+ train blockers, breakdown on how they were found?

A: 60% of the time the defect is found by the train conductor while watching the log volumes who raise alarms when they realise anomalies in the logs. More on the active monitoring and not passive waiting on the community to report. Train blockers are bugs that are so severe.

• There are two types of blocker at the group 2

A: We have the ability to rollback, but we don't want to do that and so we

• How about people who knowingly merge breaking changes as they hope to fix it in the next release.

A: We have to socialise the idea that this won't be the case anymore, if you have been relying on this we would help you figure out how to get use to the new approach. The more frequently we deploy, the smaller the issue deploys we would have as it would be easier to attribute and connect with the developer that can help resolve the issue.

• What type of tooling can we integrate with forges like gerrit for visibility?
• Q: If we shift focus between writing code, merge, and release
• We currently make a release note page that matches all the items that were released, but it doesn't track backports.
• Change log the inventory of the increment of deployment