Are you just talking about replacing fundraising_code_update/rsync_blaster, or further up the train to git with the separate MW codebase etc?

please loop in Release-Engineering-Team on this discussion. We will be changing some of the process around train deployments in the somewhat near future so we should take that into account when planning for a new deployment branch.

@Jgreen
We're talking about trying to get ourselves on the train of WMF-standardized deployments for MediaWiki core. I'm not sure how that would look when PCI gets involved. Do one of fr-tech (or -operations) have to pull the trigger for our deployments? We would still benefit from even a partial adoption of the train, where we synchronize security updates :-( and do our FR deployment of new wmf.N releases each week. Although this might just increase our maintenance burden...

@mmodell
Sure thing, thank you for noticing this task. We're trying to identify ways in which our process might be incompatible with the wmf-releng approach. A few things so far,

• We probably can't let deployments happen automatically because our payments MediaWiki server is in PCI scope.
• We have a handful of special extensions which are not deployed to the other wikis: ContributionTracking and DonationInterface. We would need special wmf-config declarations, I guess?
• We need to keep the ability to do immediate deployments and rollbacks of our extensions, ideally with no coordination overhead.

@awight: scap3 deployment would be the best bet I think. Regular mediawiki train deployments will eventually migrate to scap3 but maybe we should look at deploying payments wiki with scap3 even before we migrate everything else.

In terms of PCI, some of this hinges on whether we decide to stay SAQ A or change to SAQ A-EP, but in terms of the latter:

• Any host that can cause changes on a payments server is in scope, and needs the full suite of 2FA, malfeasance/malware detection, audit trail generation, etc. Any machine that can make changes on *that* machine (for example the laptops of users with deploy rights, production puppetmaster) needs all that stuff too.
• Using a web application firewall is allowed in lieu of third-party code review, but I think it we still need to be able to demonstrate that we're doing internal code review of changes before deploying them.
• We're required to do the external/internal penetration tests after every significant change.

Based on this, plus the stuff awight flagged about custom extensions, I agree that we should devise a process where fr-tech closely coordinates fundraising deployment with the production train. That should include a review and sign-off on changes, and regression testing.

What if we cut our own branch every Monday, based on the branch that stabilized the previous week? This would get us vetted code and allow us to make emergency patches to core.

In T130658#2353040, @cwdent wrote:

What if we cut our own branch every Monday, based on the branch that stabilized the previous week? This would get us vetted code and allow us to make emergency patches to core.

That's actually a really good path forward here.

As part of this we should also think about how we stage/test, as we got partway there but never really worked the staging step into the deployment process.

With the existing deployment tool, since we moved configuration to its own repository it would be easy to modify it to facilitate canary deployments. For example we would have Stable and Current branch checkouts and use a command line flag to deploy Current to one or more host. This is probably already possible with scap3.

In T130658#2353104, @demon wrote:

That's actually a really good path forward here.

Indeed. This also fits in nicely to our long-lived branch plans.

The train is not the only deployment process that adds changes to MW in production. What's your current process for deploying security patches? What about all our e.g. SWAT changes?

@Krenair - As far as SWAT deploy, it doesn't touch the FR infrastructure but we will deploy the same changes if they affect extensions we use or get backported to REL1_25. The latter case is another way we'd benefit from being more in sync.

I should probably not discuss our security patch process on a public ticket but suffice it to say it would be vastly simplified by using mainline MW :)

In T130658#2359609, @cwdent wrote:

or get backported to REL1_25

Err... Sure. Note that 1.25 is EOL this month.

@Krenair yikes, thanks for the heads up.