All done! @Gehel merged and did the deploy today: https://twitter.com/wikimediatech/status/877940072477319168

Looking back through IRC logs, people have definitely used this function manually (although I'm not sure why, exactly :)). I would argue we keep this function, though. Having some extra sloc for the flexibility to move a single server to a wikiversion that has never had its l10n compiled on tin is a good trade off, I think.

In T165748#3369771, @Gehel wrote:

@thcipriani Thanks for all this! I can be available to deploy this today 6pm CEST (9am PDT) if that's good for you. Otherwise, ping me on IRC and we'll find a time.

@Gehel @EBernhardson so I've made patches to migrate logstash/plugins over to scap3 that is setup more or less the same as the trebuchet deployment.

From the scap side of things the code looks fine. Looks like this will require some fiddling with pip and tox to get tests passing again :(

This is no longer happening afaict and it seems like the main followup is on T168036: rebuildTermSqlIndex.php causing beta-update-databases-eqiad to timeout and abort.

Landed D677, will close this task when new scap version (3.6.0) is released.

In T166888#3361686, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2017-06-19T22:14:36Z] <thcipriani|afk> Added non-voting operations-puppet-tests-docker job for operations/puppet repo, should (hopefully) be fast, and will timeout after 1 minute if it's not. More info https://gerrit.wikimedia.org/r/#/c/360091/ + T166888

In T167833#3346617, @hashar wrote:

I tried with either:

systemctl start zuul
/etc/init.d/zuul start

Maybe restart did the trick? Thank you @Dzahn !!!!

hrm, looks like $EXT_NAME is setup in parameter_functions.py for extensions:

In T129148#3336047, @greg wrote:

Status?

In D677#13334, @mmodell wrote:

Yet another reason we need to implement some sort of dependency graph in scap.

Why? Here's the scenario for Phabricator:

Updating phabricator involves the following steps:

1. Disable puppet
2. Stop PHD daemons
3. stop apache
4. update the phabricator code
5. apply database migrations
6. start apache
7. start phd
8. enable puppet
   
   Those steps need to happen more or less in that order although a couple of them could be swapped without causing any serious problems.
   
   The config format, even with multiple managed services, still isn't sufficiently expressive to describe this sort of simple multi-step deployment process.

In T166888#3322963, @faidon wrote:

These are all conjectures of mine, just by looking at the log, so correct me when I'm wrong :)

In D677#13289, @Paladox wrote:

Isent this https://phabricator.wikimedia.org/diffusion/PHDEP/browse/wmf%252Fstable/scap/checks.yml;d65f799196740f9bd9d4d532b2900f6c74516b1c$2 something similar to what this patch here is trying to do?

D677 should handle multiple service restart/reloads.

Check out the --limit flag. That may be all you need: https://github.com/wikimedia/scap/blob/master/scap/targets.py#L50-L62

Hi @fgiunchedi just tagged debian/3.5.8-1. Could you update scap on carbon please? And thank you :)

This is definitely something that is on the back burner but would be nice to have

wrong workboard :)

In D399#13204, @demon wrote:

This still good to land?

I agree with the idea that we should be touching IS after all the syncs that are meant to change things on the cluster; however, I think the --beta-only-change came out of a problem with how HHVM handled reclaiming TC cache space (i.e., exploding) T149872 and T103886#1401890 both talk about this.

now available from integration at https://ci-staging-docker-registry01.wmflabs.org

In T166345#3307356, @Gilles wrote:

mw-parser-output making it into the loggedout parser cache is reproducible on VM. The history of that issue is too complicated for me to figure out if that's intentional in wmf.2 (in which case the config hook needs to go) or accidental (in which case the mw-parser-output leak into the loggedout parser cache needs to be fixed).

In T166345#3307344, @Gilles wrote:

Removing the config hook would probably make the slowdown issue go away, but parser cache for loggedout would get "polluted" by mw-parser-output again. I don't know if the latter is a big deal.

@Gilles the mw-parser-output change in CommonSettings was created as part of https://wikitech.wikimedia.org/wiki/Incident_documentation/20170512-API can/should be removed now.

Unlicking this cookie D503: Clean up the deployment host provides needed functions for this, and does some tag cleanup, but not the actual gc-ing.

In T166345#3301074, @Gilles wrote:

Sounds worth a try, that would also explain why group0 and group1 are fine at the moment. I'm dealing with a sick child, though, so I'd rather not venture into that kind of live hacking myself in the next 3 hours if I can avoid it. Hopefully @aaron or @Krinkle appear before then

Had some trouble with the make-wmf-branch script. It seems that the AccountAudit extension repo was set to read-only in gerrit: https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/AccountAudit

Will branch cut for 1.30.0-wmf.3 today in an optimistic attempt keep branching on schedule; however, I will not deploy it anywhere until T166345: wmf/1.30.0-wmf.2 performance issue for Wikipedias has been resolved and 1.30.0-wmf.2 is live on all wikis.

In T166345#3298245, @Gilles wrote:

There are no xhprof runs in xhgui for the initial deployment window when the performance regression happened. With the attempts to reproduce on single machines failing, I think we'll have to plan a wider re-deployment to be able to profile this as the issue is happening.

Tested on beta cluster and all seems to be working. @hashar has volunteered to babysit the initial deployment. Now just need an opsen around for puppet wrangling during the deployment to production. @akosiaris or @Joe can you help merge these changes/get jobrunner off trebuchet?

In T166492#3297774, @hashar wrote:

@thcipriani you have clones of mediawiki repositories on deployment-tin in /home/thcipriani/mwclonetest. I guess that can be safely deleted.

To recap where this issue left off yesterday:

In T166345#3293922, @Marostegui wrote:

I believe this would merit an incident report to find out exactly what happened?

setting as UBN! since we're stuck on wmf.1 until we can figure this out.

In T165115#3267695, @Anomie wrote:

In T165115#3266238, @Legoktm wrote:

I believe this is all fixed now, but I'll let @Anomie confirm.

Should be, yes. For example, https://en.wikipedia.beta.wmflabs.org/w/api.php?action=mobileview&page=Asian+black+bear&sections=all&prop=text|sections looks correct now. Do the Mobile people need to do more testing?

The initial step is composer dependencies, so it sounds like all wikidata extensions's composer dependencies need to be tracked somehow in mediawiki/vendor.

In T162967#3271571, @bd808 wrote:

In T162967#3271249, @Addshore wrote:

In T162967#3269922, @Osnard wrote:

Yes, me :) I've also set up my own docker-based dev environment and would love to learn what other people are doing.

Awesome, there will be at least 3 of us then!
I would be keen to get some sort of docker-compose.json file merged into core at this hackathon for super easy dev environments!

You should talk to the Release-Engineering-Team folks about that general topic. They are talking/thinking about Docker related things for Foundation use generally.

tested in a limited way, works fine.

CI staging was allocated as a labs project and we've been using it for pipeline work, this is complete.

@mobrovac could you clarify a few of your requirements + rationals in comments on the google doc?