Per T158583, our reprepro now supports multiple components. docker-engine is now moved to thirdparty/k8s for production. This allows having multiple versions of docker-engine around (the package is now now as docker-ce as pointed out by T153416#3076211). I think this partially solves the issues discussed above so resolving

oh dammit, I 've never killed that commit. Sorry about that.

This is now fixed per https://grafana.wikimedia.org/dashboard/db/prometheus-cluster-breakdown?from=now-15m&to=now&cluster=parsoid&orgId=1&var-datasource=codfw%20prometheus%2Fops&var-cluster=parsoid&var-instance=All&panelId=87&fullscreen as we are back to where we were before the switchover.

Package built and uploaded to both jessie-wikimedia and stretch-wikimedia

Yes this indeed has happened and it's true for all services. We meant to return to the normal state on Monday (yesterday) but we didn't for unrelated to this reasons. We will be doing so today.

All this seems pretty correct to me and does explain what we 've experienced pretty well

In T206654#4661589, @Platonides wrote:

I can reproduce it with the version downloaded from wikipedia:

$ wget "https://es.wikipedia.org/w/index.php?title=Usuario:Danielalfredo/Taller&oldid=111186880&action=raw" -O 111186880.txt
 

Plus the code of P7674 which is only a tiny variation from the original P7672.

Maybe when moving the wikitext from the wikipedia revision to a local file something 'fixed' it?

$ md5sum 111186880.txt 
e306383a70dae20a0d6451a619ee5af2  111186880.txt

The dumped memory when converted back to utf8 has the exact same MD5 as https://es.wikipedia.org/w/index.php?title=Usuario:Danielalfredo/Taller&oldid=111186880

I think I 've managed to reproduce the problem. After fetching and trying the last 500 revisions from https://es.wikipedia.org/wiki/Usuario:Danielalfredo/Taller and failing to reproduce it, I returned to the gdb process. After a very long fight understanding Tokenizer and with how the data is stored internally in Tokenizer values I 've managed to obtain the following

I 'm adding some extra gdb info in P7669

In T206654#4658539, @Halfak wrote:

I've demonstrated that ORES timeout mechanism works for an old mwparserfromhell recursion bug in version 0.4.4 based on this report: https://github.com/earwig/mwparserfromhell/issues/190

See my work here: https://gist.github.com/halfak/281a692873d538e77c6096f5042e8be2

In T206740#4658660, @jcrespo wrote:

Forgetting the codfw -> eqiad replication was the most likely cause of overload on the application servers (and on External storage hosts).

@Dzahn anything left here ?

Successfully switched (with some aftermath and actionables but successfully nevertheless) to codfw and back per the subtasks, I am resolving this.

Mediawiki and traffic were successfully switched yesterday, swift and services today. I 'll close this as resolved

In T206654#4656931, @awight wrote:

I found this config on production, so in theory our workers should be restarting themselves:

CELERYD_MAX_TASKS_PER_CHILD: 100

Unfortunately due to the venv I could not get (yet) the niceties of py-bt and py-list working but there's already an indication that this is busy looping parsing text. The ltrace output reinforces that idea

And gdb bt output

And now we got

In T205364#4642331, @Dzahn wrote:

Maybe it makes sense to prioritize T196478 instead?

I followed http://erikimh.com/megacli-cheatsheet/ to do so

In T205256#4619239, @Ladsgroup wrote:

The reason that everything is INFO is that our uwsgi encoder is designed to do so: https://github.com/wikimedia/puppet/blob/production/modules/service/manifests/uwsgi.pp#L166

In T195710#4627694, @Framawiki wrote:

This bug was publicly divulged in https://community.otrs.com/security-advisory-2018-05-security-update-for-otrs-framework/.

Can the present solved task be public ?

Yup, https://gerrit.wikimedia.org/r/463733 is meant to fix it

As far as solving the logstash URL I think the best approach would be to just have the entire list in a dashboard. I 've updated the "scap canary" dashboard. However for sharing purposes this generated a new URL https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040. I 'll update scap.cfg with it

https://gerrit.wikimedia.org/g/mediawiki/services/zotero/+/refs/heads/master would be the repository @Mvolz I just created it and gave it the same permissions as the zotero/translators repo in gerrit. Let me know if anything else is required.

Nice finding!

Upgrade completed successfully

https://community.otrs.com/security-advisory-2018-05-security-update-for-otrs-framework/ is also relevant and get's fixed by 5.0.30 as well.

This was impossible to happen, a new box was procured in T196478

@Mvolz, SRE has a question about this migration. Assuming this gets deployed successfully next quarter, this will allow us to migrate off the current zotero infrastructure and thus remove it from WMF, right ? Does this sound plausible ?

In T201611#4606202, @thcipriani wrote:

In T201611#4606083, @akosiaris wrote:

@thcipriani, @dduvall, nodejs 10 image built and uploaded. It's the "slim" variant, I was wondering if it is enough or if we also require the "devel" variant

We'll need npm to install the test dependencies and run the tests in the pipeline, so we'll need a -devel variant for that.

@thcipriani, @dduvall, nodejs 10 image built and uploaded. It's the "slim" variant, I was wondering if it is enough or if we also require the "devel" variant

Can't reproduce this. https://grafana.wikimedia.org/dashboard/db/phabricator?orgId=1&from=1536592054293&to=1537196854293 is pointing out that apache was restarted today so this is probably why. If this is indeed T182832, we should probably merged into it.

Packages built and uploaded to both stretch-wikimedia and jessie-wikimedia. Resolving, feel free to reopen.

@elukey VM is up and running. No role assigned in puppet so you probably want to handle that. Resolving this.

VE is functional on wikitech once more.

The switchover has happened successfully, I am gonna happily resolve this. For any issues that might have crept up because of the switchover please tag them with Datacenter-Switchover-2018

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=wtp1043&service=MD+RAID still complains btw.

The steps listed in the description look correct and sufficient to me. There is one thing to add and it would be the removal from DebMonitor per https://wikitech.wikimedia.org/wiki/Server_Lifecycle

https://grafana.wikimedia.org/dashboard/db/t204083?orgId=1 shows the excessive traffic moving around the various memcached hosts for the last 1 year.

In T203121#4574382, @thcipriani wrote:

Poking this for ETA.

This one should unblock us on graphoid as well as a add a builder to help support generic jobs in CI via blubber – we're eager to get it out.

In T204033#4574445, @bd808 wrote:

Have you talked to @Joe and @akosiaris about using the existing kubernetes-testing project for this?

$::mw_primary is removed from puppet now. Resolving this.