In T212129#4833043, @aaron wrote:

The current callers don't assume the level of durability as with mysql, just that the data will likely not be randomly removed (e.g. high eviction rate, power outage, network blips).

I like black too but from but from https://black.readthedocs.io/en/stable/installation_and_usage.html it tied to having python 3.6 installed.

Add codfw in the mix as well, no reason to cap this to eqiad. Everything else LGTM

Oops, closed this by mistake. Re-opened, feel free to close when the issue is indeed resolved.

In T210260#4820556, @hashar wrote:

Following the merge of https://gerrit.wikimedia.org/r/478200 , can you possibly rebuild the two images please? :)

Repository	Tag	Image id	Created	Size
docker-registry.wikimedia.org/wikimedia-stretch	latest	ac576ceda671	13 months ago	56.1MB
docker-registry.wikimedia.org/wikimedia-jessie	latest	a81cc7ec7998	13 months ago	80.4MB

Chart merged and is available at https://releases.wikimedia.org/charts/

The migration uncovered a number of issues in graphoid that make it worthwhile to consider a Code Stewardship request. That is done in T211881, stalling this until it is resolved.

Stalling until T211811 is done

I 've had to deannotate the zotero namespace with commands like the one below

I 've slightly amended the patch to remove the now defunct sc-admins group and merged the patch per the SRE meeting's approval. Resolving this

FWIW we 've had a number of minor outages and alerts resulting in increased latency for results. The corresponding graph can be seen here https://grafana.wikimedia.org/dashboard/db/restbase-external-overview?panelId=17&fullscreen&orgId=1&from=1544017415835&to=1544026700314

In T197242#4798702, @Pchelolo wrote:

Seems like after this has been done the citation alerts started flapping much more then they used to. Also, the mean latency for citations endpoint went up from seconds to minutes.

In T122676#4796844, @Ladsgroup wrote:

Hey,

• @akosiaris tested twemproxy in prod and it fails because celery issues redis transactions and twemproxy doesn't support redis transactions. Same goes with dynomite.

I 'll do so, thanks

I can reproduce it as well. Received sizes and execution times are not consistent, ranging from a few hundreds of byes to a couple of megabytes and a few secs respectively. This and more importantly the test done above by @fgiunchedi indicate something going awry in the communication between varnish and swift.

I did just do a quick check on wikimedia-stretch image for this

I am afraid we can't really change it. It's been at 06:25am (UTC in our case) forever and people expect that. Changing it would break the current expectations of people. Note that this is true for all services and software and it hasn't really caused an issue for a long time. So we should make a better job of surfacing and fixing the issues, not changing the logrotate schedule

In T210260#4784708, @LarsWirzenius wrote:

C.UTF8 does not exist. In every other locale I try, a UTF8 suffix is an alias to the UTF-8 suffix (with the dash).

This works: docker run unitest env LC_ALL=C.UTF-8 python3 -c "print('étoile')"

I'd suggest that we use the C.UTF-8 locale, but I see no strong reason to prefer it over the en_US.UTF-8 locale.

Does this mean he have a hard deadline of 2019-04-01 for completing the migrations? Or per the "I can backport security fixes for a while" we have a couple of more months? The current goal is that by July 2019 all scb services, restbase (and probably aqs as well), proton, parsoid will be in kubernetes. That will leave turnilo and aphlict I guess.

This has already been implemented (albeit not in celery but redis). https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/394022/2/modules/ores/manifests/redis.pp

@Cmjohnson, I think we can proceed with this. I did just try to reimage the server but mgmt is not responding

@akosiaris for ores2008

ores2001, 2*ganeti, 15*mw
cc @akosiaris to know what specific actions need to be taken for Ores and Ganeti

Box is reimaged and is up and running. megacli seems the controller and the disks

Solved the wrong task. I mean to resolve T196477

Child task resolved, resolving this as well

Finally resolved

I 've helped with the debugging. Starting from apertium it was clear something automated was POSTing a lot of requests to it. It turned out they were mostly for the rus|bel langpair but that was a red herring as it was just the snapshot in time I looked at. Moving from apertium to cxserver it became clear something was POSTing to /v2/translate endpoint. The things I noted were mostly about another language pair ca|oc but again that was a snapshot in time. Then a VM IP caught my eye, one that was of wcdo.wcdo.eqiad.wmflabs. I 've jumped into said VM and stopped a process that was clearly heavily hitting the cxserver API

Should we also merge https://gerrit.wikimedia.org/r/#/c/integration/zuul/+/465324/ and release 2.5.1-wmf6 ? I 'll upload it to apt.wikimedia.org and I would rather do this once.

In T201611#4772018, @Krenair wrote:

@akosiaris: I think your last paste is slightly broken (most of the URL got muddled with the data)

Finally deployed to production

In T209088#4759458, @thcipriani wrote:

In T209088#4745829, @akosiaris wrote:

I think we should support multiple tags per image (docker anyway does support that and they cost next to nothing on the registry level AFAIK)

• Keep the ${timestamp}-production (+1 to @LarsWirzenius about splitting date and time btw) as it's nice and monotonically increasing

+1

• Add a tag based on the zuul.commit SHA1, but only if we can be reasonable sure that it's immutable. My memory fails me, but I remember some objection to this in the last meeting, does anyone remember the specifics?

I don't recall any specific objections, but I may be mis-remembering. Since we have this running as a postmerge job it should be fine, I think.

• Possibly allow the developer to influence part of the process by supporting adding a tag on git commits that are tagged, allowing developers to implement SemVer (or any other kind of versioning scheme) if they so wish. That might or might not be the wisest decision on their part, but I think we should allow people to make that decision

For now we could do something like git tag --points-at HEAD and just add a tag based on that. In future we may want something fancier.

This has now been deployed to the kubernetes staging cluster.

Upgrade done, resolving

FWIW, I 'll echo @Ladsgroup and @fgiunchedi. Having the data is obviously useful. Representing them in grafana on the other hand it probably not so practical. I also have my doubts as to whether a graph would help identify the culprits of load spikes, mostly due to the nature of the service, but I am be at fault here.

@akosiaris labsdb1006/7 are involved -- and if we do this with a failover, I want to make sure the additional tables for T201544 are replicated? Can we confirm that?

I think we should support multiple tags per image (docker anyway does support that and they cost next to nothing on the registry level AFAIK)

In T209271#4741841, @fselles wrote:

After looking into it a little bit, packaging harbor would be challenging. Harbor is a set of microservices published as containers. The installation and dev guide refers to docker-compose as a strong requirement for running harbor components, in order to run this docker-compose we need to build the container images and hosted them in our own docker registry which seems a sort of catch-22 problem (other people could rely on downloading it from DockerHub).

In T209265#4742019, @Joe wrote:

In T209265#4742010, @akosiaris wrote:

IIRC this is because of the expand_data directive in https://github.com/wikimedia/puppet/blob/production/modules/puppetmaster/files/production.hiera.yaml#L8

It's confusing indeed. That being said @Joe has expressed an interest in unifying the hiera backends so maybe we can get rid of this behavior.

Not really, the idea was to *avoid* having huge files. We might get rid of the behaviour where it won't make the files unmanageable.

So doing some easy calculations:

$ find hieradata/common -type f | xargs wc -l 
...
 3082 total

IIRC this is because of the expand_data directive in https://github.com/wikimedia/puppet/blob/production/modules/puppetmaster/files/production.hiera.yaml#L8

In T122676#4732649, @Ladsgroup wrote:

Which now brings us to the question of what's the next step?

Upgrade completed successfully. Also checked with a SELECT * FROM version of the 2 sql statements displayed in https://community.otrs.com/security-advisory-2018-09-security-update-for-otrs-framework/ and no results were returned so no issues there.

@Papaul I 'd say ignore it. That system+disk self/array is scheduled for decomission, to be replaced with backup2001 (T196477). The data in it is a copy of the data from helium so we ain't gonna lose something if more disks fail. There is no point in maintaining. After talking with @MoritzMuehlenhoff on IRC it seems like we can do a fresh reinstall of backup2001/backup1001 next week with the new stretch point release and set up the service on them and then decomission this

For what is worth, the upstream task is https://github.com/celery/celery/issues/3500. Closed WONTFIX apparently.

In T182249#4730735, @Halfak wrote:

I like the proposal of depooling one datacenter. What do you think @akosiaris? Is this crazy?

In T199853#4727463, @Krinkle wrote:

For the record, if https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/472032/ is merged, the space occupied by xenon logs will increase (upto 2X).

Xenon occupies a mostly constant amount of space, but that mount may double from 25G upto 50G. This will essentially chip away at the space reserved for XHGui, which I estimated (in the task description) as being able to fit current and next 5 years of data. With this change in estimate for Xenon, that would instead accomodate (100G/2G per month) about 4 years instead.

FWIW, metrics_host: in config-vars.yaml, which is used by scap to build config.yaml, specifically the

This help generate F27065389

Indeed. Thanks!

In T204558#4685714, @Krenair wrote:

Sorry, reopening this one because one was missed (on account of being inaccessible when Cumin was being run to find all trusty instances): compiler.puppet.eqiad.wmflabs - it's still not responding to ping or SSH.
@akosiaris, apparently you set this up 4 years ago, any chance you can shed any light on why it's not responding despite being status active instead of shutoff?

Session notes added to
https://www.mediawiki.org/wiki/Wikimedia_Technical_Conference/2018/Session_notes/Identifying_the_requirements_and_goals_for_dependency_tracking_and_events