In T199219#5396104, @Ladsgroup wrote:

The other thing I want to mention and was missing here is overhead of encryption and TLS handshakes. In the @BBlack's example, we still use TLS but if you use plain http request, it's considerably faster (in both overhead of encryption and decryption):

ladsgroup@mwmaint1002:~$ time curl -H 'Host: www.wikidata.org' 'http://appservers-ro.discovery.wmnet/wiki/Special:EntityData/Q7251.ttl?revision=992109551&flavor=dump' > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  123k    0  123k    0     0   508k      0 --:--:-- --:--:-- --:--:--  510k
real	0m0.256s
user	0m0.008s
sys	0m0.004s

Unless there's any reason to encrypt requests internally, I think this would help us greatly.

In T229287#5395752, @MSantos wrote:

In T229287#5395623, @akosiaris wrote:

Thanks for running this.
My only point is that the max CPU looks suspiciously close to 1, which is the default value in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/_scaffold/values.yaml#25. This could be artificially limiting the app and could explain the errors. If you have already set it to higher values during your benchmarking disregard the next sentence. Otherwise, you might want want to bump it (considerably, say 10) and rerun the benchmark.
Great work, thanks a lot!

I ran into that problem in the beginning, than I set the CPU limit to 2 and memory to 6GB. Do you think CPU should be even higher?

Thanks for running this.

LGTM. Naming wise I 'd say let's do failoid{1,2}001.(eqiad|codfw).wmnet instead of the less obvious tureis/roentgenium that we have now.

I see 'mathoid' => 'http://deployment-docker-mathoid01.eqiad.wmflabs:10044' so that last part has been addressed, so yes, we can close this.

restrouter was temporarily deployed in the staging cluster today. Deployment was rolled back as it was failing, trying to reach out to restbase on port 7233, where restbase does not listen on yet. As soon as we figure out the exact details of the migration plan this should be ready to go. Those are

This still leaves all the servers currently installed which have a MAC based SLAAC address i.e. they do not have interface::add_ip6_mapped. It seems to me that it would be useful to ensure interface::add_ip6_mapped is added to the standard module so it is applied universally. However we still have 965 machines in this state as such changing everything in one go may pose to high a risk.

In T229287#5386764, @MSantos wrote:

So, I finish the setup and started some preliminary tests with the endpoint /v1/feed/onthisday, you can see it in the following image.

In T229287#5384518, @MSantos wrote:

I am now facing an odd issue that seems related to the k8s instance I'm running. When hitting some endpoints I got the following error:

{
  status: 504,
  type: "internal_http_error",
  detail: "Error: unable to get local issuer certificate",
  method: "post",
  uri: "https://en.wikipedia.org/w/api.php"
}

It looks like a service-runner requirement that is missing, @akosiaris do you have any ideas why this could be happening? cc/ @Pchelolo

For posterity's and transparency's sake, pasting the answer I already gave to @MSantos via email

In T229236#5376652, @Ladsgroup wrote:

@Lydia_Pintscher @alaa_wmde So the current user agent of graphoid service is graphoid (yurik at wikimedia). Yurik has left Wikimedia for a couple years I think. I made a patch to fix it but it fails because blubber version 3 is not supported anymore (does it mean we can't merge anything in graphoid right now? @akosiaris knows better)

hosts removed from puppet code, remove from puppetdb, certs revoked and VMs removed from the cluster. Resolving.

In T208566#5371633, @hashar wrote:

The Gemfile had Puppet 4.8.2 to match the version provided by Debian Jessie:

I 've uploaded to 4.10.2 today using https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/524525/ fwiw. I had not seen this task, sorry about that. Feel free to revert if I caused any problems (altough a fleet wide PCC said ok)

idp1001.wikimedia.org has been installed and is up and running, I 'll resolve this

+1. Thanks @Joe , thanks @Dzahn. I 've added both of you to the gerritadmin group.

Despite the designation as CI, we will be treating these uniformly as far as ganeti goes (we will handling the capacity allocations within ganeti) so:

Indeed the refreshes are for ganeti100[1-4] so row C it is. Try to spread them across 1G racks.

Your gerrit username (the one that is depicted in the username line on F29825695, can not change. That's something that is in gerrit, there is nothing we can do about that (aside from creating a new account, and even then deleting an account is not safe). What can change is the Full name (which is what is used in most screens/dashboards), which should happen when clicking that reload button, after which you should be kicked out of gerrit and forced to login again.

Lowering priority from UBN, since that was on Jun 19th and it's been 4 days already.

I think the issue is on the stream-config.yaml file, not the config.yaml template. Using .Files.Get means the file is taken as is and not invoked as template. I 've uploaded a change to fix that.

poolcounter1004 has just been added

Sigh, this was already done. I just hope the info added will be useful at some point in the future as a guide

• emptying ganeti1006 will require some 15-30 mins of advance notice. Emptying it (in case me or @MoritzMuehlenhoff are not around) can be done per https://wikitech.wikimedia.org/wiki/Ganeti#Node_operations using:

sudo gnt-node migrate -f ganeti1006

• emptying ganeti1008 will require some 15-30 mins of advance notice. Emptying it (in case me or @MoritzMuehlenhoff are not around) can be done per https://wikitech.wikimedia.org/wiki/Ganeti#Node_operations using:

sudo gnt-node migrate -f ganeti1008

• emptying ganeti1005 will require some 15-30 mins of advance notice. Emptying it (in case me or @MoritzMuehlenhoff are not around) can be done per https://wikitech.wikimedia.org/wiki/Ganeti#Node_operations using:

• emptying ganeti1007 will require some 15-30 mins of advance notice. Emptying it (in case me or @MoritzMuehlenhoff are not around) can be done per https://wikitech.wikimedia.org/wiki/Ganeti#Node_operations using:

conf1001 is fine to powerdown (no depool necessary), perform all wanted actions and then poweron as it will repool itself automatically

Agreed with @Krenair, closing for now.

Good to know. Sorry for misunderstanding this!

I 've brought this up in the weekly SRE meeting. Overall there's a number of concerns. I 'll be listing them below in no particular order

I think we can resolve this, right? I am gonna be bold and resolve it, feel free to reopen if needed

In T226236#5345617, @hashar wrote:

Thanks, I can confirm the component is around and it addresses the concern of mixing up upgrades with Toolforge. However that imports 18.09.7 but we need the previous version 18.06.x for now :-\

LGTM

Is this still ongoing? Do I understand correctly that the change to Resolved status was erroneuous? Should we reopen?

Changed merged, this should be resolved now.

Indeed. done. Thanks!

In T227529#5336709, @Aklapper wrote:

^ Worth to document this somewhere™ (or in more places) to avoid repeating?

In T227529#5335623, @bd808 wrote:

In T227529#5333480, @akosiaris wrote:

I am resolving this, feel free to reopen is something is amiss

The cn and sn for uid=waldir,ou=people,dc=wikimedia,dc=org are both waldyrious (lower case w). Wikitech will never be able to authenticate as MediaWiki will canonicalize the username to start with a capital letter W and wikitech is configured to enforce same case matching for user account lookup.

In T228196#5342893, @thcipriani wrote:

For that particular image I can recreate locally:

$ docker pull docker-registry.wikimedia.org/releng/composer-test:0.1.7-s1
0.1.7-s1: Pulling from releng/composer-test
8d22d214682d: Already exists 
dd5d82f356b7: Already exists 
e74dee1208c4: Verifying Checksum 
69208455aa1f: Download complete 
f1cba75babe0: Verifying Checksum 
2cd12524c0dc: Download complete 
3d3adb31207d: Download complete 
e17ba03e55ec: Waiting 
e9dd2befc159: Verifying Checksum 
filesystem layer verification failed for digest sha256:e9dd2befc159629b6a09232c6478fa48bedfc34117be1a04c1e337ebd0a46d27

In T224794#5339362, @wiki_willy wrote:

@akosiaris or @Volans - we can order drive replacements for this, since it's out of warranty, but I was trying to figure out how this correlates with the new replacement of backup1001. Do you need replacement drives on helium, to be able to complete the migration of data over to backup1001? I'll follow up on IRC with you later tonight as well. Thanks, Willy

User has been added to the cluster. Resolving, feel free to reopen

In T211881#5339132, @greg wrote:

In T211881#5332195, @akosiaris wrote:

the hardware and the Operating System powering this service are going out of support.

What's the timeline for that?

In T198939#5336414, @MoritzMuehlenhoff wrote:

In T198939#5336152, @jcrespo wrote:

What about the puppet database on m1?

The database should all be ephemeral data about past server state, so no need to retain, but adding @akosiaris for confirmation given he's the primary upstream author.

Sure, just done. I also removed the @wikimedia.org one, I hope that's what you wanted.

I am resolving this, feel free to reopen is something is amiss