I had a quick look, I agree with the above investigation (I reproduced as well) and can add (in case it's not obvious) that for some reason libenchant seems to prefer on ores1009 aspell instead of ispell. It is become even more complicated because of hunspell usage as well, lower down strace. I dug into it for about 1 hour or so without an obvious cause/solution being revealed to me.

I 'll remove myself from this if you don't mind. I guess I was added because of T240175#5723047 but that was an answer to a very cut and clear question. Overall, I honestly I have no idea what the project is about, nor how I could be of help (at least for now). Feel free to readd me if you disagree.

I 've deleted that instance. It's been a long time since we last used it and is probably not particularly useful right now. I 'll resolve this.

I should probably be removed as an admin. I have no usage of it and no idea what that machine is.

Feel free to delete machines and project.

For what is worth, I have no usage of these machines nor the project.

In T238048#5712471, @jcrespo wrote:

Hi, @akosiaris, thanks for the reviews and feedback. Could I have further your thoughts on T238048#5701519 and T238048#5701534.

In T238048#5711820, @jcrespo wrote:

"Offsite Job" seems to be correctly configured as "Copy", but it is not showing any activity. Needs checking.

In T238048#5701534, @jcrespo wrote:

Same for bast1001:

29-Nov 13:19 backup1001.eqiad.wmnet JobId 162657: Start Restore Job RestoreFiles.2019-11-29_13.18.59_50
29-Nov 13:19 backup1001.eqiad.wmnet JobId 162657: Using Device "FileStorageArchive" to read.
29-Nov 13:19 backup1001.eqiad.wmnet-fd JobId 162657: Ready to read from volume "archive0003" on File device "FileStorageArchive" (/srv/archive).
29-Nov 13:19 backup1001.eqiad.wmnet-fd JobId 162657: Forward spacing Volume "archive0003" to addr=323517936212
29-Nov 13:19 dbprov2001.codfw.wmnet-fd JobId 162657: Error: openssl.c:78 Encryption session provided an invalid symmetric key: ERR=error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error
29-Nov 13:19 dbprov2001.codfw.wmnet-fd JobId 162657: Error: openssl.c:78 Encryption session provided an invalid symmetric key: ERR=error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed
29-Nov 13:19 dbprov2001.codfw.wmnet-fd JobId 162657: Error: openssl.c:78 Encryption session provided an invalid symmetric key: ERR=error:0607A082:digital envelope routines:EVP_CIPHER_CTX_set_key_length:invalid key length
29-Nov 13:19 dbprov2001.codfw.wmnet-fd JobId 162657: Error: restore.c:741 Failed to initialize decryption context for /srv/tmp/srv/home_pmtpa/aaron/.gitignore
29-Nov 13:19 dbprov2001.codfw.wmnet-fd JobId 162657: Error: openssl.c:78 OpenSSL digest Verify final failed: ERR=error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error
29-Nov 13:19 dbprov2001.codfw.wmnet-fd JobId 162657: Error: openssl.c:78 OpenSSL digest Verify final failed: ERR=error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed
29-Nov 13:19 dbprov2001.codfw.wmnet-fd JobId 162657: Error: openssl.c:78 OpenSSL digest Verify final failed: ERR=error:04091077:rsa routines:INT_RSA_VERIFY:wrong signature length
29-Nov 13:19 dbprov2001.codfw.wmnet-fd JobId 162657: Error: restore.c:1680 Signature validation failed for file /srv/tmp/srv/home_pmtpa/aaron/.cache/motd.legal-displayed: ERR=Signature is invalid
29-Nov 13:19 dbprov2001.codfw.wmnet-fd JobId 162657: Error: openssl.c:78 Encryption session provided an invalid symmetric key: ERR=error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error
29-Nov 13:19 dbprov2001.codfw.wmnet-fd JobId 162657: Error: openssl.c:78 Encryption session provided an invalid symmetric key: ERR=error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed

@akosiaris does this ring any bell? I find hard to belive that a new version of openssl couldn't decrypt a file encrypted with an older version. Maybe I am using the wrong options.

Do you think we can keep using the server without replacing the disk or do we have to buy 1 disk and keep on site in case the disk failed

Problem fixed by the change above

In T236386#5706868, @Ottomata wrote:

@akosiaris I merged and applied https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/551610 in staging. My main app isn't coming up, and I suspect it is because it can't reach Kafka at the 9093 TLS port on the logstash hosts.

We haven't even started the process of refreshing those nodes and they host important services. I 'd rather we just replaced the disk.

need to be able to understand the pooled status

So it does look like it's service-runner specific then.

In T91649#5700798, @Tgr wrote:

In T91649#5656529, @akosiaris wrote:

Sentry is no longer open source (by the definition of OSI and by the admission of upstream as well). More at https://forum.sentry.io/t/re-licensing-sentry-faq-dAiscussion/8044.

To be more exact, Sentry is now business source, ie. eventually opensource with a non-compete restriction. (The code can be used in any way, except it cannot be used to offer paid error monitoring services; and it becomes proper Apache-2.0 three years after being published.) That is of course a no-go for any code that needs to be integrated with MediaWiki (using it would probably violate both the Sentry license and the GPL); using the Sentry server as a self-contained service should IMO still be considered. The license meets all the conditions for which we care about open source, and it's a reasonable balance between the spirit of open source (far closer to it than open core, which we do use) and not letting large commercial entities cannibalize your project. But of course that's a far larger discussion that would have to involve WMF Legal and TechCom at the minimum.

In T238658#5675793, @Ottomata wrote:

I can't seem to develop this locally due to a barrage of version problems.
From what I can tell, in prod we use.
k8s 1.12.9
helm 2.12.2
helmfile 0.66.0 2.
To use minikube with k8s 1.12.9, I tried installing minikube 1.1.0 (released at about the same time as 1.12.9).
I failed making this combo work locally (macOS). I think perhaps my Docker Desktop (with hyperkit?) has been keeping itself up to date, and no longer works with older kubectl? Not sure.

In T219923#5685137, @Pchelolo wrote:

Graphoid is likely going away, so we shouldn't work on this.

This was mentioned in T238890 (of which it is not a blocker) but we will anyway needs this for all services.

Just so that you aren't caught off guard

More investigation in T238823. It's quite possibly expected.

Could be totally different but with @jijiki we 've seen this behavior elsewhere as well. The latest installment is T238789. Per that logstash dashboard, it's the mediawiki's that send the RST to the echostore.

\o/. Thanks for taking care of this!

In T238792#5679882, @Mholloway wrote:

Looks like there are currently issues accessing docker-registry.wikimedia.org, which may be to blame here.

In T236386#5672742, @Ottomata wrote:

@Joe @akosiaris @ema I'd like to move forward with these patches this week, hopefully sooner rather than later. Can you find some time to review? I'll add them all to ticket description for easier reference.

krypton is no more since 7a36b4e7a94f486a400f0363c263c446c33bba80, resolving.

I 'll boldly resolve (no update since July), feel free to reopen

Should we resolve this?

This has been fixed in 3229da692ef3a003a860d6b0024c9ef4813ce13d. The reason production tagged releases are now again available is because we gave up on having swagger/openapi specs and removed helm.yaml (the thing informing the pipeline that integration tests should be run) in https://gerrit.wikimedia.org/r/#/c/mediawiki/services/zotero/+/538171/.

This has been done, resolving

In T238593#5674571, @ema wrote:

By going through SAL and the irc logs on #wikimedia-operations I've reconstructed the events as follows. There are some parts I don't understand so please fill the gaps.

• 2019-11-15 16:31 phab1003 has 169 busy apache workers according to this grafana dashboard which seems unusual. The number of busy workers is usually lower (below 20), but in the past it had been occasionally about as high (eg: it was 159 on 2019-09-21 at 00:00).

Adding some extra info as we 've discussed with @ssingh before this task was filed. After the discussion it became obvious that a ganeti VM would not be a good candidate for this, mainly because of the disk size requirement.

In T70113#5669912, @hashar wrote:

The alarm got removed via https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/550943/ for some reason ...

I think it's in the ~35mins "schedule" now, but other than that, it's still present https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-cluster=api_appserver&var-method=GET&var-code=200&from=1573809235233&to=1573845235233

I think we can solve this by just adding system:heapster role to our prometheus kubernetes group. I 'll try that approach but for now I am lowering to Low as it isn't causing an issue with metrics gathering (at least the metrics we currently rely on)

Namespaces and tokens have been created and populated. @Ottomata, you are clear for deployment. I am guessing after that we need LVS, discovery, public endpoint exposing.

I 'll be bold and resolve this since we now have fixed b). We can always reopen and evaluate if a) becomes an issue (aka I am going with 3.)

In T238048#5655633, @jcrespo wrote:

@akosiaris Could you give a quick look to see if these seems like a complete archive contents?
{P9597}

As noted in T91649 (duplicating here just for more dissemination), sentry is no longer open source (by the strict definition of the term). More at https://forum.sentry.io/t/re-licensing-sentry-faq-dAiscussion/8044

Sentry is no longer open source (by the definition of OSI and by the admission of upstream as well). More at https://forum.sentry.io/t/re-licensing-sentry-faq-dAiscussion/8044.

In T202982#5627916, @Pchelolo wrote:

Seems like it's been fixed, the only thing left to be done is to remove the hacky line from puppet.

All builds done on boron end up in /var/cache/pbuilder/result/* and we don't expire debs from there.

In T235013#5580957, @Dzahn wrote:

@Ladsgroup git-lfs is not installed on the prod servers and the puppet git:::clone class also does not support changing the command yet. So this breaks cloning on the prod servers.

In T236277#5631769, @Volans wrote:

@CDanis the problem is that all of those identify clients, while for the CA validation we're mostly interested in the server side. So while that surely would help, it's a 1:1 mapping. Also there might be places that have hardcoded the path to the CA cert for validation, either in the puppet repo or, potentially, in other repos too (as a default for example, dunno).
I don't know if this CA is also used in the k8s world for example.

In T236277#5631321, @jbond wrote:

In T236277#5631271, @akosiaris wrote:

IMPORTANT: The puppet CA cert (and correspondingly key), is used as a "master" (a failsafe in case the actual host key is not around) key for bacula backups. That is, if we lose it we won't be able to restore backups for hosts that no longer are around. This is documented under https://wikitech.wikimedia.org/wiki/Bacula#Restore_from_a_non-existent_host_(missing_private_key).

@akosiaris I only intend to update the public key with this change as such everything should still work post-change, or am i missing something else?

As @Joe said, that's expected. It's how misbehaving services are killed in order to recover. Here's also a breakdown in case anyone is interested

IMPORTANT: The puppet CA cert (and correspondingly key), is used as a "master" (a failsafe in case the actual host key is not around) key for bacula backups. That is, if we lose it we won't be able to restore backups for hosts that no longer are around. This is documented under https://wikitech.wikimedia.org/wiki/Bacula#Restore_from_a_non-existent_host_(missing_private_key).

Let a minor comment, namely let's keep helium around for a bit more.

Rolling it back. Per https://en.wikipedia.org/wiki/RDRAND and further tests, rdrand is already present in IvyBridge, which is what we pass as the base anyway.