I just tested it again and got 500 Internal Server Error - Your request caused an error on the server. again. I am also not able to git pull from gerrit, so I don't think the key was deployed despite the error.

The extension does not distinguish between hCaptcha being unreachable due to the upstream being gone vs our proxy being gone. Therefore, I think that having blackbox monitoring of hcaptcha.com that does not go through our proxy would still have additional value. @jijiki let me know if I'm wrong :-)

The monitoring created in T404204 is sufficient -- see https://grafana-rw.wikimedia.org/d/441b2def-52e9-49d6-acad-91f5bb748989/hcaptcha-reverse-proxy-proxoid?viewPanel=panel-36 . Further work is not needed.

I am leaving this open as a reminder to delete the old key, but I'm currently unable to do that (blocked by T411816). If it's in the way, feel free to close it.

Not needed with the current setup.

Is anything still needed beyond the functionality in sudo cookbook -d sre.discovery.datacenter status all? That provides the following table:

Service                       Type           eqiad     codfw
=================================================================
apertium                      Active/Active  pooled    pooled    
api-gateway                   Active/Active  pooled    pooled    
apt                           Active/Passive pooled              
apus                          Active/Active  pooled    pooled    
citoid                        Active/Active  pooled    pooled    
config-master                 Active/Active  pooled    pooled    
cxserver                      Active/Active  pooled    pooled    
device-analytics              Active/Active  pooled    pooled    
docker-registry               Active/Passive           pooled    
echostore                     Active/Active  pooled    pooled    
eventgate-analytics           Active/Active  pooled    pooled    
[...]

In T411191#11413343, @ssingh wrote:

But there are more considerations than this. Given that this setup is not like a typical LB (or Liberica), we don't have the concept of depool thresholds and such, and therefore we have to be careful in case we end up depooling everything (what if the upstream service is temporarily down and we end up depooling both hosts in a site?). And while there is that fallback mechanism to the old system, this is something to keep in mind.

Thank you for tagging this task with good first task for Wikimedia newcomers!

Thanks @Jhancock.wm , looks good!

Never mind, the mtail-provided metrics appear to be sufficient.

Due to the sensitive nature of the proxy configuration, I am leaning towards the option that requires less configuration within nginx, which happens to be nginxlog-exporter. As the only thing it can see is the logs, it is easier to verify that no sensitive user information is being leaked anywhere.

In T407094#11357931, @Nahid wrote:

It looks like the dot < . > at the end of the public key is missing in the patch. The dot is actually part of the key.

There is a space before the dot in the key as pasted in the task description. The key encoding does not allow spaces, so if the dot is part of the key, then the key as pasted cannot be correct. Can you please paste the correct key?

@Jhancock.wm great, thanks for the update!

Done, ping me in case of trouble :-)

Done, let me know if something isn't working :-)

In T408164#11305956, @BTullis wrote:

@JMoore-WMF has supplied me with an SSH key via our authenticated WMF Slack org, but said this:

I just realized that the public key i included in the ticket was the wrong one- accidentally included the one for Wikimedia cloud SSH access
will update the ticket accordingly

I had previously run the cross-validation against this key as shown in T408164#11305495 so I am confused as to why this didn't get picked up.
It could be one of either:

1. The cross-validate script didn't work correctly.
2. The key isn't actually used in WMCS, although perhaps it is/was intended to be used there.
3. I executed the check incorrectly.

Done, @Virginie.caplet let me know in case something doesn't work :-)

@BTullis can you please approve the analytics-admins access? Thanks!

In T408008#11303431, @DMburugu wrote:

Yes, I approve this.

Thanks @Jhancock.wm, appreciated! No worries, this is really not urgent.

@DMburugu can you please approve this? Thanks!

In T407094#11302712, @Raine wrote:

@SKaram-WMF just to confirm, do you need SSH access or only dashboards and such? Thanks!

@SKaram-WMF just to confirm, do you need SSH access or only dashboards and such? Thanks!

Noting that for analytics-privatedata-users, "Explicit approval is not required for WMF or WMDE Staff." (T381824, T370424).

@Virginie.caplet I assume your developer account username is vicaplet-wmde, correcting it.

confirmed key oob

@mark can you please approve this from the SRE side? Thanks!