I haven't looked carefully yet, sorry (I will), but given that this seems to be spikes rather than a persistent issues, does this coincide with anything else during this period?

I see spikes of up to 10s happening from my laptop several times a day (tested with a simple python requests.get in a loop). So I believe that this is a problem with hCaptcha's infrastructure, not ours.

We should still look into mitigating it via caching in some way, so not closing the task yet.

In T418865#11672882, @Raine wrote:

I see spikes of up to 10s happening from my laptop several times a day (tested with a simple python requests.get in a loop). So I believe that this is a problem with hCaptcha's infrastructure, not ours.

We should still look into mitigating it via caching in some way, so not closing the task yet.

To clarify, you're running the requests through the proxy, or direct to hCaptcha's infrastructure? Can you share your script?

I was runnning the following:

URL = 'https://js.hcaptcha.com/1/api.js'

import requests
import time

def get_hcaptcha_js_time():
    start_time = time.perf_counter_ns()
    response = requests.get(URL,timeout=30)
    end_time = time.perf_counter_ns()
    return (end_time - start_time) / 1000000

def graph_bar(time):
    return '*' * int(time / 10)

if __name__ == '__main__':
    max_time = 0
    while True:
        curr = get_hcaptcha_js_time()
        max_time = max(max_time, curr)
        print(f'time {curr:5.0f}ms, max {max_time:5.0f}ms {graph_bar(curr)}')
        time.sleep(10)

I realised that since my laptop is on wifi, it's not really conclusive. I will rerun this from one of our hosts.

Thanks. I'd recommend pinging https://js.hcaptcha.com/1/73f27c192b38c05ce2ebce596a0e28f88a2a56bf/secure-api.js since that is the URL we are querying via the proxy.

In T418865#11673762, @kostajh wrote:

I'd recommend pinging https://js.hcaptcha.com/1/73f27c192b38c05ce2ebce596a0e28f88a2a56bf/secure-api.js since that is the URL we are querying via the proxy.

Thanks, done.

I've been running the script on urldownloader for about 6 hours so far and I have not seen the variability, so ignore the above, it may have been my laptop's wifi. Sorry about that.

I have a hypothesis: it is possible that this is caused by nginx's keepalive misbehaving. I will look into that next.

Hi,

With the end of quarter approaching the team has couple priorities to derisk, so it will be difficult to reach that investigation before next quarter.

If anyone on this thread needs immediate action please let me know.

Matthieu

Change #1249929 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/puppet@production] hcaptcha: Enable nginx caching for secure-api.js

https://gerrit.wikimedia.org/r/1249929

Change #1249929 merged by Ssingh:

[operations/puppet@production] hcaptcha: Enable nginx caching for secure-api.js

https://gerrit.wikimedia.org/r/1249929