Page MenuHomePhabricator
Create Task
Maniphest T374887

Log messages at ERROR level on http channel: Special:ExtensionDistributor unable to connect to https://graphite.wikimedia.org
Open, Needs TriagePublic
Actions

Description

After enabling ERROR logging for all channels on group0 wikis (T228838), errors like this appeared:

https://logstash.wikimedia.org/goto/a458bea8bae20b6c5597fc538ad54f8c

HTTP/1.1 - NULL cURL error 7: Failed to connect to graphite.wikimedia.org port 443: Connection timed out (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://graphite.wikimedia.org/render/?...

(not copying the whole message because it's weirdly detailed and I don't want to leak something)

The errors are logged when users access https://www.mediawiki.org/wiki/Special:ExtensionDistributor (or Special:SkinDistributor).

The volume is low, but I'm filing a task anyway for evaluation, since we were not recording these logs before.

Details

SubjectRepoBranchLines +/-
ExtensionDistributor: reach graphite via the service meshoperations/mediawiki-configmaster+2 -2
service_proxy: Add a listener for the http interface of graphiteoperations/puppetproduction+7 -0
Customize query in gerrit

Related Objects

Event Timeline

matmarex created this task.Sep 16 2024, 8:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 16 2024, 8:45 PM
Joe subscribed.Sep 23 2024, 1:10 PM

I'm not sure what is this extension doing, but http requests to our edge (so, using the public graphite url) will for sure fail, and even if we use the internal url, we'd need to open a firewall rule for it.

gerritbot added a comment.Sep 23 2024, 1:50 PM

Change #1075016 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] service_proxy: Add a listener for the http interface of graphite

https://gerrit.wikimedia.org/r/1075016

gerritbot added a project: Patch-For-Review.Sep 23 2024, 1:50 PM
gerritbot added a comment.Sep 23 2024, 1:57 PM

Change #1075017 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/mediawiki-config@master] ExtensionDistributor: reach graphite via the service mesh

https://gerrit.wikimedia.org/r/1075017

colewhite added a project: Observability-Metrics.Sep 23 2024, 10:37 PM
colewhite subscribed.
akosiaris subscribed.Sep 24 2024, 10:27 AM

This probably was to embed graphite powered graphs in MediaWiki regarding extension download numbers. And the URL hasn't worked in quite a while anyway.

Krinkle added a project: Unstewarded-production-error.Oct 10 2024, 4:04 PM
Krinkle moved this task from Untriaged to Sep 2024 on the Wikimedia-production-error board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits