Some sites try and fail to serve favicon.ico
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	matmarex
	Sep 17 2024, 6:34 PM

Description

Steps to reproduce: Visit one of these URLs:
https://donate.wikimedia.org/favicon.ico
https://quality.wikimedia.org/favicon.ico
https://office.wikimedia.org/favicon.ico
https://wikimania2011.wikimedia.org/favicon.ico
https://vote.wikimedia.org/favicon.ico
https://ng.wikimedia.org/favicon.ico
https://collab.wikimedia.org/favicon.ico
https://mai.wikimedia.org/favicon.ico
https://u4c.wikimedia.org/favicon.ico

Expected: Either it shows an icon, or an HTTP 404 error.

Actual: It shows an error message like this: Failed to fetch URL "https://donate.wikimedia.org/static/favicon/wmf.ico". https://donate.wikimedia.org/static/favicon/wmf.ico is in fact a functioning URL that serves a favicon.

The problem does not affect all sites; for example, https://meta.wikimedia.org/favicon.ico works and shows an icon.

The failure is associated with a log message on the http channel: https://logstash.wikimedia.org/goto/f8ce6f4531b782ec7b12daa6aea91b32
GET https://donate.wikimedia.org/static/favicon/wmf.ico HTTP/1.1 - NULL cURL error 7: Failed to connect to donate.wikimedia.org port 443: Connection timed out (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://donate.wikimedia.org/static/favicon/wmf.ico
(this logging is new after we enabled additional log messages in T228838, which is how I found this error)

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Replace favicon.php with static.php	operations/mediawiki-config	master	+21 -116
	Add various .wikimedia.org domains to $wgLocalVirtualHosts	operations/mediawiki-config	master	+11 -0

Customize query in gerrit

Related Objects

Mentioned In: T373737: Disable irrelevant extensions on SUL3 login domain
T383729: SUL3 Phase 0: Account creation and login on test wikis
Mentioned Here: T373737: Disable irrelevant extensions on SUL3 login domain
T383729: SUL3 Phase 0: Account creation and login on test wikis
T228838: Consider enabling all MW log channels by default for WMF

Event Timeline

matmarex created this task.Sep 17 2024, 6:34 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 17 2024, 6:34 PM

Vgutierrez claimed this task.Sep 18 2024, 9:51 AM

Vgutierrez triaged this task as Medium priority.

Vgutierrez edited projects, added Traffic; removed SRE, WMF-General-or-Unknown.

Vgutierrez subscribed.

Provided URLs are currently handled by mw-web:

vgutierrez@carrot:/tmp$ ./T374997.sh 
https://donate.wikimedia.org/favicon.ico
< server: mw-web.eqiad.main-6646476df4-7z5sc
https://quality.wikimedia.org/favicon.ico
< server: mw-web.eqiad.main-6646476df4-hmc86
https://office.wikimedia.org/favicon.ico
< server: mw-web.eqiad.main-6646476df4-8xnx7
https://wikimania2011.wikimedia.org/favicon.ico
< server: mw-web.eqiad.main-6646476df4-m6l62
https://vote.wikimedia.org/favicon.ico
< server: mw-web.eqiad.main-6646476df4-t6cgb
https://ng.wikimedia.org/favicon.ico
< server: mw-web.eqiad.main-6646476df4-2x87n
https://collab.wikimedia.org/favicon.ico
< server: mw-web.eqiad.main-6646476df4-4qwnw
https://mai.wikimedia.org/favicon.ico
< server: mw-web.eqiad.main-6646476df4-zckls
https://u4c.wikimedia.org/favicon.ico
< server: mw-web.eqiad.main-6646476df4-fx86f

akosiaris moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.Sep 19 2024, 3:13 PM

Change #1074365 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/mediawiki-config@master] Add various .wikimedia.org domains to $wgLocalVirtualHosts

https://gerrit.wikimedia.org/r/1074365

gerritbot added a project: Patch-For-Review.Sep 20 2024, 8:14 AM

@matmarex , I am pretty confident I 've found the reason and have a patch at https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1074365. I 'd appreciate a review, mostly for avoiding side effects I am not aware of.

Thanks for investigating, and for the patch. That list is definitely not complete, it's just the top entries I saw in the logs last Tuesday, so adding them to $wgLocalVirtualHosts won't completely resolve the issue (although it will reduce the frequency of it).

This comment above $wgLocalVirtualHosts is somewhat concerning:

// At a minimum, all SUL-linked wikis must be included here for cross-wiki HTTP
// requests to work (GlobalUserPage, cross-wiki notifications, etc.).
// TODO: add remaining MediaWiki sites

I think the "etc." there is doing a lot of work, to me it says we have no idea what will actually be affected by this change. The logs are mostly (entirely?) for private and closed wikis, some non-SUL (like officewiki), so I'm not sure if we actually want them to be able to talk to each other.

I would hold this change, nothing is really broken because of this bug. I will see if I have the time to look at the favicon.php script and see why it's doing things in such a complicated way. It seems to be basically unchanged since 2012, maybe there's an easier way to do it now.

nothing is really broken because of this bug

To be clear, our wikis are not even using the broken favicon URLs. If you go to https://donate.wikimedia.org/, it loads its icons from https://donate.wikimedia.org/static/favicon/wmf.ico (using <link rel="icon" href="/static/favicon/wmf.ico">), and similarly for any other wiki. It's probably just scrapers and weird clients requesting https://donate.wikimedia.org/favicon.ico.

Change #1075211 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/mediawiki-config@master] Replace favicon.php with static.php

https://gerrit.wikimedia.org/r/1075211

Change #1075211 merged by jenkins-bot:

[operations/mediawiki-config@master] Replace favicon.php with static.php

https://gerrit.wikimedia.org/r/1075211

Mentioned in SAL (#wikimedia-operations) [2025-01-15T14:27:18Z] <lucaswerkmeister-wmde@deploy2002> Started scap sync-world: Backport for [[gerrit:1111343|Yet more authentication domain overrides (T383729 T373737)]], [[gerrit:1111598|Move Beta Cluster favicons to this repository]], [[gerrit:1075211|Replace favicon.php with static.php (T374997)]]

Mentioned in SAL (#wikimedia-operations) [2025-01-15T14:34:58Z] <lucaswerkmeister-wmde@deploy2002> tgr, matmarex, lucaswerkmeister-wmde: Backport for [[gerrit:1111343|Yet more authentication domain overrides (T383729 T373737)]], [[gerrit:1111598|Move Beta Cluster favicons to this repository]], [[gerrit:1075211|Replace favicon.php with static.php (T374997)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-01-15T14:43:10Z] <lucaswerkmeister-wmde@deploy2002> Finished scap sync-world: Backport for [[gerrit:1111343|Yet more authentication domain overrides (T383729 T373737)]], [[gerrit:1111598|Move Beta Cluster favicons to this repository]], [[gerrit:1075211|Replace favicon.php with static.php (T374997)]] (duration: 15m 52s)

While testing the changes, we realized that the icons were previously also accessible under URLs like https://meta.wikimedia.org/w/favicon.php, which now returns the message "Unknown file path". I think that's a correct thing to do, and that the script probably shouldn't even be directly accessible from the web. Favicons should only be served at URLs like https://meta.wikimedia.org/static/favicon/community.ico (used in <link> HTML tags) and https://meta.wikimedia.org/favicon.ico (used by default when <link> tags are not present).

URLs in the task description now work correctly.

Change #1074365 abandoned by Alexandros Kosiaris: