Kill the skins/common/ directory in mediawiki/core, moving the contents somewhere else
Closed, ResolvedPublic

Milimetric added a comment.Via ConduitAug 14 2014, 10:26 PM

Hi Bartosz,

I ran:

select uri_path,

      count(*)
 from webrequest
where webrequest_source='bits'
  and year=2014
  and month=8
  and day=1
  and uri_path like '%skins/common/%'
group by uri_path

on our experimental hadoop cluster, and got the results that I am attaching to this bug. (This is showing the aggregated hits to the pattern you requested for August 1st, 2014. If you'd like a different approach, please let me know.

Milimetric added a comment.Via ConduitAug 14 2014, 10:27 PM

Created attachment 16199
hits to */skins/common/** on August 1st

Attached: august.1.2014.bits.skins.common.tsv

matmarex added a comment.Via ConduitAug 14 2014, 10:47 PM

Created attachment 16200
mwgrep skins/common by Ori

Another data point.

Thanks for the logs, Dan!

Attached: raw[1]

matmarex added a comment.Via ConduitAug 17 2014, 5:56 PM

I took Dan's logs, manually removed some clearly botched requests, removed everything older than 1.24wmf6 (the directories for earlier ones don't even exist anymore, it's a bit worrying we have requests for them…), grouped by paths relative to skins/common/ and cut off at ten requests.

I'm going to look into particular entries in detail and file bugs / submit patches, but this makes it pretty clear which things are worth being very careful and which probably aren't. We should start cleaning up and do this again in two months or so.

images/magnify-clip.png 87619775
images/poweredby_mediawiki_88x31.png 76777484
images/icons/fileicon-ogg.png 3358726
images/magnify-clip-rtl.png 1163918
images/button_image.png 907018
images/button_extlink.png 904338
images/button_hr.png 903505
images/button_sig.png 903341
images/button_media.png 902563
images/button_headline.png 888904
images/button_nowiki.png 885905
images/button_link.png 776108
images/button_bold.png 718359
images/button_italic.png 717930
IEFixes.js 153079
images/Checker-16x16.png 132899
images/redirectltr.png 117337
images/cyrl/button_bold.png 114188
images/cyrl/button_italic.png 114169
images/cyrl/button_link.png 114090
images/de/button_italic.png 60100
images/de/button_bold.png 60097
images/feed-icon.png 48668
images/bullet.gif 28780
images/ar/button_headline.png 15045
images/ar/button_bold.png 15021
images/ar/button_italic.png 15013
images/ar/button_link.png 14997
images/ar/button_nowiki.png 14963
images/question-small.png 13177
images/spinner.gif 11633
images/ajax-loader.gif 11506
images/ 11312
images/help-question.gif 11270
images/help-question-hover.gif 11231
wikibits.js 10563
ajax.js 10537
images/question.svg 6550
images/feed-icon.svg 6520
images/icons/fileicon-mid.png 3871
images/arrow_right_25.png 1882
images/arrow_last_25.png 1872
images/Arr_d.png 1858
images/fa/button_italic.png 1855
images/fa/button_bold.png 1852
images/fa/button_link.png 1850
images/fa/button_headline.png 1849
images/arrow_disabled_left_25.png 1843
images/fa/button_nowiki.png 1835
images/arrow_disabled_first_25.png 1834
images/redirectrtl.png 947
images/closewindow19x19.png 568
images 518
shared.css 405
images/arrow_first_25.png 371
images/arrow_left_25.png 370
feed.css 310
images/arrow_disabled_right_25.png 178
images/arrow_disabled_last_25.png 174
images/be-tarask/button_link.png 156
images/be-tarask/button_italic.png 154
images/be-tarask/button_bold.png 153
images/Arr_u.png 128
images/icons/ 108
images/icons/fileicon.png 101
images/button_template.png 97
images/magnify-clip(png).jpg 35
images/tipsy-arrow.gif 33
images/icons/fileicon-pdf.png 33
images/poweredby_mediawiki_88x31.not 18
images/icons/fileicon-html.png 15
images/Arr_r.png 14
images/icons/fileicon-svg.png 14
images/magnify-clip.pnf 12

matmarex added a comment.Via ConduitAug 17 2014, 6:54 PM

Many of the requests for images @embed-ded in CSS are probably from IE 6 and should disappear after we move the style files (I have already submitted patches for that above).

images/magnify-clip.png 87619775
images/magnify-clip-rtl.png 1163918

Filed bug 69277 to track these. Gonna be fun.

IEFixes.js 153079

No longer exists after a68ce00d.

images/bullet.gif 28780

This is surprising, none of our code seems to refer to it. Needs investigating.

images/spinner.gif 11633

Some of these is ApiSandbox, see bug 69672.

matmarex added a comment.Via ConduitAug 17 2014, 6:58 PM

(In reply to Bartosz Dziewoński from comment #12)

> images/magnify-clip.png 87619775
> images/magnify-clip-rtl.png 1163918

Filed bug 69277 to track these. Gonna be fun.

bug 69673*

gerritbot added a comment.Via ConduitAug 17 2014, 8:38 PM

Change 154525 had a related patch set uploaded by Bartosz Dziewoński:
Remove some unused files from skins/common/

https://gerrit.wikimedia.org/r/154525

gerritbot added a comment.Via ConduitAug 17 2014, 9:24 PM

Change 154525 merged by jenkins-bot:
Remove some unused files from skins/common/

https://gerrit.wikimedia.org/r/154525

Milimetric added a comment.Via ConduitAug 18 2014, 1:35 PM

Interesting findings Bartosz, please send an email to Analytics-l when you want to do this again. Tools built on usage data that could be used for ongoing cleanup sound like a good idea.

gerritbot added a comment.Via ConduitAug 19 2014, 1:54 PM

Change 153368 merged by jenkins-bot:
Move installer files from skins/common/ to mw-config/

https://gerrit.wikimedia.org/r/153368

Schnark added a comment.Via ConduitAug 20 2014, 9:50 AM

(In reply to Bartosz Dziewoński from comment #12)

> images/bullet.gif 28780

This is surprising, none of our code seems to refer to it. Needs
investigating.

[[MediaWiki:Common.css]] uses this (and perhaps other wiki styles, too).

gerritbot added a comment.Via ConduitAug 20 2014, 9:22 PM

Change 155442 had a related patch set uploaded by Bartosz Dziewoński:
Revamp classic edit toolbar not to hardcode paths in HTML

https://gerrit.wikimedia.org/r/155442

gerritbot added a comment.Via ConduitAug 21 2014, 8:46 PM

Change 155631 had a related patch set uploaded by Bartosz Dziewoński:
Delete skins/common/images/button_template.png

https://gerrit.wikimedia.org/r/155631

gerritbot added a comment.Via ConduitAug 21 2014, 8:53 PM

Change 155631 merged by jenkins-bot:
Delete skins/common/images/button_template.png

https://gerrit.wikimedia.org/r/155631

Edokter added a comment.Via ConduitAug 22 2014, 8:19 AM

Where are the images going to end up from Wikipedia's viewpoint? Many userscripts and gadgets also depend on it so I'd like a heads up when the location is known.

Legoktm added a comment.Via ConduitAug 22 2014, 3:59 PM

(In reply to Erwin Dokter from comment #22)

Where are the images going to end up from Wikipedia's viewpoint? Many
userscripts and gadgets also depend on it so I'd like a heads up when the
location is known.

Commons usually. Bartosz has been using mwgrep to check if the images are used in sitewide CSS/JS, and updating the urls of those that are.

gerritbot added a comment.Via ConduitAug 22 2014, 6:26 PM

Change 155771 had a related patch set uploaded by Bartosz Dziewoński:
Move feed.css to resources/

https://gerrit.wikimedia.org/r/155771

Edokter added a comment.Via ConduitAug 22 2014, 6:42 PM

(In reply to Kunal Mehta (Legoktm) from comment #23)

Commons usually. Bartosz has been using mwgrep to check if the images are
used in sitewide CSS/JS, and updating the urls of those that are.

That is not what I ment. I can't rely on Common as it does not have all the skin images. Right now, I can call the images locally in /w/skins/common/images/. Where are they going to be moved to from the viewpoint of the local installation? Will it be /w/resources/common/images/? Or perhaps /w/resources/skins/common/images/? Or may we perhaps have symbolic links mapping to their new location?

Just a very simple question.

matmarex added a comment.Via ConduitAug 22 2014, 6:47 PM

(In reply to Erwin Dokter from comment #25)

Where are they going to be moved to from the
viewpoint of the local installation? Will it be /w/resources/common/images/?
Or perhaps /w/resources/skins/common/images/?

Nowhere. It's irrelevant. The paths to files are *not* stable and should not be relied on. (The old ones in skins/common/ were an exception.)

So far I've been able to find all of the files I removed on Wikimedia Commons (https://commons.wikimedia.org/), under the same filenames too.

If there is anything that's used by on-wiki scripts and isn't there, it should be uploaded there (and I will upload it if I run into anything).

Or may we perhaps have
symbolic links mapping to their new location?

MediaWiki supports Windows hosts too, which don't support UNIX-style symbolic links, so we can't do that in the main repository. Perhaps you could convince ops to keep the old URLs valid in some way, but I'm not going to try convincing anyone it's a good idea :)

Edokter added a comment.Via ConduitAug 22 2014, 6:54 PM

(In reply to Bartosz Dziewoński from comment #26)

Nowhere. It's irrelevant. The paths to files are *not* stable and should not
be relied on. (The old ones in skins/common/ were an exception.)

Commons files are no more stable; the ones without protection are vulnerable to vandalism. I do consider the local paths to be safer then Commons. So please... where are they going?

matmarex added a comment.Via ConduitAug 22 2014, 7:08 PM

(In reply to Erwin Dokter from comment #27)

Commons files are no more stable; the ones without protection are vulnerable
to vandalism. I do consider the local paths to be safer then Commons. So
please... where are they going?

It depends on the image; I'm trying to move them somewhere that makes sense. Generally somewhere under resources/. Some, which were unused (both in extensions and on Wikimedia wikis, as far as I could tell), were also removed entirely.

I intend to make a list and add it to release notes; I was planning to do that later (would make it easier to have several independent patches pending review at the same time), but I guess I can do that now (later today).

Schnark added a comment.Via ConduitAug 23 2014, 9:14 AM

(In reply to Erwin Dokter from comment #27)

Commons files are no more stable; the ones without protection are vulnerable
to vandalism. I do consider the local paths to be safer then Commons. So
please... where are they going?

If the gadget doesn't have to support IE 7 and the image is small enough, using data URIs is also an option.

gerritbot added a comment.Via ConduitAug 23 2014, 6:44 PM

Change 155910 had a related patch set uploaded by Bartosz Dziewoński:
TablePager: Load images via CSS backgrounds rather than HTML <img>s

https://gerrit.wikimedia.org/r/155910

gerritbot added a comment.Via ConduitAug 23 2014, 7:40 PM

Change 155914 had a related patch set uploaded by Bartosz Dziewoński:
TablePager: Redo arrow icons from scratch as CSS backgrounds

https://gerrit.wikimedia.org/r/155914

gerritbot added a comment.Via ConduitAug 26 2014, 2:50 PM

Change 156292 had a related patch set uploaded by Bartosz Dziewoński:
Delete feed.css

https://gerrit.wikimedia.org/r/156292

gerritbot added a comment.Via ConduitAug 26 2014, 2:51 PM

Change 155771 abandoned by Bartosz Dziewoński:
Move feed.css to resources/, introduce resources/src/assets/

Reason:
I'll revisit the 'assets' directory idea later.

Not overwriting this patch so that I can refer to the discussion from elsewhere.

https://gerrit.wikimedia.org/r/155771

gerritbot added a comment.Via ConduitAug 26 2014, 3:06 PM

Change 156292 merged by jenkins-bot:
Delete feed.css

https://gerrit.wikimedia.org/r/156292

saper added a comment.Via ConduitAug 26 2014, 7:54 PM

Seems like a slight botched preparation of the mediawiki/skins project :(

Frankly I think it's a wasted effort here. Some files have been here nearly forever and I am not sure it is worth chasing links to those resources.

Cool URIs do not change :) - just provide 302 redirects instead, if you really need to move them.

matmarex added a comment.Via ConduitAug 28 2014, 9:47 PM

(In reply to Marcin Cieślak from comment #35)

Seems like a slight botched preparation of the mediawiki/skins project :(

Indeed :( There are limits to what one person can do.

Frankly I think it's a wasted effort here. Some files have been here nearly
forever and I am not sure it is worth chasing links to those resources.

This bug, and the patches, are not really about chasing links (although I fix the important ones) but about just moving the files elsewhere. As you can see, it's taking surprisingly long, because, again, I'm the only person working on it and no one wants to review patches like that.

Cool URIs do not change :) - just provide 302 redirects instead, if you
really need to move them.

From comment 26:

Perhaps you
could convince ops to keep the old URLs valid in some way, but I'm not going
to try convincing anyone it's a good idea :)

saper added a comment.Via ConduitAug 28 2014, 10:05 PM

(In reply to Bartosz Dziewoński from comment #36)

(In reply to Marcin Cieślak from comment #35)
> Seems like a slight botched preparation of the mediawiki/skins project :(

Indeed :( There are limits to what one person can do.

> Frankly I think it's a wasted effort here. Some files have been here nearly
> forever and I am not sure it is worth chasing links to those resources.

This bug, and the patches, are not really about chasing links (although I
fix the important ones) but about just moving the files elsewhere. As you
can see, it's taking surprisingly long, because, again, I'm the only person
working on it and no one wants to review patches like that.

That's why I think bug 69678 is not worth your time, really (but I'm not the one to judge it).

> Cool URIs do not change :) - just provide 302 redirects instead, if you
> really need to move them.

From comment 26:
> Perhaps you
> could convince ops to keep the old URLs valid in some way, but I'm not going
> to try convincing anyone it's a good idea :)

Personally I am fine with the symlink solution and I'd close all of it as WONTFIX.

Or maybe why not MediaWiki search for skins also in some other directory, say /w/skins-ext (or whatever the name) and leave /w/skins/common in peace.

What do you think?

gerritbot added a comment.Via ConduitAug 30 2014, 10:38 PM

Change 153370 merged by jenkins-bot:
Move mediawiki.legacy.* modules from skins/common/ to resources/

https://gerrit.wikimedia.org/r/153370

gerritbot added a comment.Via ConduitAug 30 2014, 11:02 PM

Change 155910 merged by jenkins-bot:
TablePager: Load images via CSS backgrounds rather than HTML <img>s

https://gerrit.wikimedia.org/r/155910

gerritbot added a comment.Via ConduitAug 30 2014, 11:04 PM

Change 155914 merged by Bartosz Dziewoński:
TablePager: Redo arrow icons from scratch as CSS backgrounds

https://gerrit.wikimedia.org/r/155914

gerritbot added a comment.Via ConduitAug 31 2014, 12:51 AM

Change 157490 had a related patch set uploaded by Bartosz Dziewoński:
Move mediawiki.skinning.* modules from skins/common/ to resources/

https://gerrit.wikimedia.org/r/157490

gerritbot added a comment.Via ConduitSep 3 2014, 1:48 PM

Change 158091 had a related patch set uploaded by Bartosz Dziewoński:
Delete closewindow19x19.png

https://gerrit.wikimedia.org/r/158091

gerritbot added a comment.Via ConduitSep 3 2014, 5:42 PM

Change 158091 merged by jenkins-bot:
Delete skins/common/images/closewindow19x19.png

https://gerrit.wikimedia.org/r/158091

gerritbot added a comment.Via ConduitSep 4 2014, 11:09 AM

Change 157490 merged by jenkins-bot:
Move mediawiki.skinning.* modules from skins/common/ to resources/

https://gerrit.wikimedia.org/r/157490

gerritbot added a comment.Via ConduitSep 5 2014, 2:50 PM

Change 158627 had a related patch set uploaded by Bartosz Dziewoński:
Article: Don't hardcode <img> tags on redirect page

https://gerrit.wikimedia.org/r/158627

gerritbot added a comment.Via ConduitSep 5 2014, 6:04 PM

Change 158648 had a related patch set uploaded by Bartosz Dziewoński:
Move file type icons to new assets/ directory

https://gerrit.wikimedia.org/r/158648

gerritbot added a comment.Via ConduitSep 5 2014, 6:04 PM

Change 158649 had a related patch set uploaded by Bartosz Dziewoński:
Move mediawiki.png to assets/ directory

https://gerrit.wikimedia.org/r/158649

gerritbot added a comment.Via ConduitSep 5 2014, 6:04 PM

Change 158650 had a related patch set uploaded by Bartosz Dziewoński:
Move remaining skin files to assets/ directory

https://gerrit.wikimedia.org/r/158650

matmarex added a comment.Via ConduitSep 5 2014, 6:07 PM

And with this, I have patches pending or merged to make MediaWiki stop using skins/common/ entirely.

What remains is getting the remaining copies of files actually deleted, and probably fixing up some unavoidable bugs.

Patches relevant to this bug that are still pending:

gerritbot added a comment.Via ConduitSep 7 2014, 9:26 PM

Change 158648 merged by jenkins-bot:
Move file type icons to new assets/ directory

https://gerrit.wikimedia.org/r/158648

gerritbot added a comment.Via ConduitSep 7 2014, 9:26 PM

Change 158649 merged by jenkins-bot:
Move mediawiki.png to assets/ directory

https://gerrit.wikimedia.org/r/158649

gerritbot added a comment.Via ConduitSep 7 2014, 9:36 PM

Change 158627 merged by jenkins-bot:
Article: Don't hardcode <img> tags on redirect page

https://gerrit.wikimedia.org/r/158627

matmarex added a comment.Via ConduitSep 7 2014, 11:12 PM

I put up a note on the release page: [[mw:MediaWiki 1.24#Directory changes]].
Haven't updated RELEASE-NOTES-1.24 yet.

gerritbot added a comment.Via ConduitSep 9 2014, 12:33 PM

Change 159290 had a related patch set uploaded by Bartosz Dziewoński:
Move default logo to assets/ directory

https://gerrit.wikimedia.org/r/159290

gerritbot added a comment.Via ConduitSep 14 2014, 12:19 PM

Change 158650 merged by jenkins-bot:
Move footer icons to assets/ directory

https://gerrit.wikimedia.org/r/158650

gerritbot added a comment.Via ConduitSep 16 2014, 12:51 PM

Change 155442 merged by jenkins-bot:
Revamp classic edit toolbar not to hardcode paths in HTML

https://gerrit.wikimedia.org/r/155442

Jdforrester-WMF added a comment.Via ConduitSep 16 2014, 10:37 PM

Last remaining item in gerrit is https://gerrit.wikimedia.org/r/#/c/159290/ per https://gerrit.wikimedia.org/r/#/q/bug:69277,n,z – then "just" gadgets.

matmarex added a comment.Via ConduitSep 16 2014, 11:18 PM

(In reply to James Forrester from comment #58)

Last remaining item in gerrit is https://gerrit.wikimedia.org/r/#/c/159290/

Yup, also remaining to do:

  • Remove the stray files in skins/common/ that are waiting for caches to expire
  • Release notes listing what exactly was removed

then "just" gadgets.

I've been trying to mwgrep as I was moving/deleting files and this seems to be a much lesser issue than I anticipated. Unless I missed something, the only files that still need action are bullet.gif and possibly wikibits.js.

It's been about a month, so I asked Dan to re-run the query from comment 8.
I wonder how the request counts have changed with the patches that have already been deployed. There are probably still many requests now, but hopefully when we re-run it again next month the numbers will hover around zero. :)

Milimetric added a comment.Via ConduitSep 17 2014, 6:26 PM

I ran the query again for September 16th, but I got some fairly detailed URLs that I'm not sure I should post. I'm checking with some people and I'll post if they think it's ok.

Milimetric added a comment.Via ConduitSep 17 2014, 10:30 PM

I'm posting the file. I just redacted all results that had <= 20 hits. There were some weird, overly-specific URLs in there. I left everything as it came back from my query: https://bugzilla.wikimedia.org/show_bug.cgi?id=69277#c8

Milimetric added a comment.Via ConduitSep 17 2014, 10:31 PM

Created attachment 16507
hits to */skins/common/** on September 16th

Attached: skins.common.access.2014.09.16.tsv

matmarex added a comment.Via ConduitSep 18 2014, 10:46 AM

(The assets/ directory has been renamed to resources/assets/ in 1cf5a6e9.)

gerritbot added a comment.Via ConduitSep 21 2014, 5:17 AM

Change 161770 had a related patch set uploaded by Legoktm:
Move default logo to resources/assets/ directory

https://gerrit.wikimedia.org/r/161770

gerritbot added a comment.Via ConduitSep 21 2014, 5:23 AM

Change 159290 merged by jenkins-bot:
Move default logo to resources/assets/ directory

https://gerrit.wikimedia.org/r/159290

gerritbot added a comment.Via ConduitSep 21 2014, 5:24 AM

Change 161770 merged by jenkins-bot:
Move default logo to resources/assets/ directory

https://gerrit.wikimedia.org/r/161770

Jdforrester-WMF added a comment.Via ConduitSep 21 2014, 5:47 AM

Still a few files left to go…

waldyrious added a comment.Via ConduitSep 21 2014, 5:45 PM

(In reply to James Forrester from comment #67)

Still a few files left to go…

Link for convenience: http://git.wikimedia.org/tree/mediawiki%2Fcore.git/HEAD/skins%2Fcommon

gerritbot added a comment.Via ConduitSep 26 2014, 2:28 PM

Change 163168 had a related patch set uploaded by Bartosz Dziewoński:
Delete skins/common/

https://gerrit.wikimedia.org/r/163168

gerritbot added a comment.Via ConduitSep 26 2014, 2:28 PM

Change 163169 had a related patch set uploaded by Bartosz Dziewoński:
Add release notes for skins/common/ removal

https://gerrit.wikimedia.org/r/163169

gerritbot added a comment.Via ConduitSep 26 2014, 2:28 PM

Change 163171 had a related patch set uploaded by Bartosz Dziewoński:
Add release notes for skins/common/ removal

https://gerrit.wikimedia.org/r/163171

gerritbot added a comment.Via ConduitSep 26 2014, 2:58 PM

Change 163168 merged by Bartosz Dziewoński:
Delete skins/common/

https://gerrit.wikimedia.org/r/163168

gerritbot added a comment.Via ConduitSep 26 2014, 6:12 PM

Change 163210 had a related patch set uploaded by Paladox:
Delete skins/common/

https://gerrit.wikimedia.org/r/163210

gerritbot added a comment.Via ConduitSep 27 2014, 12:29 AM

Change 163210 abandoned by Paladox:
Delete skins/common/

https://gerrit.wikimedia.org/r/163210

gerritbot added a comment.Via ConduitSep 27 2014, 1:10 AM

Change 163171 merged by jenkins-bot:
Add release notes for skins/common/ removal

https://gerrit.wikimedia.org/r/163171

gerritbot added a comment.Via ConduitSep 27 2014, 1:11 AM

Change 163169 merged by jenkins-bot:
Add release notes for skins/common/ removal

https://gerrit.wikimedia.org/r/163169

matmarex added a comment.Via ConduitSep 27 2014, 1:11 AM

(In reply to Bartosz Dziewoński from comment #59)

Release notes listing what exactly was removed

Done for 1.24 and synchronised on master, see above.

Remove the stray files in skins/common/ that are waiting for caches to expire

Removed for 1.24, see above. Will do so on master, too, but only after finishing with bug 69678. I think this is the only thing remaining before we can finally close this bug.

Edokter added a comment.Via ConduitSep 27 2014, 8:45 AM

Where will bullet.gif end up?

Please don't tell me "it depends" or anything like that... Monobook (as does Common.css) still needs this and I cannot trace where it has been/will be moved.

I absolutely need to know.

Edokter added a comment.Via ConduitSep 27 2014, 9:05 AM

Never mind... found it in /skins/MonoBook.

matmarex added a comment.Via ConduitSep 30 2014, 8:43 PM

(In reply to Dan Andreescu from comment #62)

Created attachment 16507 [details]
hits to */skins/common/** on September 16th

(I did look at this, but it just so happened that the changes I was specifically interested in evaluating didn't go into production soon enough for this to contain data useful for me :( )

Attached: skins.common.access.2014.09.16.tsv

Milimetric added a comment.Via ConduitSep 30 2014, 8:57 PM

Bartosz, let me know if you need another run at some point. If I don't answer just ping me on IRC (milimetric in #wikimedia-analytics).

Milimetric added a comment.Via ConduitOct 29 2014, 2:49 PM

Created attachment 16942
hits to */skins/common/** on October 27th

Attached: skins.common.access.tsv

Reedy added a comment.Via ConduitOct 29 2014, 2:52 PM

wtf

/static-1.21wmf10/skins/common/images/magnify-clip.png 14926
/static-1.21wmf8/skins/common/images/magnify-clip.png 11160

All these links to ancient mediawiki dirs...

Reedy added a comment.Via ConduitOct 29 2014, 3:31 PM

(In reply to Dan Andreescu from comment #83)

Created attachment 16942 [details]
hits to */skins/common/** on October 27th

Are you able to tell us where these requests are coming from? A lot of those are long long dead urls... I presume they're 404-ing

Attached: skins.common.access.tsv

Milimetric added a comment.Via ConduitOct 29 2014, 3:41 PM

Well, you can query the logs with SQL on our hadoop cluster if you'd like more info (you have rights to stat1002.eqiad.wmnet right Sam?) I mean, I could run queries for you but it sounds like you just want to investigate. For example, to get a breakdown by status, you could do:

sta1002> hive
hive (default)> use wmf_raw;
hive (wmf_raw)>
select uri_path,

      http_status,
      count(*)
 from webrequest
where webrequest_source='bits'
  and year=2014
  and month=10
  and day=28
  and uri_path like '%skins/common/%'
group by uri_path,
      http_status

having count(*) > 20
;

You can see what else is available (like referrer) with:

hive (wmf_raw)> describe webrequest;

If you just want to poke around smaller bits of data, you can add something like "and hour=12". Without that, the query above could take over an hour.

Reedy added a comment.Via ConduitOct 29 2014, 3:47 PM

reedy@tin:/srv/mediawiki-staging$ ssh -A stat1002
Permission denied (publickey).
reedy@tin:/srv/mediawiki-staging$

Nope :(

Aklapper added a comment.Via ConduitOct 31 2014, 11:44 AM

(In reply to Dan Andreescu from comment #86)

Well, you can query the logs with SQL on our hadoop cluster if you'd like
more info (you have rights to stat1002.eqiad.wmnet right Sam?) I mean, I
could run queries for you

I guess either Reedy needs to get access, or Dan could run those queries?

Milimetric added a comment.Via ConduitOct 31 2014, 3:35 PM

Sorry to not update the bug - RT ticket has been filed to get Sam access: https://rt.wikimedia.org/Ticket/Display.html?id=8773

I don't have access to see it, but I can follow up if it hasn't been resolved.

matmarex added a comment.Via ConduitNov 10 2014, 3:33 PM

(In reply to Dan Andreescu from comment #83)

Created attachment 16942 [details]
hits to */skins/common/** on October 27th

Limiting this to data from 1.25wmfX and grouping, we get:

images/Checker-16x16.png 14826
images/magnify-clip.png 13622
images/bullet.gif 1407
images/closewindow19x19.png 700
shared.css 250
wikibits.js 152
images/spinner.gif 95
images/ajax-loader.gif 54
images/redirectltr.png 37
images/button_template.png 29

No idea what the deal with Checker-16x16.png is now, let's ignore it, it can't be a very important asset anywhere.

Some more of these files are already deleted and get neglibigle number of requests. Getting rid of that, we have these stats for files that still exist:

images/magnify-clip.png 13622
images/bullet.gif 1407
images/redirectltr.png 37
images/magnify-clip-rtl.png 0
images/redirectrtl.png 0

I'll spend some more time hunting down magnify-clip.png and bullet.gif, let's remove the rest already.

Attached: skins.common.access.tsv

matmarex added a comment.Via ConduitNov 10 2014, 3:36 PM

Correction:

images/magnify-clip.png 13622
images/bullet.gif 1407
wikibits.js 152
images/redirectltr.png 37
images/magnify-clip-rtl.png 0
images/redirectrtl.png 0
ajax.js 0

gerritbot added a comment.Via ConduitNov 10 2014, 3:38 PM

Change 172256 had a related patch set uploaded by Bartosz Dziewoński:
Delete skins/common/{ajax.js, wikibits.js, images/{magnify-clip-rtl.png, redirectltr.png, redirectrtl.png}}

https://gerrit.wikimedia.org/r/172256

Edokter added a comment.Via ConduitNov 11 2014, 8:48 AM

(In reply to Bartosz Dziewoński from comment #90)

No idea what the deal with Checker-16x16.png is now, let's ignore it, it
can't be a very important asset anywhere.

That is the backdrop for transparent images on file description pages, and often called from Common.css.

Limiting this to data from 1.25wmfX and grouping, we get...

Is this the entire WMF cluster, or just one wiki? If the first, the request can come from any project's Common.css.

matmarex added a comment.Via ConduitNov 11 2014, 12:32 PM

(In reply to Erwin Dokter from comment #93)

(In reply to Bartosz Dziewoński from comment #90)
> No idea what the deal with Checker-16x16.png is now, let's ignore it, it
> can't be a very important asset anywhere.

That is the backdrop for transparent images on file description pages, and
often called from Common.css.

I know what it is and where it used to be called. I only have no idea what the deal is with the thousands requests per day, as I went through the wikis a few weeks ago and replaced all of the references. Perhaps some bright soul decided to revert my changes somewhere.

> Limiting this to data from 1.25wmfX and grouping, we get...

Is this the entire WMF cluster, or just one wiki? If the first, the request
can come from any project's Common.css.

The entire cluster. I am aware that the request can come from anywhere, and I've been tracking down and replacing these references in the past months.

MarkAHershberger added a comment.Via ConduitNov 12 2014, 3:44 PM

This isn't really relevant to the release of 1.24.0 or the tarball in general as other bugs exist for dealing with it.

gerritbot added a comment.Via ConduitNov 14 2014, 9:27 PM

Change 173417 had a related patch set uploaded by Bartosz Dziewoński:
Delete skins/common/images/magnify-clip.png

https://gerrit.wikimedia.org/r/173417

gerritbot added a comment.Via ConduitNov 14 2014, 9:57 PM

Change 173434 had a related patch set uploaded by Bartosz Dziewoński:
Delete skins/common/images/bullet.gif

https://gerrit.wikimedia.org/r/173434

gerritbot added a comment.Via ConduitNov 14 2014, 10:11 PM

Change 172256 merged by jenkins-bot:
Delete skins/common/{ajax.js, wikibits.js, images/{magnify-clip-rtl.png, redirectltr.png, redirectrtl.png}}

https://gerrit.wikimedia.org/r/172256

gerritbot added a comment.Via ConduitNov 14 2014, 10:22 PM

Change 173417 merged by jenkins-bot:
Delete skins/common/images/magnify-clip.png

https://gerrit.wikimedia.org/r/173417

gerritbot added a comment.Via ConduitNov 14 2014, 10:23 PM

Change 173434 merged by jenkins-bot:
Delete skins/common/images/bullet.gif

https://gerrit.wikimedia.org/r/173434

Jdforrester-WMF added a comment.Via ConduitNov 14 2014, 10:24 PM

Done. Gosh.

matmarex added a comment.Via ConduitNov 14 2014, 10:31 PM

And the directory is gone from master too! At last! Thank you, everyone who helped wit this boring cleanup work :D

(MW 1.24 will also be released without it.)

MarkAHershberger added a comment.Via ConduitNov 15 2014, 4:18 AM

Rats. And I just sent out a note about not seeing any need to send out an RC this week. I'm sorry I missed this.

Bartosz, could you cherry pick these to REL1_24 so they'll be ready next week?

matmarex added a comment.Via ConduitNov 15 2014, 3:29 PM

It has been done a long time ago, see comment 72 and comment 77 and https://gerrit.wikimedia.org/r/#/c/163168/ and such. We couldn't delete them on master at the time because of fear of breaking on-wiki customizations.

Add Comment

Column Prototype
This is a very early prototype of a persistent column. It is not expected to work yet, and leaving it open will activate other new features which will break things. Press "\" (backslash) on your keyboard to close it now.