Page MenuHomePhabricator

Return free images by default
Closed, ResolvedPublic2 Story Points

Description

A/C

See T152155.

Event Timeline

bmansurov created this task.Dec 2 2016, 5:09 PM
Restricted Application added a subscriber: Aklapper. ยท View Herald TranscriptDec 2 2016, 5:09 PM
ovasileva triaged this task as High priority.Dec 5 2016, 3:16 PM
bmansurov updated the task description. (Show Details)Dec 7 2016, 5:37 PM
bmansurov set the point value for this task to 2.Dec 7 2016, 5:39 PM
Restricted Application added subscribers: JEumerus, Dereckson, Matanya. ยท View Herald TranscriptDec 8 2016, 11:51 PM

@Jdlrobson - has the maintenance script completed?

MaxSem removed a subscriber: MaxSem.Jan 18 2017, 1:16 AM

@EBernhardson: We wanted to let you know that we're scheduling this for our upcoming sprint, which starts today. We are more confident about the next deploy but it'd be good to have you (or someone capable of running the maintenance script) on hand.

Yes the maintenance script is finished, should be safe to turn back the other way. Should be verifiable with a few sample api requests by syncing the change to an mwdebug* host after merging, and before syncing out to the wider cluster

bmansurov moved this task from To Do to Doing on the Reading-Web-Sprint-90-๐ŸŒ board.

Change 333094 had a related patch set uploaded (by Bmansurov):
API: return free images by default

https://gerrit.wikimedia.org/r/333094

bmansurov added a comment.EditedJan 19 2017, 11:01 PM

Some notable pages that don't have page images:

EDIT: I see an image now.

These pages have images:

These pages should have both free and non-free images, but they don't:

The non-free image is not in the page props. the free image is ranked higher because its ratio scores more than the non-free image's ratio, despite it's position being lower. So this is also correct.

These pages have both free and non-free images:

Given some manual testing I think the patch is ready to be merged given there are no problems with it.

debt added a subscriber: debt.Jan 23 2017, 5:35 PM

@bmansurov I'm a little confused by your comment and edits. Are these test cases you would like us to test on or are you saying there is something wrong with the current state?
I guess we should merge this post branch cut so we can test more extensively on beta cluster. Do we have a test plan for there?

I wanted to test whether the test scrip correctly finished. Data in production servers seem to be correct. I'm not sure what we would test extensively on beta cluster. If we're worried about data, then we should test API end points on production servers. If we're worried about code, then testing a couple of pages on beta cluster would be enough.

I'll merge this tomorrow after the branch is cut. Will give us some time to run some check on BC

MaxSem removed a subscriber: MaxSem.Jan 24 2017, 1:58 AM

Change 333094 merged by jenkins-bot:
API: return free images by default

https://gerrit.wikimedia.org/r/333094

FWIW I agree with @pmiazga that we should make this default configurable in case we want to change again in future. This really should have been a one line change.

For all those following along at home, if you want to make use of non-free images you'll want to make your changes to how the API is consumed by Thursday 2nd Feb.

@bmansurov @EBernhardson I'm seeing some weirdness
When I search for "Sa" on https://en.m.wikipedia.beta.wmflabs.org/ I'm seeing no page images for search term "Sa"

https://en.m.wikipedia.beta.wmflabs.org//w/api.php?action=query&format=json&prop=pageimages&titles=Samurai
returns no page image.
https://en.m.wikipedia.beta.wmflabs.org/w/api.php?action=query&format=json&prop=pageimages&titles=Samurai&pilicense=any does.

@Tgr I have a theory that all images used on the beta cluster are not the free license. Do we need to do something on the beta cluster to make this work? It's making me very uncomfortable not being able to see page images on the beta cluster.

bmansurov added a comment.EditedJan 24 2017, 8:51 PM

I think we need to run the maintenance script on beta labs, or edit those pages manually.

Tgr added a comment.Jan 24 2017, 8:57 PM

@Tgr I have a theory that all images used on the beta cluster are not the free license. Do we need to do something on the beta cluster to make this work? It's making me very uncomfortable not being able to see page images on the beta cluster.

Images uploaded to the beta cluster, probably not. You need to add machine-readable metadata to the description page for that, which on Commons is done via the license templates; on beta I doubt anyone bothers.

Images loaded from Commons (which is configured as a remote repo for beta) should work as expected, though.

So I edited Samurai and it now shows (purging had no impact) so @bmansurov you are probably right about the maintenance scripts... so it seems like all pages on the beta cluster would need to be edited (purging not good enough) or a maintenance script would need to be run on the beta cluster to verify this is behaving correctly. @EBernhardson able to help with that? This should not be an issue on production given all pages have now had page images regenerated.

NB: https://en.m.wikipedia.beta.wmflabs.org/wiki/File:Samurai.jpg and https://en.m.wikipedia.beta.wmflabs.org/wiki/File:President_Barack_Obama.jpg are both coming from Commons

Thanks @Tgr for the response. Let me know if you think of anything else.

Tgr added a comment.Jan 24 2017, 9:10 PM

PageImages runs on links update and purging does not trigger that (a null edit does). You can just SSH into beta tin and run refreshLinks to trigger it for everything; beta is probably small enough for that to finish in a reasonable amount of time.

Beta is pretty small, i kicked off the initImageData.php script from deployment-tin.eqiad.wmflabs and it's progressing pretty quickly.

I have some suspicion beta is misconfigured, but not sure what or how it should be. Since starting the initImageData.php job the jobs are sending messages to logstash like:

FSFileBackend::doPrepareInternal: cannot create directory /srv/mediawiki/php-master/images/thumb/f/f9/Joo_Chia...
bmansurov removed bmansurov as the assignee of this task.Jan 24 2017, 9:57 PM

I have some suspicion beta is misconfigured, but not sure what or how it should be. Since starting the initImageData.php job the jobs are sending messages to logstash like:

FSFileBackend::doPrepareInternal: cannot create directory /srv/mediawiki/php-master/images/thumb/f/f9/Joo_Chia...

Filed as T156203.

phuedx assigned this task to ovasileva.Jan 26 2017, 1:38 PM
phuedx reassigned this task from ovasileva to bmansurov.Jan 26 2017, 1:42 PM

This is mostly โ€“ if not entirely โ€“ a technical task. Over to you!

(I don't mind signing this off but I'd like to see a list of test pages on the Beta Cluster.)

bmansurov closed this task as Resolved.Jan 27 2017, 6:51 PM
bmansurov removed bmansurov as the assignee of this task.

Tested pages:

For future reference, here is the list of test pages. Other pages can be found by searching for a small text in the mobile search overlay and visually inspecting pages that have images.