Most recent final timestamp: 20220426131656.

@ERayfield I really like the logic flow! Your notes above are very helpful. But I don't quite see the pros and cons of sending files. What does

@ERayfield What were your findings here?

As noted in the task above, I have that it is still a huge failure rate with only .05% of images being sent to photoDNA successfully.

I grepped the logs today and got 574 successes out of 971476 files sent.

Most recent timestamp: 20210430135351

I was thinking the first approach. Creating the statistic on an ongoing basis is out of scope for this task, as we need to get these numbers sooner to know how to proceed.

Or this message: Creating moderation request for file

We could maybe use Ellen's output

@jsn.sherman suggests using number of files rather than existing error messages.

Oh! I got exactly one "No hash match found" message now. Also, David Rochford says that there have been a total of 25 emails sent with matches.

It doesn't look like they have, but I was searching for "Hash match found for file" and "No hash match found for file". I wish we had Ellen's messages in now, because it would have been helpful.

I can confirm that I saw no success messages in the logs today.

There appears to be a file type check in MediaModerationService::processUploadedMedia but only if checkOnUpload (MediaModerationCheckOnUpload) is true and it seems to not be true for us currently.

From Cindy on sending files versus urls: "I recall that change being made. I will check my email archive to see if I can reconstruct the reason. I have a feeling it was because they have a relatively small size limit if you send the file itself."

We can also increase the logging level per https://phabricator.wikimedia.org/T303312.

Question from engineering talk: should we stop sending urls and send the photo instead? Why wasn't this done in the past? https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MediaModeration/+/576338 seems to have done this before. The next step might be asking Cindy Cicalese.

@ERayfield will be updating this today with how she had approached the code.

I approve this request.

@ERayfield aside from color ;), it looks like just what was asked for to me! I'd go ahead and push it up for @aminalhazwani and the team to review. We could also set up a patch demo for it. @Scardenasmolinar knows how to do this.

With some local testing I did find that if there's an error in the creation of the thumbnail it's not always caught. I'll need more info to know if that's what's happening here.

I got a response that needs more investigation:

I got some replies:

I followed up today because I haven't heard any kind of response.

I have sent an email to PhotoDNA support.

I've reached out to Nicholas to confirm the best way to reach PhotoDNA.

For my own notes: I'll be asking about the error we're seeing, as well as developer credentials for a sandbox account, if available.

I've reached out to @drochford in slack to make sure I contact the correct person.

@jsn.sherman Add null option to be able to run local tests.

@jsn.sherman summarized:

@ERayfield suggests that we have a starting message then dots for each batch run then an ending message.

A few more pieces of info I gathered today.

The error code is 3206.

@drochford Yup, It looks like a few hundred thousands have already run.

Hmm, I ran the script starting at 20220104105243 with the batch-count halved and now got a message saying "Script processed all files. Nothing left!".

Hmm, even more curious @Urbanecm...and that you were able to see it on the debug server.

The current hypothesis is that the "+fawiki" is not currently merging with the empty default array. @jsn.sherman and @eigyan are looking into why that would cause an issue. We may need to switch it for "fawiki" but it's also worth understanding why this is happening.

I'm actually on a call with @eigyan and other members of the team as we try to reproduce this right now.

Thank you so much for continuing to look into this @Urbanecm! I just went to a random page on fawiki and added a parameter for an example survey, but we've concluded the example surveys are not deployed (wisely). The survey was reverted so I'm assuming you mean on beta?

Thank you so much for this update @Urbanecm, and for the revert. We'll look into this on our end.

@drochford Yes, how many do we want?

The privacy policy can be configured in LocalSettings.php. Currently in QuickSurvey.vue, the footer is outside the "completed" conditional. If we wanted it to disappear after survey completion, we could just put inside the conditional.

This is tracked better in https://phabricator.wikimedia.org/T258603.

@Madalina This is ready for your review. Let me know if you have questions!

Thanks for the links @jsn.sherman!

@jsn.sherman
Small feedback below:

What were the underlying pitfalls that led to this failure?
The documentation for new extension developers did not link me to the information I needed to know.

Could you update these docs?

Vue migration to OOUI js guide

from @jsn.sherman and @essex

1. Library in question is wvui
2. Proposal to make a new component and try including it.

@jsn.sherman Do my changes above address your feedback?

It would required a deeper code re-write but we could also make the survey show up on mediasearch as in the screenshot that's part of the ticket above. I reached out to Luca to ask if we know what pages editors visit most often. Locally, this is how that appears now but we could probably improve it.

So I've reviewed the ticket above. They originally wanted to put a survey on the page where people used media search but decided against it, and eventually used a static main namespace wiki page with info about media search.

I paused the script just after 20220104105243 because there are millions of jobs on grafana and I wanted to let it work through those jobs first. When I restart it, I will half the batch count.

Luca also pointed me to https://phabricator.wikimedia.org/T258419, where they used QuickSurveys before on commons!

Luca from Structured Data suggested placing the survey under the options on the right side. We would also need to think about mobile view.

I added the Structured Data tag because we might want some review from them. I also pinged @Seddon on slack to see the best way to reach out to the team because I couldn't find a specific channel.

I'm looking at the file pages on Commons and I'm not sure where the survey could fit. Same with the main page.

Survey on my local mainpage:

{F34932007}Well I got a survey to appear on a file page, but there would definitely be styling needed.

I'm currently looking at https://commons.wikimedia.org/w/index.php?title=Special:Version to see which extensions Commons uses.

Apparently. you can also target pages outside the main namespace through the audience pageIds array.

I'd guess to do this we'd want to add a config option to display on file pages because on other wikis we probably wouldn't want to display them outside the main namespace.

I found where we filter for main namespace in includes/SurveyContextFilter.php:L35.

I'm re-running the script and still seeing these errors. I don't have a lot more information about why, or how many images are getting correctly scanned. I'd like the script to give more feedback in the long run.

I just started the job at 20210430135017.

We are discussing this task in the engineering meeting. Some outstanding questions:

Hmm so I was able to confirm that the api key is current.

@TAndic @eigyan I can actually confirm that I made an editor account on https://ca.wikipedia.beta.wmflabs.org/, made 5 edits, and then saw the survey on https://ca.wikipedia.beta.wmflabs.org/wiki/Applebutter.{F34915694}