This investigation is still ongoing in a way but I feel like we have a good idea for where to begin working on the tool.

MacBook 15". Here's a screenshot of what I see:

In T167345#3375142, @MusikAnimal wrote:

Here's what I've got for a splash page: https://xtools-dev.wmflabs.org/ Hopefully it doesn't look that stupid... (I love the logo, by the way, that's not the stupid part!)

In T168200#3371299, @leila wrote:

Thanks @Niharika . I fixed the hyperlinks.

Missing hyperlinks in the Hindi survey

• नीति = privacy policy (lead section)
• सेवा की शर्तों = terms of service (lead section)
• आप के द्वारा सर्वेक्षण में दी गई प्रतिक्रियाएं = your survey resopnses (2nd section)
• उपयोग की शर्तों = terms of use (3rd section)
• डेटा प्रतिधारण दिशानिर्देश = data retention guidelines (4th section)

@Daniel-Barrows How do you define "popularity"? What metric(s) would you like to see and how do you imagine it being displayed?

Hi @leila, no worries at all. I'll take a look at this tomorrow first thing. Was a little busy today.

@kaldari, are we also sure that PageContentInsertComplete is *not* called during weird history merges or revision deletions, page moves etc.? Otherwise it'll have the same flaw as using rev_parent_id.

I can't reproduce either but I'm on a MacBook Pro, whereas Danny is on a somewhat dated MacBook Air. That might be the problem.

Ah, okay. Makes sense. I put the translation here because I don't have edit permissions on that page. :)

@leila Here's the translation - सर्वेक्षण डेटा किसी तीसरी पार्टी द्वारा संभाला जा रहा है। गोपनीयता।
The last word is the translation of "Privacy".
Are you sure you didn't mean to put "Privacy policy" instead of just "privacy"? Sounds a bit odd.

In T168200#3358319, @leila wrote:

@Niharika I forgot to ask you for one translation, I think. Can you translate these two sentences/phrases?

While I agree this behavior is annoying - it's probably a tough problem too. Even sites like Twitter/Facebook can't load fast enough if you scroll too quickly (that is not to say we shouldn't think about it). I don't know how often people will load the editor and scroll down very quickly - but I don't imagine that number to be very high.

I ran @Milimetric's query on enwiki with an additional rev_timestamp where clause to look at the past month's records:

mysql:wikiadmin@db1080 [enwiki]> select rev_page, count(*) as duplicate_rev_parent_id_zeroes from revision where rev_timestamp >= 20170515000000 and rev_parent_id = 0 group by rev_page having count(*) > 1;
+----------+--------------------------------+
| rev_page | duplicate_rev_parent_id_zeroes |
+----------+--------------------------------+
| 54073829 |                              2 |
| 54112112 |                              2 |
| 54112566 |                              2 |
| 54166124 |                              2 |
| 54169722 |                              2 |
| 54179829 |                              2 |
| 54188344 |                              2 |
| 54193238 |                              2 |
| 54200374 |                              2 |
| 54204127 |                              2 |
| 54257308 |                              2 |
| 54280581 |                              2 |
| 54280697 |                              2 |
| 54283421 |                              2 |
+----------+--------------------------------+
14 rows in set (53 min 42.84 sec)

Maybe it helps.

I could be wrong but here's the queries I ran on recentchanges on enwiki right now -

mysql:wikiadmin@db1080 [enwiki]> select count(*) from recentchanges where rc_last_oldid = 0;
+----------+
| count(*) |
+----------+
|   933550 |
+----------+
1 row in set (48.56 sec)

Jamesofur> Niharika: no, that release form can be ignored/eliminated

In T167901#3348937, @Securitash wrote:

What if anything am I doing right?

See also - https://99designs.com

In T167187#3345216, @dbarratt wrote:

It looks like the owner-only consumers are broken. :( I created T167813. In the meantime I'll ignore authentication.

@Jalexander would you be able to get me access to the server so I can get the real OAuth consumer?

The patch is complete except for missing illustrations which is tracked in T165541: Illustration for Syntax Highlighting beta feature.

This needs both an LTR and RTL illustration.

In T167187#3342641, @dbarratt wrote:

@Niharika so from reading T159898 and your recommendations, it looks like in this ticket I'll be rewriting the existing tool as a new tool with the same features plus the modifications listed? Are there any guidelines for how this should be done (languages, libraries, etc.) or is it up to me?

In T164441#3295011, @kaldari wrote:

@Nirzar: Small problem. There is a very similar icon being used for the refToolbar Gadget (when wizards are turned off):

The actual icon is https://en.wikipedia.org/wiki/File:Curly_Brackets.svg.

@kaldari - are we doing something like this for the NWE version too?
Assuming this ticket is only for the old wikitext editor.

Yep. This would make for a good GSoC/Outreachy project.

I'd like to point out that these tools don't really get a lot of use and it might not be worth putting in too much effort into them.

In T166400#3332466, @dbarratt wrote:

Where should the code live? Should I request a new repository on https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests ?

@TBolliger I think T166400: Split out the 3 standalone tools from the DMCA toolbox and deploy them on tool labs would be an easier task to tackle first and it will get @dbarratt familiar with the DMCA/CP takedown tools codebase among other things.
I'll be back in SF starting next Monday and will be happy to talk to David about it. Will schedule a meeting.

@eranroz Meanwhile can you help us figure out what's wrong? Thank you.

Thanks for reporting this, @Framawiki and sorry about us not keeping on top of things.
From what I can see, Eranbot is not gathering any data for French and Spanish wikis. I don't immediately see a reason for this because it's configured to work in the exact same way as English. The error logs don't seem to say anything helpful.

Not very related but WikiCite happens to use the same logo.

In T159898#3295448, @TBolliger wrote:

Great job, Niharika.

My only feedback is that I don't think we need to worry about re-writing the strategy tool.

4 story points? We aren't sticking with Fibonacci now? This task, given its current description, will earn a 5 from me.

Johan completed this a little while ago.

Also, Special:BlockList lets you search usernames and do some filtering so the number we show will have to reflect that filtering as well which makes it significantly more complicated that the autoblock count.

@kaldari, what do you think of the 'Number of blocks' count? I'm not convinced that it's going to be useful to anyone. Here's my conversation with Leon about it earlier:

20:21:52 N<Niharika> musikanimal: And do we want the total number of blocks or not? It's mentioned in the ticket.
20:21:53 M<musikanimal> yeah total number of blocks + autoblocks at the top of Special:BlockList
20:22:26 N<Niharika> musikanimal: Is that number useful?
20:22:36 M<musikanimal> that one is more just for fun, to be honest
20:22:48 M<musikanimal> there were always a be a lot of active blocks
20:23:17 M<musikanimal> not really important, I just recommended it to be consistent with Special:AutoblockList
20:23:36 N<Niharika> musikanimal: Yeah, its very very high apparently. I'm wondering if we're just adding more cruft to the software.
20:24:00 N<Niharika> Okay, let's see what kaldari has to say to this. I'll add this convo to the ticket.
20:24:13 M<musikanimal> eh, maybe, for enwiki it's pretty much a useless number, but probably not so for smaller wikis

In T107707#3267312, @Tgr wrote:

The announcement says

This extension does not give you notifications when somebody successfully logs into your account from an unknown device or IP. It is technically possible to generate those, but if somebody else has logged in, they could just as easily see those notifications and do a password reset (which the notification encourages you to do). The ideal way to handle this is to issue email notifications for this case, but since most Wikipedia accounts do not have emails associated with them, this wouldn't be useful to majority of the users. So for the time being, we have settled for not issuing these notifications.

That reasoning does not make sense to me. High-value accounts (functionaries etc) almost certainly have an email address enabled, most power users do have an email address set, so most of the people who would actually get targeted do get a benefit from such warnings (unlike failed login notifications which have very limited practical value).

Also, an attacker does not gain anything from doing a password reset. They could change the email address but then the real owner at least gets warned.

I've moved over the investigation results to the task description. Will create new tasks.

The filters could also be configurable on a JSON page. On this same config page, we might allow admins to add users to a blacklist, so that they don't show up in the report. This is to avoid having new admins or users who visit the report the first time and come to the same conclusion that User:Kaldari (or whomever) should not be autopatrolled.

They are sent externally via an API post. The highest risk spot for this is the Child Protection form where you upload the offending image to the tool and it sends it to the National Center for Missing and Exploited Children (who shares with law enforcement along with our other data and additional data they may have).

Are we doing any sort of encryption on the data before sending it? Should we? Also for when the user uploads the offending image to the tool?

I believe this can be resolved now. This time the bot ran through all projects except Biography in 15 days. The update script for Biography (using the non-promise version) is still running.

@kaldari, here's the response I got from James when I asked him if the tool can live on tool labs.

While not necessarily against it I'd have to check with legal on that to be sure. While we're not storing anything in the way of private data we are processing it (offending images are uploaded/processed and sent externally for example and IP data is processed) and we'd be storing OAuth login information for people who could use it to do Checkusers or access suppressed information on the sites. I'm not sure what level of exposure we're ok with on labs given that it may be a bit more exposed then, say, production infrastructure

@nzr Can we get some design feedback on this? There are screenshots above.

Is this task for writing the code that makes the extension a beta feature or for making the deployment patches?

In T141154#3271854, @Trizek-WMF wrote:

On the project page, it is written "The bot currently only posts on English-language Wikipedia."

I don't think the scope of Community-Tech is English Wikipedia only. When this service will be available for all wikis?

Syntax highlighting will help you easily distinguish article text from wikitext

It's not clear to me what that means. The article text is in Wikitext. They aren't different entities. Every link, citation, template, they all work together to form the "article text".

The task description says -

if it's not there, and the user requested alerts, send an email with the IP and other details of the login.

In T164178#3256813, @kaldari wrote:

@Niharika: Since using promises provides such a dramatic speed improvement, I would hate for us to throw that away for the majority of WikiProjects. Here's a slightly hacky idea that we could implement quickly:

• Change ReportUpdater::updateReports() to skip any projects that have over 1 million pages.
• Create a command-line script called checkBigReports.php (or something like that) that only handles projects with over 1 million pages (or is even hard-coded to handle just WikiProject Biography) and uses a non-promise version of getMonthlyPageviews().
• Add checkBigReports.php to the crontab.

The bot is currently using promises for fetching redirects and it's...lightning fast somehow...Examples below -

In T159898#3255735, @TBolliger wrote:

I don't think there is a need for the tools to live on the private server (since the tool itself doesn't store any private data) but I will confirm this with James.

I believe the CP tool needs to live on a private server because we store a hashed version of the offending image(s).

I added the new notification to the page. Can someone check it and make sure I didn't do anything silly with the translate tags? I have no experience with it. And I believe someone will have to mark the new version for translation?

This is more of a combined investigation for this ticket and T159467: Investigate Child Protection Takedown Form for requested updates.

In T164178#3246056, @Stevietheman wrote:

In T164178#3245629, @Niharika wrote:

@Stevietheman I can't find a reference for it now, but the pageviews API does do caching on their end so caching on our end might not be as big a speed boost as it looks.

Two possible issues here:

1. How durable is the caching? I have noticed this process running for a few hours, then waiting a day and running a few more hours. At least with a temp table, it would be like a dedicated cache continuously until disposed of.

Good point. This could speed up the bot a bit.

In T164178#3245637, @Niharika wrote:

In T164178#3245627, @MusikAnimal wrote:

In T164178#3245463, @kaldari wrote:

@Niharika: Let's try throwing a gc_collect_cycles() at the end of the foreach ( $pages as $page ) { loop and re-running WikiProject Biography.

I did a quick test on my local and gc_collect_cycles didn't seem to do anything, returning 0 (nothing collected) every time. So I guess hold off on that for right now. I did find this Guzzle issue, which is probably related. We're not using GuzzleHttp\Pool but I'm suspicious that the same problem persists – that resources aren't released after the requests have finished. Going to try a few other things...

Thanks for looking into this! Lets see if unset() does anything for us.

In T164178#3245627, @MusikAnimal wrote:

In T164178#3245463, @kaldari wrote:

@Niharika: Let's try throwing a gc_collect_cycles() at the end of the foreach ( $pages as $page ) { loop and re-running WikiProject Biography.

I did a quick test on my local and gc_collect_cycles didn't seem to do anything, returning 0 (nothing collected) every time. So I guess hold off on that for right now. I did find this Guzzle issue, which is probably related. We're not using GuzzleHttp\Pool but I'm suspicious that the same problem persists – that resources aren't released after the requests have finished. Going to try a few other things...