I was typing this for the other ticket but will type here instead. I know that the task title specifically talked about min-edits which is why I tried to clarify in the discussion on that task. While I know that this script creates a voter list creates a list of voters using global edits that isn't actually the problem that I need solved. I need the list to be CHECKED globally

For example: someone who made 200 edits on enWiki (20 of which were within the past X months) and 150 edits on esWiki can vote with their global account from Meta (and not have to go back to enWiki or esWiki to vote). That's the need we're having here, not just the list which I'm confidant we can create.

We will have two elections, one is tentatively scheduled to start May 3rd (FDC Elections) for one week and one that is tentatively scheduled to start May 17th (Board Elections) for two weeks. Both elections will be run by the same election committee and most likely use the exact same voter list.

Because of that timing I can not promise you for a 4 week freeze, we were blocked by the board for so long as they figured some things out that we are under a time crunch. I plan for this to be one of the first things decided upon by the committee however who should be meeting for the first time at the end of this week. My plan is to have the voter requirements frozen by Monday (preferably Friday but depends a bit on availability) which will be 3 weeks before the first election.

In T95262#1184798, @Jalexander wrote:

I need the list to be CHECKED globally

That's an architecture statement, not a product statement.

For example: someone who made 200 edits on enWiki (20 of which were within the past X months) and 150 edits on esWiki can vote with their global account from Meta (and not have to go back to enWiki or esWiki to vote). That's the need we're having here, not just the list which I'm confidant we can create.

It sounds like you want someone to be able to click a link on meta and have that take them to the ballot page. That is a product requirement with several implementation options.

Or you could make it a requirement that you want to have a link on any wiki that takes the user to the ballot page. That is almost equivalent since it could be implemented by linking to the page on meta described above. Neither requirement seems to imply a need to implement authorisation from a global list in SecurePoll_Election::getQualifiedStatus().

The reason we have jump pages is so that we can implement the voter requirements which reference a particular "home" wiki. In particular, in the 2013 requirements had "not be blocked on the project you are voting from" and "not be a bot", which are both evaluated dynamically at the time of the vote, on the local wiki. If those requirements are dropped, then we won't need jump pages any more.

The 2013 vote already had the requirement "not be blocked on more than one project", which is implemented in SecurePoll_Auth::getCentralBlockCount(). This seems redundant with "not be blocked on the project you are voting from". If the "not be a bot" requirement was similarly made global, then we could drop local jump pages and have everyone vote from meta.

In T95262#1188481, @tstarling wrote:

In T95262#1184798, @Jalexander wrote:

I need the list to be CHECKED globally

That's an architecture statement, not a product statement.

I apologize, it's the way it is pictured in my mind, I have no specific desire for how it's implemented.

For example: someone who made 200 edits on enWiki (20 of which were within the past X months) and 150 edits on esWiki can vote with their global account from Meta (and not have to go back to enWiki or esWiki to vote). That's the need we're having here, not just the list which I'm confidant we can create.

It sounds like you want someone to be able to click a link on meta and have that take them to the ballot page. That is a product requirement with several implementation options.

Or you could make it a requirement that you want to have a link on any wiki that takes the user to the ballot page. That is almost equivalent since it could be implemented by linking to the page on meta described above. Neither requirement seems to imply a need to implement authorisation from a global list in SecurePoll_Election::getQualifiedStatus().

Aye, if this can be done without a global list then I'm perfectly happy. I think you have the thoughts correct. I would put the requirements here, in order, as:

1. [requirement] Have a global user click a button on meta and be able to vote without ever going back to their home wiki.
2. [very nice to have] Have a global user click a button/link on any SUL wiki and be able to vote without having to move out of the flow (sending them to the meta check page is perfectly fine).

The reason we have jump pages is so that we can implement the voter requirements which reference a particular "home" wiki. In particular, in the 2013 requirements had "not be blocked on the project you are voting from" and "not be a bot", which are both evaluated dynamically at the time of the vote, on the local wiki. If those requirements are dropped, then we won't need jump pages any more.

The 2013 vote already had the requirement "not be blocked on more than one project", which is implemented in SecurePoll_Auth::getCentralBlockCount(). This seems redundant with "not be blocked on the project you are voting from". If the "not be a bot" requirement was similarly made global, then we could drop local jump pages and have everyone vote from meta.

Aye, I agree, it's redundant. I'm perfectly happy to drop the "not be blocked on this wiki" requirement and just rely on the global block count. I'm happy to drop the local bot check requirement. It would be preferred if we can make the "not be a bot" check globally because otherwise we'd have to manually check.

The final vote qualifications for editors are below:

You may vote from any one registered account you own on a Wikimedia wiki. You may only vote once, regardless of how many accounts you own. To qualify, this one account must:

not be blocked on more than one project,

not be a bot,

have made at least 300 edits before 15 April 2015 across Wikimedia wikis,

and have made at least 20 edits between 15 October 2014 and 15 April 2015.

So "not be a bot" now means "not be in the bot group on any wiki"?

In T95262#1219547, @tstarling wrote:

So "not be a bot" now means "not be in the bot group on any wiki"?

Correct, that's actually always been the rule (at least for the past couple board elections that I've worked with), though it doesn't sound like it was auto checked, just checked by the committee.

"have made at least 20 edits between 15 October 2014 and 15 April 2015" -- one might expect this time interval to end at 2015-04-15 23:59:59 or thereabouts, but the 2013 requirements interpreted the equivalent statement as ending at the start of the named day, i.e. in this case 2015-04-15 00:00:00. Any opinion on which is correct?

I would do it at 23:59:59. I actually think this caused a couple annoyances last year and we manually added people who met the requirements because of the change.

The plan at the moment is:

• Implement a central list feature in SecurePoll_Election::getQualifiedStatus(), linked to gu_id. This is necessary despite what I said earlier since not all qualified voters will have a user_id on metawiki at the time the edit counts scripts are run. So the options are either to create a user_id for all of them, or implement a central list.
• Have an election on metawiki jump to an election on votewiki, like in 2013
• No elections configured on the other wikis, unlike in 2013

Current status:

• populateEditCount.php is running and is almost done
• I'm working on rewriting voterList.php to compile a list by gu_id, ignoring the local user table
• I would like a retrospective review of https://gerrit.wikimedia.org/r/#/c/206740/ (e.g. @ori), but for bugs only, given the schedule is extremely tight if this is going to happen by May 3.

In T95262#1240134, @tstarling wrote:

The plan at the moment is:

• Implement a central list feature in SecurePoll_Election::getQualifiedStatus(), linked to gu_id. This is necessary despite what I said earlier since not all qualified voters will have a user_id on metawiki at the time the edit counts scripts are run. So the options are either to create a user_id for all of them, or implement a central list.
• Have an election on metawiki jump to an election on votewiki, like in 2013
• No elections configured on the other wikis, unlike in 2013

Current status:

• populateEditCount.php is running and is almost done
• I'm working on rewriting voterList.php to compile a list by gu_id, ignoring the local user table
• I would like a retrospective review of https://gerrit.wikimedia.org/r/#/c/206740/ (e.g. @ori), but for bugs only, given the schedule is extremely tight if this is going to happen by May 3.

Thank you very much Tim, I know the time table is tight and I apologize for that. The entire time table for the election is significantly tighter then it should be given delays earlier in the process. If you need anything from us or additional resources let me know.

Change 206999 had a related patch set uploaded (by Tim Starling):
[WIP] Central list feature and BV 2015 list script

https://gerrit.wikimedia.org/r/206999

@Jalexander: For the 2013 Board election there were bulk emailing scripts, will they be needed this time around?

The tightest part of the schedule seems to be the 2-day period between May 1 to May 2, when all translations and vote setup for the FDC election needs to occur, as well as community questions and discussion. It's not just software development that is strained. May 3 is a Sunday, so I don't know how that is going to work.

In T95262#1240777, @tstarling wrote:

@Jalexander: For the 2013 Board election there were bulk emailing scripts, will they be needed this time around?

We will be doing the bulk emailing but only for the board election (probably about half way through the 2 week election), not for the FDC Election, so we have a bit more time. Initially I didn't expect those had to change much but I guess we'll actually have to tweak it to look at the new global list you're creating. I can make a new task for that if you'd like.

The tightest part of the schedule seems to be the 2-day period between May 1 to May 2, when all translations and vote setup for the FDC election needs to occur, as well as community questions and discussion. It's not just software development that is strained. May 3 is a Sunday, so I don't know how that is going to work.

Yeah :-/. It is not in anyway what we wanted, we tried to start this in Late December/Early January however the board was not quite ready until much more recently. We're working to set up the election and translations as much as possible in advance (I have professional translations going out multiple times this week to help) but it will still be tight.

Change 206999 merged by jenkins-bot:
Central list feature and BV 2015 list script

https://gerrit.wikimedia.org/r/206999

Change 207064 had a related patch set uploaded (by Tim Starling):
Central list feature and BV 2015 list script

https://gerrit.wikimedia.org/r/207064

Change 207065 had a related patch set uploaded (by Tim Starling):
Central list feature and BV 2015 list script

https://gerrit.wikimedia.org/r/207065

Change 207064 merged by jenkins-bot:
Central list feature and BV 2015 list script

https://gerrit.wikimedia.org/r/207064

Initial voter list compilation should be done in about 2 hours. It has about 40000 qualified voters and counting.

What is the plan for configuring the election? Do we have the inclusion (staff, devs, etc.) and exclusion (bots etc.) lists?

As it stands, people who have local accounts that are not attached to CentralAuth but are qualified nonetheless will not be in the list. I'm not sure if that would affect enough people to require a fix in software, or if we can just add any such people to the inclusion list. Maybe @Legoktm can comment?

In T95262#1244257, @tstarling wrote:

Initial voter list compilation should be done in about 2 hours. It has about 40000 qualified voters and counting.

Beautiful, thank you very much. At the moment should it 'work' at that point? (if I set up a test election with that voter list for example)

What is the plan for configuring the election? Do we have the inclusion (staff, devs, etc.) and exclusion (bots etc.) lists?

The current plan is for me to configure the election (we have an initial test one up at the moment though obviously with out the current voter list). The new interface has the ability for election administrators to add to the inclusion and exclusion list without having to touch the database. That should continue to work with this set up but we will test that tomorrow and if it doesn't doing so via the old ,db, level is still doable.

Traditionally (at least over the past couple years) we've done that on a 'as needed' level throughout the election because the number of people needing it was so low however to jump start the process I plan to send an email to wikitech-l and wmfall tomorrow with a link to the eligibility checker to see if anyone thinks they should be added.

As it stands, people who have local accounts that are not attached to CentralAuth but are qualified nonetheless will not be in the list. I'm not sure if that would affect enough people to require a fix in software, or if we can just add any such people to the inclusion list. Maybe @Legoktm can comment?

This shouldn't really affect anyone, the only accounts still outstanding with the problem are accounts that shouldn't be voting (Steward and Oversight which I'm lagging at fixing right now) and if there are a couple stray unexpected ones I think we should be able to handle on an as needed special basis. Lego can confirm that though.

It's easy enough to query the s7 wikis for unattached accounts, since they are on the same cluster as centralauth so I can do a join. This gives some idea of the scale of the problem. There are about 770 unattached accounts, but none would be qualified. There are only 6 with non-zero bv_short_edits, and at least 3 of those are bots. Two of those bots would qualify on the basis of edit count: they are called "Flow talk page manager" and "New user message". There is only one verifiable human with non-zero bv_short_edits: ديفيد عادل وهبة خليل. He has bv_short_edits=1.

In T95262#1244289, @tstarling wrote:

It's easy enough to query the s7 wikis for unattached accounts, since they are on the same cluster as centralauth so I can do a join. This gives some idea of the scale of the problem. There are about 770 unattached accounts, but none would be qualified. There are only 6 with non-zero bv_short_edits, and at least 3 of those are bots. Two of those bots would qualify on the basis of edit count: they are called "Flow talk page manager" and "New user message". There is only one verifiable human with non-zero bv_short_edits: ديفيد عادل وهبة خليل. He has bv_short_edits=1.

hmm, thanks, wonder why those aren't attached but certainly suggests it's unlikely to be an issue big enough to code in.

Change 207065 merged by jenkins-bot:
Central list feature and BV 2015 list script

https://gerrit.wikimedia.org/r/207065

In T95262#1244259, @Jalexander wrote:

Beautiful, thank you very much. At the moment should it 'work' at that point? (if I set up a test election with that voter list for example)

Yes, I will deploy the relevant code change now so that it will work as soon as voterList.php is done. It is untested, but three people have squinted at the code, so fingers crossed.

voterList.php is done now, 53712 users. Do you want the list dumped to a file for review?

In T95262#1244305, @tstarling wrote:

voterList.php is done now, 53712 users. Do you want the list dumped to a file for review?

If easy that would be great, if not that's fine.

In T95262#1244305, @tstarling wrote:

voterList.php is done now, 53712 users. Do you want the list dumped to a file for review?

Also, thank you, I know this was a 'hurry up and hack this together' type deal which has happened all too often with the elections. My hope is to have a more permanent election committee so that these things can be thought of long ahead of time :-/ but I really appreciate the work making sure we get what it done.

I'm currently working under the assumption that this was the latest code deploy so posting here but could be completely unrelated. We had a test election running between meta and voteWiki which was working just fine but now when someone attempts to vote they are getting a server error page.

Error:

Request: GET http://meta.wikimedia.org/wiki/Special:SecurePoll/vote/334, from 10.64.0.102 via cp1054 cp1054 ([10.64.32.106]:3128), Varnish XID 2996319900
Forwarded for: 198.73.209.37, 2601:9:8480:c6a:9801:29e5:4769:a124, 10.64.0.102, 10.64.0.102
Error: 503, Service Unavailable at Wed, 29 Apr 2015 09:13:02 GMT

Replication:

1. Attempt to vote at http://meta.wikimedia.org/wiki/Special:SecurePoll/vote/334

Or

1. Go to meta SecurePoll at http://meta.wikimedia.org/wiki/Special:SecurePoll
2. Click vote on last poll "Test S/O/N poll"

[vote requirements are 1 edit and account before 4/15. It is not using the central vote list and was set up before earlier Tuesday]

I'm heading to sleep but wanted to drop this here in case anyone was able to see the issue.

In T95262#1244259, @Jalexander wrote:

Traditionally (at least over the past couple years) we've done that on a 'as needed' level throughout the election because the number of people needing it was so low

Personal requests aren't really a good way to produce voter lists. I've added a link to the requirements; one of them is «have commit access and have made at least one merged commits in git to Wikimedia Foundation utilized repos between 15 October 2014 and 15 April 2015».

Hundreds of people qualify for this. Emails can be easily extracted from gerrit and matched server-side with SUL accounts; wikitechwiki usernames too, but then it's harder to match them with SUL accounts. People have multiple emails so this won't be a complete list, but it's certainly fair and saves a lot of manual work.

Queries with an unprivileged user find 342 emails:

$ { QUERY='status:merged AND NOT owner:l10n-bot  AND NOT ownerin:wmf'; ssh -p 29418 nemobis@gerrit.wikimedia.org gerrit query "$QUERY age:14d AND NOT age:74d" ; ssh -p 29418 nemobis@gerrit.wikimedia.org gerrit query "$QUERY age:74d AND NOT age:134d" ; ssh -p 29418 nemobis@gerrit.wikimedia.org gerrit query "$QUERY age:134d AND NOT age:194d" ; } | grep -E "^\s+email:" | sort -u | wc -l
282
$ { QUERY='status:merged AND NOT owner:l10n-bot AND ownerin:wmf'; ssh -p 29418 nemobis@gerrit.wikimedia.org gerrit query "$QUERY age:14d AND NOT age:74d" ; ssh -p 29418 nemobis@gerrit.wikimedia.org gerrit query "$QUERY age:74d AND NOT age:134d" ; ssh -p 29418 nemobis@gerrit.wikimedia.org gerrit query "$QUERY age:134d AND NOT age:194d" ; } | grep -E "^\s+email:" | sort -u | wc -l
60

In T95262#1244520, @Jalexander wrote:

I'm currently working under the assumption that this was the latest code deploy so posting here but could be completely unrelated. We had a test election running between meta and voteWiki which was working just fine but now when someone attempts to vote they are getting a server error page.

Fixed in I676c15.

In T95262#1244727, @Nemo_bis wrote:

In T95262#1244259, @Jalexander wrote:

Traditionally (at least over the past couple years) we've done that on a 'as needed' level throughout the election because the number of people needing it was so low

Personal requests aren't really a good way to produce voter lists. I've added a link to the requirements; one of them is «have commit access and have made at least one merged commits in git to Wikimedia Foundation utilized repos between 15 October 2014 and 15 April 2015».

Hundreds of people qualify for this. Emails can be easily extracted from gerrit and matched server-side with SUL accounts; wikitechwiki usernames too, but then it's harder to match them with SUL accounts. People have multiple emails so this won't be a complete list, but it's certainly fair and saves a lot of manual work.

Queries with an unprivileged user find 342 emails:

$ { QUERY='status:merged AND NOT owner:l10n-bot  AND NOT ownerin:wmf'; ssh -p 29418 nemobis@gerrit.wikimedia.org gerrit query "$QUERY age:14d AND NOT age:74d" ; ssh -p 29418 nemobis@gerrit.wikimedia.org gerrit query "$QUERY age:74d AND NOT age:134d" ; ssh -p 29418 nemobis@gerrit.wikimedia.org gerrit query "$QUERY age:134d AND NOT age:194d" ; } | grep -E "^\s+email:" | sort -u | wc -l
282
$ { QUERY='status:merged AND NOT owner:l10n-bot AND ownerin:wmf'; ssh -p 29418 nemobis@gerrit.wikimedia.org gerrit query "$QUERY age:14d AND NOT age:74d" ; ssh -p 29418 nemobis@gerrit.wikimedia.org gerrit query "$QUERY age:74d AND NOT age:134d" ; ssh -p 29418 nemobis@gerrit.wikimedia.org gerrit query "$QUERY age:134d AND NOT age:194d" ; } | grep -E "^\s+email:" | sort -u | wc -l
60

Thanks Nemo, that's a good idea, I'll try to get that in as soon as possible.