Page MenuHomePhabricator

Allow local wikis to set up elections
Open, HighPublicFeature

Description

Feature summary

While for some elections (such as the Board elections) it makes sense to use a "central" wiki like votewiki (vote.wikimedia.org), for elections that only pertain to individual wikis, we should allow the wiki functionaries of those wikis to set up the election locally.

Use cases
Currently, the only way a WMF wiki can set up an election is to request it to be created on votewiki and this involves a busy process in which Trust-and-Safety team also has to be involved. This has limited the adoption of SecurePoll. At least in the case of fawiki (which has been the leading community using SecurePoll for local elections), this process has been perceived as cumbersome.

Although I cannot find a task on Phabricator about this, I recall a conversation about this in the past where two arguments were raised against this:

  • Some elections require CLI level actions (such as setting up encryption) and having control on the creation of elections (and effectively, rate-limiting the elections using the T&S process) can help ensure these needs are met and we don't end up with a backlog of CLI level actions
  • Installing SecurePoll databases on individual wikis would complicate replication of local DBs into Cloud-Services and can pose potential security risks

It appears that the second argument is not valid anymore, as we have robust data replication scripts that accurately control which tables can or cannot be replicated (such as check_private_data.py and maintain-views.yaml). The first argument still holds, except most elections are not encrypted (I believe the only ones encrypted are those for the Board).

Benefits

Allowing locally-defined elections has the added advantage that other wikis won't see them in the list of elections and the list would be less cluttered. Also, the current process in which votewiki's interface language must be change to that of specific communities during their election period would be unnecessary. This also allows local wikis to create more elections, use SecurePoll to convert some of the current voting processes (adminship, etc.) into ones with close ballots, conduct aggregated surveys about key project ideas, and so forth.

Other considerations

The expansion for SecurePoll to local wikis can be done in a calculated way, e.g. at first only wikis with local CUs could be allowed to use it, because those CUs can also work as scrutineers of the votes and are already NDA'ed to see the private details of votes.

Event Timeline

Aklapper changed the subtype of this task from "Task" to "Feature Request".Feb 7 2022, 8:51 PM

@Huji: Thanks for reporting this. For future reference, please use the feature request form (linked from the top of the task creation page) to create feature requests. Thanks.

@Aklapper I did consider that. But this is not really a "software feature" which is that form is called. Should we rebrand it as "Feature request", given that its structure would apply to config changes too? I am going to update this task to use that structure anyway.

jrbs triaged this task as High priority.Sep 22 2022, 10:32 PM

I have tried to understand the requirements of this task, but as I am not very familiar with the given infrastructure, I have some questions

  • Do I understand correctly that vote.wikimedia.org is hosted outside of the regular Wikimedia Cluster (the one that hosts the Wikipedia projects)?
  • Regarding the encryption topic: I only found a dependency to the GPG CLI tool within the code. Is this serverside requirement an issue on the Wikimedia Cluster?

Regarding the "key blockers":

  • CLI operations -> As far as I could see from the code there are no CLI operations required for an encrypted election, besides having GPG set up on the server during installation time. Which CLI operations are meant? Is it the scripts like purgeDecryptionKeys.php and purgePrivateVoteData.php? They do not look like something you need to perform for a regular election.
  • DB replication -> This is said to be solved. Can someone confirm this please? What exactly are/were the issues? As far as I could see an encrypted election would just store the votes as encrypted values in the DB. I didn't see any issued with that in means of replication.

Given the title of this task, I am tempted to say one can use SecurePoll already in local wikis, as long as the extension can be deployed to the cluster.
Maybe someone could describe the current process of "requesting" an election as a numbered list and then state which steps need to be improved.

Spoke about this in our call. I'll answer what I can and pass anything else onto @drochford

I have tried to understand the requirements of this task, but as I am not very familiar with the given infrastructure, I have some questions

  • Do I understand correctly that vote.wikimedia.org is hosted outside of the regular Wikimedia Cluster (the one that hosts the Wikipedia projects)?

I don't know the complete answer to this technically, but it is not an SUL wiki (as in, it does not make use of the centralauth system). Voting in an election at the moment requires one to "jump" to votewiki, which means the election needs to be set up there, and that can only be done by a small number of people.

  • Regarding the encryption topic: I only found a dependency to the GPG CLI tool within the code. Is this serverside requirement an issue on the Wikimedia Cluster?

I don't know the answer to this. @drochford

Regarding the "key blockers":

  • CLI operations -> As far as I could see from the code there are no CLI operations required for an encrypted election, besides having GPG set up on the server during installation time. Which CLI operations are meant? Is it the scripts like purgeDecryptionKeys.php and purgePrivateVoteData.php? They do not look like something you need to perform for a regular election.

There are a few of these:

  • DB replication -> This is said to be solved. Can someone confirm this please? What exactly are/were the issues? As far as I could see an encrypted election would just store the votes as encrypted values in the DB. I didn't see any issued with that in means of replication.

I think this was explained later in the task description (as we discussed on the call).

Given the title of this task, I am tempted to say one can use SecurePoll already in local wikis, as long as the extension can be deployed to the cluster.
Maybe someone could describe the current process of "requesting" an election as a numbered list and then state which steps need to be improved.

The current process of requesting an election is quite obscure since SecurePoll requires those CLI steps (setting up encryption) and because it requires access to votewiki, which is not an SUL wiki (and so that access must be requested, at the moment from the Foundation). There are some security reasons for that, mostly that the access one gets as an electionadmin is akin to CheckUser access — arguably even more powerful since one doesn't need to actively run a tool to see voter data (more on that in T270342#6702969).

The ideal set up process would be thus:

  1. A wiki decides it needs to run an election (e.g. an ArbCom election, a request for adminship, etc)
  2. The wiki nominates some user(s) to serve as admins to set up the election - eligibility criteria would ideally be more configurable than it is now since some wikis have different criteria, though that's a different question
  3. The wiki nominates some user(s) to view the voter list (to scrutinise it for duplicate votes etc.)
  4. The selected admins then tally the election and share the result with their community

Probably missing some steps here, but I think that's the overall ideal flow.

  • Regarding the encryption topic: I only found a dependency to the GPG CLI tool within the code. Is this serverside requirement an issue on the Wikimedia Cluster?

I don't know the answer to this. @drochford

My understanding, and correct me if I'm wrong @jrbs , if HW manage to get the tally timeout issue working in the UI, we would no longer need the CLI interface to manually decrypt the votes for tallying. Is that correct Joe?

What other actions utilize the CLI interface?

My understanding, and correct me if I'm wrong @jrbs , if HW manage to get the tally timeout issue working in the UI, we would no longer need the CLI interface to manually decrypt the votes for tallying. Is that correct Joe?

I believe so yeah. At the moment it appears to try, but silently times out. I believe that's the case for elections above some relatively small number of votes (100 or so).

What other actions utilize the CLI interface?

The big ones for WMF global elections are the creation of tables for the voter eligibility (probably not solvable in the scope of this task) - the technical process is covered here: https://wikitech.wikimedia.org/wiki/SecurePoll#Voter_eligibility

There's also the GPG key generation. I'm not sure that's possible to move into the interface though.

I will discuss with the AHT Team and get back shortly.

I'll set something up with Tim and David to chat on this.

Some notes from our video call today:

  • The background job for tallying does not seem to time out anymore. This was successfully tested by @jrbs with about 1500 encrypted votes. I already found some code changes regarding this issue from about one year ago during my research in late December '22. We now assume tallying to be no issue anymore. Yet, some UX improvements would be nice (e.g. some progress information)
  • The next blocker is now the process of "setting up" a vote, which currently requires to involve technical staff
  • Checking "rules" like "not be blocked in more than one project; not be a bot; have made at least 300 edits before X across Wikimedia wikis; have made at least 20 edits between X and Y." is very expensive.
  • @tstarling identified three main parts in the process of compiling a list of eligible users for a vote:
    • Overall performance: For the "rule based" compilation, lots of queries are required across different DBs and tables. Can take up to 6h. Probably not suitable for job queue background processing.
    • CLI only process: Compilation must be done on the CLI; but there is already PopulateVoterListJob. We first need to check the state of this code.
    • Separate tables for each election: Requires a patch (Gerrit) to the codebase and schema changes to the DB for single votes; Could be superseded by "summary tables" or "external service data" (see below)
  • Proposal: Change the way the "rule based compilation" is performed, e.g. by relying on secondary data sources ("user activity summary tables" within MW core; Analytics team data), rather than on primary ones (actual revision table of various wikis from within the cluster). This would improve the overall performance and maybe allow for "on the fly" compilation during the set up process.

Next steps:

  • Hallo Welt team to check the current PopulateVoterListJob implementation and come up with a technical proposal to change the "rule based compilation" process during the set up phase.
  • @jrbs or @drochford to contact the analytics team and check if their data can be used to feed the compilation process.

Some notes from our video call today:

  • The background job for tallying does not seem to time out anymore. This was successfully tested by @jrbs with about 1500 encrypted votes. I already found some code changes regarding this issue from about one year ago during my research in late December '22. We now assume tallying to be no issue anymore. Yet, some UX improvements would be nice (e.g. some progress information)
  • The next blocker is now the process of "setting up" a vote, which currently requires to involve technical staff
  • Checking "rules" like "not be blocked in more than one project; not be a bot; have made at least 300 edits before X across Wikimedia wikis; have made at least 20 edits between X and Y." is very expensive.
  • @tstarling identified three main parts in the process of compiling a list of eligible users for a vote:
    • Overall performance: For the "rule based" compilation, lots of queries are required across different DBs and tables. Can take up to 6h. Probably not suitable for job queue background processing.
    • CLI only process: Compilation must be done on the CLI; but there is already PopulateVoterListJob. We first need to check the state of this code.
    • Separate tables for each election: Requires a patch (Gerrit) to the codebase and schema changes to the DB for single votes; Could be superseded by "summary tables" or "external service data" (see below)
  • Proposal: Change the way the "rule based compilation" is performed, e.g. by relying on secondary data sources ("user activity summary tables" within MW core; Analytics team data), rather than on primary ones (actual revision table of various wikis from within the cluster). This would improve the overall performance and maybe allow for "on the fly" compilation during the set up process.

I think a lot of this applies only to "global" elections which have voters from multiple projects (and thus require a script setup to compile the voter lists). This task would be explicitly about "local" elections which are run on a per-wiki basis.

Local elections are typically organised and run by community already, with the exception of the setup on votewiki which needs WMF support (or, technically speaking, someone with access to the maintenance server and to votewiki). See for example the coordination task for fawiki 2021 elections (T292685) -- the vast majority of that work was led by volunteers, and T&S was really just there to unblock votewiki accounts and set up the election. Likewise the enwiki 2022 elections were a similar process, though had to be encrypted. There is some argument that could be made about having a disinterested third party holding the decryption key, but that could still be T&S or really anyone who can give it to the local community for tallying the election.

Looking at the task description ... there might not actually be anything blocking this other than the CLI issue, plus the need to add the appropriate userrights to allow people to create polls). I would have some questions for exploration though:

  1. Is the method for connecting local wikis to votewiki likely to complicate this? Could you for example make a local election but also allow that wiki to take part in global elections? That is not clear to me.
  2. Is the SecurePoll extension already installed on Wikimedia wikis by default?
  3. Are there any other Security ramifications to consider?

Scripts to calculate eligibility is a normal thing. One also has to do very similar things to complile some MassMessage lists to send out invitations to a context that is happening for another year, etc. Mostly in bigger communities there is expertise to do that.

Scripts to calculate eligibility is a normal thing. One also has to do very similar things to complile some MassMessage lists to send out invitations to a context that is happening for another year, etc. Mostly in bigger communities there is expertise to do that.

That sort of seems like a make-work blocker, I don't think it is in the way of this. That function It is something people may choose to do or not do. If they choose to make a manual list SP already supports importing such a list. Being able to set the checkbox/radio button/etc automatic eligibility selectors is the default / primary eligibility method.

Given what I have learned during my research for T335201, I think we currently have the following situation:

  1. SecurePoll is installed on most Wikimedia Wiki Projects (and probably configured properly, e.g. regarding gpg)
  2. SecurePoll can be used for "local elections" on those wikis already
  3. In a "local elections", the list of eligible users must either be provided manually via Special:SecurePoll/votereligibility of can be calculated in a background job from the local wikis user base and under consideration of "rules" (PopulateVoterListJob). Both functionalities seem to work properly at the moment.
  4. In case of "local elections", there is no notification implemented for the eligible users. This could be a dedicated improvement task. The "global election" case has such a step. This must be considered in such a task.
  5. Tallying using the JobQueue seems to work properly at the moment, even for larger elections. Even encryption of "local elections" using GPG should be possible with the current setup already.

Unfortunately, this is blocked on our move away from GPG, since GPG only exists on the votewiki install of SecurePoll. More info in T209892#8863583

This does also mean that a potential pilot would also be stalled on that task's resolution.