Problem overview

• Tallying a large, encrypted election leads to timeout. For details, see: T269029#6697060
• We implemented a speedup (T271824), but this wasn't enough to stop the timeout in production for board-election-sized polls

Current election admin workflow

• On submitting the TallyPage form, the page becomes unresponsive
• After two minutes, an unformatted timeout error is shown
• The admin must instead use the maintenance script, tally.php. The script:
  • Accepts either an XML dump of election or the election name
  • Calls Election::tally
  • Outputs the result as HTML or text

Suggested improved workflow

• If election has too many votes:
  • Show a formatted error immediately (T276354 - we also define "too many votes" in that task)
  • Start a process to tally asynchronously
  • Store results so tallying only happens once
  • Show progress/results on TallyPage

Start a process to tally asynchronously

We can create a Job, e.g. TallyElectionJob that performs the same function as tally.php, and push this to the job queue from the form's submit callback, TallyPage::submitForm, when an election is too large.

There is one existing job in SecurePoll, PopulateVoterListJob, which is enqueued by VoterEligibilityPage.

Store results so tallying only happens once

Rather than outputting the result, as tally.php does, the TallyElectionJob could store the results in the database. The securepoll_properties table would be the obvious place to do this, since the tally result is data about a (finished) election, and this table is used as a freely-structured store of data about an election.

We don't currently store the results of an election, so we should consider if there are security or privacy concerns around storing the result. I'm inclined to think not, for the following reasons:

• The result will only be stored once an election has finished
• The result is a summary and doesn't specify who voted for what
• Doing this will not change who can access the result. As before, the result can be accessed by election admins via the TallyPage or by people with server access (who can run the maintenance script or see the database directly).

What would we store? ElectionTallier has two methods for outputting the results: getHtmlResult and getTextResult. Neither of these is ideal: getHtmlResult commits us to a particular HTML structure, and getTextResult could involve some awkward parsing.

One thing we could do instead is add a new method: getJsonResult. This could be a fair bit of work: we'd need to implement getJsonResult for each type of Tallier, and we'd probably want each type of tallier to know how to convert the JSON back to HTML, since the results for different tally types will have different JSON structures.

Show progress/results on TallyPage

The PopulateVoterListJob spawns several jobs and stores its progress in securepoll_properties so that the VoterEligibilityPage can report on progress.

Since TallyElectionJob would be a single job, we can't report progress in the same level of detail, but we could at least report whether the tallying is underway. We could use securepoll_properties in a similar way, and set a property, e.g. tally-started. TallyPage would then have 3 states:

• No tally has been started; show the form
• Tally has started but not finished; show a message that tallying is underway
• Tally has finished; show the result

Optional extra: notifying admins when tallying is done

We could use the Echo extension to notify admins when the tally job completes. One example of a job that does this is in the MachineVision extension.

How it works in MachineVision:

• Configure a new type of Echo event via hooks
• Define how the notification should be displayed
• Create the Echo event when the job is finished

Although this looks like a neat experience, I'm not sure it's needed for SecurePoll, since it is a fair amount of overhead for a small number of notifications. I think it would only be useful once for each election, when the first admin creates the tally. Subsequent admins would just be able to view the result straight away.

@Niharika @jrbs @drochford A few product questions arose from the engineering meeting:

• Is it OK to store the results of a finished election in the database, security- and privacy-wise? If not it would be difficult to display the results to the user once the job has finished. We guessed it would probably be OK since the results will only be stored once the elections is finished, would only contain the final counts (i.e. wouldn't contain who voted for what), and would presumably be made public soon after the election had finished anyway.
• Do we need to admins to be able to tally over and over again, e.g. after votes are struck? We'll assume yes for now, but would be helpful to know for sure.

We also noted that there's a size limit for the pr_value field in the securepoll_properties table, which is where we'd store the result - but we wouldn't expect the election results to be anywhere near it.

In T276355#6906432, @Tchanders wrote:

@Niharika @jrbs @drochford A few product questions arose from the engineering meeting:

• Is it OK to store the results of a finished election in the database, security- and privacy-wise? If not it would be difficult to display the results to the user once the job has finished. We guessed it would probably be OK since the results will only be stored once the elections is finished, would only contain the final counts (i.e. wouldn't contain who voted for what), and would presumably be made public soon after the election had finished anyway.

I think that's fine for the reasons you mentioned.

• Do we need to admins to be able to tally over and over again, e.g. after votes are struck? We'll assume yes for now, but would be helpful to know for sure.

I'll assume yes too.

We also noted that there's a size limit for the pr_value field in the securepoll_properties table, which is where we'd store the result - but we wouldn't expect the election results to be anywhere near it.

Awesome.

@Tchanders I agree with your comment about the Echo notification being a lot of overhead for little gain. I'll be happy for us to not do that and instead use the underway message to let the user know to come back to the page in a little bit. Does that sound good?

I agree with your comment about the Echo notification being a lot of overhead for little gain. I'll be happy for us to not do that and instead use the underway message to let the user know to come back to the page in a little bit. Does that sound good?

That sounds fine to me!