When I download the example video you linked to I can see GPS coordinates in the global metadata labeled as LOCATION and LOCATION-eng when I probe the file with ffprobe. Here is the output that I get:

Here are the transcoded videos: https://drive.google.com/file/d/11rW4A0XqAWhJWYjNBHwIBcYm9yRTu_F0/view?usp=sharing

I cannot overwrite the files since they were not uploaded by me. I can provide the files to you if that's okay, though.

In my testing when working with a single input and output file ffmpeg seemed to keep most global standard metadata.

Is this something that can still be reproduced? I don't have access to the deployed V2C, but downloading and transcoding all of these videos with my local version of V2C doesn't result in black videos.

It looks like this bug was introduced when we added short circuit logic at the beginning of the round that checks if all remaining hopefuls are eligible to be elected or eliminated with the current remaining seat count. This change was introduced to workaround the infinite loop bug if I recall correctly (http://phabricator.wikimedia.org/T291821). If I move that logic to the end of the round (and update the local round object) instead then we get an elected message as well. There is the problem though that the quota doesn't align with the new elected winner, which looks suspect. Is this fine? If not, would it be fine if we also added a note that says, "Candidate 'A' elected to fill remaining seats"? The same could also apply where we eliminate remaining hopefuls once we reach the seat limit.

Sounds good to me. I split the PR in two:

I have a patch open for this: https://github.com/WikipediaLibrary/externallinks/pull/441

@Amdrel https://phabricator.wikimedia.org/p/Amdrel/ I'm looking through
the order of operations for reviewing your PRs and just wanted to make sure
I was reviewing them in the order you intended:

Is this the correct order?

1. https://github.com/WikipediaLibrary/externallinks/pull/429
2. https://github.com/WikipediaLibrary/externallinks/pull/431
3. https://github.com/WikipediaLibrary/externallinks/pull/440
4. https://github.com/WikipediaLibrary/externallinks/pull/439

We also need to run the new fill commands as well so the program totals continue to render properly. How far back do we want to go for archiving aggregates? Would aggregates older than a year work?

It looks like all of the ballots have the same root cause.

After investigating 12_9_4996_1067098093 I believe the new result may actually be more correct. Candidate 9 does appear to have slightly more votes because there is single ballot with multiple candidates chosen 1 1 2 3 4 5 6 7 8 9 0 in the .blt file that includes candidate 9 in it. Candidate 9 has the same amount of single votes as 10 11 and 12 do. The votes from the referenced ballot get added to candidate 9 during round 2. I suspect what may have happened was the previous implementation using floating point numbers was unable to account for the extra low ranked vote. I would have to debug the old version to compare to be sure, but the fact that OpenSTV can reproduce this result with 15 points of precision may make sense if this is the case, though 10 points of precision should have been enough which seems odd to me.

Sorry. I have edited my above comments to include links to the after results in a user page.

The quota and votes for candidates A and B don't exactly match. I think possibly just because we need to update the precision on the tally page to reflect above changes. @STran will know more.

Also notice how before we had a final round where Candidate A is explicitly elected. We no longer seem to do that and it might look as if they are being elected with ~509 votes when the quota is ~669.

I looked into how the pagers work a bit more and it looks like we would have to render a table directly with all tallies instead of using a pager if we move everything to securepoll_properties.

If we're certain that adding another table will cause issues (albeit one that should be small) then I can explore using the properties table.

In T387701#10876614, @Dreamy_Jazz wrote:

Have we asked for DBA input on this new table?

I spent some time looking into this. I performed the investigation under the assumption that any new databases we would potentially add would be running on the same instance and disk as the current database as for this project everything runs together on the same instance to help lower costs.

As far as implementing tally modifiers in the UI goes, would it make more sense to create a new page for editing these separate from CreatePage? At least in the case of eliminated candidates it wouldn't make sense to include this during creation, though it would make sense if being edited.

I'm thinking it would make sense to move tallies into their own one to many table (election -> tallies) rather than keeping the results in securepoll_properties. We should be able to migrate all existing tallies into this new table.

I implemented the calculation of monthly totals in my patch and that solves the issue of having to pull most of the archives from object storage for the calculation of program-level totals. It doesn't result in much additional database bloat and is performant (~200MB mostly coming from user totals). Filters still work as well.

My monthly aggregate jobs are complete and I've got some stats on what these archives will look like. To work with the existing filters I have the archives split by organisation_id, collection_id, full_date and on_user_list (if false include all aggregates).

I'm worried about how this is going to work with the date range filters on pages like /organisations/<pk> and /programs/<pk> if we pulled archive .json files over XHR client-side to be rendered. We need to be able to support any range of months for the chart and totals, so the archives need to be split up to enable that. If we have archives of aggregate data split by month and data from the last 6 years is requested for an organization's collection then that could be over 60 archives that need to be downloaded to generate the graph and calculate the totals below it. I don't know exactly how large the final archives are going to be yet file size wise since I'm still running the monthly aggregate jobs against my local data.

In T370533#10629706, @jsn.sherman wrote:

verified that the migration completed and loaded up pageproject and user aggregates; we're still having timeouts on both, but we can assume that we just can't make it over the line of request timeouts. Accepting this as an improvement.

The CSV downloads for pages would have to pull from the aggregate archives since link stats per page are included, but for the website view it's grouped by project name instead.

In T370980#10626966, @jsn.sherman wrote:

Could we not provide the information via a static page, treating the archive json urls like API responses for xhr?

I've marked the PR as ready to review. I briefly looked at write performance and haven't identified any unused indexes or any other issues with the writes themselves causing slowdowns. We're still bottlenecked by SELECT operations that precede writes to both aggregates and link events.

When we archive aggregate data and remove it from the tables it won't be viewable from the programs and organisations pages anymore. How far back are we planning to keep data? We could make a required date option for the aggregate archival commands so we have control over that regardless of what retention period we decide on.

I uploaded a patch that makes the reason required only for vanish rejections while retaining the old behavior for renames.

I have a work in progress PR here that adds a couple: https://github.com/WikipediaLibrary/externallinks/pull/417

The patch I just added moves the execution of the automatic vanish to a job which can be configured to execute on Meta-Wiki.

Understood! Is there any existing mechanism to insert cross-wiki log entries easily? If not, it looks like CentralAuth handles cross-wiki actions (such as renaming users and moving their pages) by scheduling jobs and maybe a similar approach can be used for inserting the log entries for auto-approved vanishes.

From my local testing it seems like the log entry's destination is currently determined by the wiki that the renamer uses to approve the vanish request regardless of which wiki the request was originally made on by the user.

It might be worth checking the cron job lock if we don't see it complete during its next scheduled run. While debugging the job I noticed if the lock is set when restarting the containers while a job is running that they can get stuck.

In T370896#10542517, @Scardenasmolinar wrote:

In T370896#10542179, @Amdrel wrote:

If I'm reading https://phabricator.wikimedia.org/T370977 correctly we'll be modifying the aggregate jobs to run once a month rather than daily as they do now. With that change it sounds like we would need to do this automated archival once a month instead of daily so we can ensure the aggregate jobs have all of the data they need.

The aggregates jobs run daily on information from the linkevents table. The task in T370977: Create cron to aggregate old data would run monthly on the information from the *_aggregates table. Therefore, the relevant linkevent information is already stored in the aggregates table and can be disposed of. The only caveat is that some aggregate commands (mainly the pageprojects command) have been known to run for more than 24 hours, so you would have to consider that before archiving the information from the linkevent table. Let me know if you have any follow-up questions about the dataflow (it can be a bit of a headache).

If I'm reading https://phabricator.wikimedia.org/T370977 correctly we'll be modifying the aggregate jobs to run once a month rather than daily as they do now. With that change it sounds like we would need to do this automated archival once a month instead of daily so we can ensure the aggregate jobs have all of the data they need.

Implementing a toggle for public access to the dump on the edit page would not work because elections cannot be edited once they are finished. I do think a release date field makes sense, and so does doing it based on the tally date. Either of those solutions would not require us to change the current edit page locking behavior.

I found that the infinite loop issue goes away if I change the tallier to use fixed-point arithmetic instead of floats. I get the same winners as OpenSTV for 20_6_5000_1301235635.blt and 20_7_5000_425367464.blt, with the exception being the randomly chosen winners in the first ballot. The Gerrit patch I just uploaded changes the tallier to use bc functions with a scale of 9 to accomplish this. A couple of existing tests are broken with this patch since the number of rounds needed to complete the tally shrunk, but the winners and eliminated assertions are still passing.

The patch I added doesn't count partial blocks against the central block threshold if 'Must not be partial blocked' is unchecked. I saw it was mentioned in the original description that we might want an admin toggle for this. Please let me know if this is something we want for this patch.

Attached is a patch for core that fixes this issue. The issue I found was while querying page information and temporary user auto-creation is enabled, a placeholder user is created to test permissions against instead as a temporary user may have the ability to edit a page while an anonymous user cannot. However, blocks are checked against this placeholder as well, and since that placeholder user isn't associated with the request, IP blocks, XFF, and cookie blocks get skipped.

I reproduced this and I'd like to note that this bug seems only to take effect when using the standard editor, and not VisualEditor.

I took a look and the checks were already adjusted in the following patch https://gerrit.wikimedia.org/r/c/mediawiki/extensions/OAuth/+/993162 (T355378).