If screen will allow me scrollback once they're done running, I can provide a full list of the wikis which failed. (Thankfully enwiki, which is by far the largest wiki in terms of user eligibility, seems to be working, but a few other large wikis are not)

In T326408#8504656, @jrbs wrote:

Thankfully enwiki, which is by far the largest wiki in terms of user eligibility, seems to be working, but a few other large wikis are not

This feels the most weird thing for me... Surely it should work on all wikis, or no wikis? :/

Change 876194 had a related patch set uploaded (by Reedy; author: Reedy):

[mediawiki/extensions/SecurePoll@master] wm-scripts: Get Flow DB_REPLICA in a different way

https://gerrit.wikimedia.org/r/876194

In T326408#8504676, @Reedy wrote:

In T326408#8504656, @jrbs wrote:

Thankfully enwiki, which is by far the largest wiki in terms of user eligibility, seems to be working, but a few other large wikis are not

This feels the most weird thing for me... Surely it should work on all wikis, or no wikis? :/

It might fail later ... but the ones that have failed so far (the scripts are still running) failed immediately

If we're patching this now should I cancel the scripts for the time being?

Flow isn't enabled on enwiki, hence why the script runs fine... So you're fine to let that one run. I suspect, on the wikis where Flow is enabled, it will have failed.

But for proof my patch fixes the issue:

reedy@deploy1002:~$ mwscript eval.php wikidatawiki
> $flowDbr = ( new Flow\DbFactory( $wgFlowDefaultWikiDb, $wgFlowCluster ) )->getLB()->getConnection( DB_REPLICA );

> var_dump( $flowDbr->tableExists( 'flow_revision' ) );
bool(false)

> $dbFactory = Flow\Container::get( 'db.factory' );

> $flowDbr = $dbFactory->getDB( DB_REPLICA );

> var_dump( $flowDbr->tableExists( 'flow_revision' ) );
bool(true)

Why does Flow need to be so special?

In any case, the worst that might need to happen is re-running the script over the list of wikis in https://noc.wikimedia.org/conf/highlight.php?file=dblists/flow.dblist

The flow tables must exist in wikidata already, otherwise flow wouldn't have worked at all. I think the connection is probably misplaced, probably flow is on x1 and needs correct connection to correct cluster. I can take a look a bit later.

In T326408#8504704, @Ladsgroup wrote:

The flow tables must exist in wikidata already, otherwise flow wouldn't have worked at all. I think the connection is probably misplaced, probably flow is on x1 and needs correct connection to correct cluster. I can take a look a bit later.

Ignore this, I didn't reload the page to see all the changes.

Change 876194 merged by jenkins-bot:

[mediawiki/extensions/SecurePoll@master] wm-scripts: Get Flow DB_REPLICA in a different way

https://gerrit.wikimedia.org/r/876194

Change 876056 had a related patch set uploaded (by Reedy; author: Reedy):

[mediawiki/extensions/SecurePoll@wmf/1.40.0-wmf.17] wm-scripts: Get Flow DB_REPLICA in a different way

https://gerrit.wikimedia.org/r/876056

Change 876057 had a related patch set uploaded (by Reedy; author: Reedy):

[mediawiki/extensions/SecurePoll@wmf/1.40.0-wmf.14] wm-scripts: Get Flow DB_REPLICA in a different way

https://gerrit.wikimedia.org/r/876057

Change 876056 merged by jenkins-bot:

[mediawiki/extensions/SecurePoll@wmf/1.40.0-wmf.17] wm-scripts: Get Flow DB_REPLICA in a different way

https://gerrit.wikimedia.org/r/876056

Change 876057 abandoned by Reedy:

[mediawiki/extensions/SecurePoll@wmf/1.40.0-wmf.14] wm-scripts: Get Flow DB_REPLICA in a different way

Reason:

https://gerrit.wikimedia.org/r/876057

Mentioned in SAL (#wikimedia-operations) [2023-01-06T14:06:22Z] <reedy@deploy1002> Synchronized php-1.40.0-wmf.17/extensions/SecurePoll/cli/wm-scripts/ucoc2023/populateEditCount.php: T326408 (duration: 07m 09s)

reedy@deploy1002:/srv/mediawiki-staging$ cat ~/uc2023.sql 
select count(*) from ucoc2023_edits;
reedy@deploy1002:/srv/mediawiki-staging$ foreachwikiindblist flow sql.php ~/uc2023.sql | grep count
arwiki:      [count(*)] => 0
arwikiversity:      [count(*)] => 0
bswiki:      [count(*)] => 0
cawiki:      [count(*)] => 0
cawikiquote:      [count(*)] => 0
commonswiki:      [count(*)] => 1
cswiki:      [count(*)] => 0
elwiki:      [count(*)] => 0
eowiki:      [count(*)] => 0
euwiki:      [count(*)] => 0
fawiki:      [count(*)] => 0
fiwiki:      [count(*)] => 0
fiwikimedia:      [count(*)] => 0
frwiki:      [count(*)] => 0
frwikiquote:      [count(*)] => 0
frwikisource:      [count(*)] => 0
frwikiversity:      [count(*)] => 0
frwikivoyage:      [count(*)] => 0
frwiktionary:      [count(*)] => 0
gomwiki:      [count(*)] => 0
hewiki:      [count(*)] => 0
idwiki:      [count(*)] => 0
kabwiki:      [count(*)] => 0
kkwiki:      [count(*)] => 0
lvwiki:      [count(*)] => 0
mediawikiwiki:      [count(*)] => 0
nowiki:      [count(*)] => 0
officewiki:      [count(*)] => 0
orwiki:      [count(*)] => 0
outreachwiki:      [count(*)] => 0
pawiki:      [count(*)] => 0
plwiki:      [count(*)] => 0
ptwiki:      [count(*)] => 0
ptwikibooks:      [count(*)] => 0
sewikimedia:      [count(*)] => 0
specieswiki:      [count(*)] => 0
svwiki:      [count(*)] => 0
techconductwiki:      [count(*)] => 0
test2wiki:      [count(*)] => 0
testwiki:      [count(*)] => 0
tewiki:      [count(*)] => 0
urwiki:      [count(*)] => 0
viwikisource:      [count(*)] => 0
wawiki:      [count(*)] => 0
wawiktionary:      [count(*)] => 0
wikidatawiki:      [count(*)] => 3
zhwiki:      [count(*)] => 0
zhwikisource:      [count(*)] => 0

I've removed those wikidata and commons rows...

In T326408#8504821, @Reedy wrote:

I've removed those wikidata and commons rows...

I'm now re-running the scripts on commonswiki and s8 (which is just wikidatawiki).

Reedy and I spoke and we'll link up when the other scripts are finished (to survey what kind of cleanup work needs to be done), probably on Monday

Hey All,

I'm getting a few emails from people regarding this issue - is there something that could be done? Bit concerned that there are people who are encountering this issue and aren't escalating to the project team, which downstream, will effect our vote count.

@jrbs what's odd: I voted this morning, but clicking on the SecurePoll link right now, it says I'm not eligible. So either your list of voters must have changed somehow during the day or something different is wrong with the tool.

In T326408#8532853, @SNg_WMF wrote:

Hey All,

I'm getting a few emails from people regarding this issue - is there something that could be done? Bit concerned that there are people who are encountering this issue and aren't escalating to the project team, which downstream, will effect our vote count.

I'm not sure if this is an accurate assessment, sorry.

Since Flow was created (2013?) this is the first attempt at trying to automatically include those people with Flow edit counts.

The workflow for most (all?) users where this was the case prior, was to get added manually, as it's been since 2013.

It seemingly didn't work quite as expected, sure. It's unclear if it's actually making any users ineligible that would have been automatically. It would seem unlikely.

It's part of the bigger problem that we have features (both Flow and SecurePoll in this case) deployed on our projects, without any real maintainers.

In T326408#8532774, @jrbs wrote:

RWeissburg (WMF) is especially odd as this was on the list to add manually with importGlobalVoterList.php

This one is indeed odd, but I'd presume unrelated... Maybe? Unless it's a bigger issue...

In T326408#8532774, @jrbs wrote:

For the sake of troubleshooting, the users that have had to be added manually so far are:

Chrs
Contaminadas
Denny
H-stt
Kriz Ju
Legowerewolf
Nthep
Paradise Chronicle
RWeissburg (WMF)
Redwidgeon
Shushugah
Webfil

Can you provide me some details (privately if necessary) of where those users eligibility comes from? Edits on which wiki etc? Whether actual edits or whether Flow edits.

I'll have a look and see if we can work out a pattern.

In T326408#8532928, @Johannnes89 wrote:

@jrbs what's odd: I voted this morning, but clicking on the SecurePoll link right now, it says I'm not eligible. So either your list of voters must have changed somehow during the day or something different is wrong with the tool.

That is extremely weird. I wonder if this might be related at all to the config revert performed this morning T327196

The correct list seems to be hooked in:

wikiadmin@10.64.48.58(votewiki)> select * from securepoll_properties where pr_entity=1414 and pr_key not in ("wikis","gpg-sign-key","gpg-encrypt-key");
+-----------+-----------------------+-----------------------------------------------------------+
| pr_entity | pr_key                | pr_value                                                  |
+-----------+-----------------------+-----------------------------------------------------------+
|      1414 | admins                | JSutherland (WMF)|HakanIST|AmandaNP|DerHexer|Operator873  |
|      1414 | disallow-change       | 0                                                         |
|      1414 | encrypt-type          | gpg                                                       |
|      1414 | must-answer-all       | 0                                                         |
|      1414 | need-central-list     | ucoc-vote-2023                                            |
|      1414 | need-list             | 1414/list/need-list                                       |
|      1414 | not-sitewide-blocked  | 1                                                         |
|      1414 | remote-mw-script-path | https:$wgServer/w                                         |
|      1414 | request-comment       | 1                                                         |
|      1414 | return-url            | https://meta.wikimedia.org/wiki/Universal_Code_of_Conduct |
|      1414 | shuffle-options       | 0                                                         |
|      1414 | shuffle-questions     | 0                                                         |
|      1414 | voter-privacy         | 0                                                         |
|      1414 | wikis-val             | *                                                         |
+-----------+-----------------------+-----------------------------------------------------------+
14 rows in set (0.002 sec)