I am an engineering manager at WMDE, and Jakob's line manager. By submitting this request I approve it at WMDE's end.

@sbassett minor update: we haven't deployed the service to use it on test.wikidata.org. There have been some additional technical issues on the way (with the deployment infrastructure, not with our service), and it is a middle of summer holiday season here at WMDE, so things move slow. We should do the test deployment to test.wikidata.org in a week or two. For the completeness, we'll update on this here when it has happened.

That sounds good @MoritzMuehlenhoff, thanks!

Re being in WMDE group being equivalent to being in NDA group, just for the record: there are/were members of WMDE non-engineering staff (Product Management, for example) who were in WMDE group but haven't signed the NDA ldap group's NDA. This could be seen e.g. with https://tools.wmflabs.org/wmde-access/ So technically there would be some users who shouldn't just go to the NDA group, and are not subject to this request.

Thanks @Ladsgroup for merging the quickfix.

Changed the priority to match the priority of the task this one is blocking.

@Steinsplitter apologies for the bug, it got overlooked when fixing T224303. The bug fix is on its way.

In T225004#5291475, @MoritzMuehlenhoff wrote:

Hi Leszek,
we have two ways to approach this: If you specifically only need Logstash access, we can extend the configuration for Logstash to check the membership of the wmde LDAP group in addition to wm/nda. Or if we add all WMDE engineers to the NDA group, then they'll also be able to access the additional services listed at https://wikitech.wikimedia.org/wiki/LDAP/Groups. Let me know your preference/intention.

In T207168#5290126, @Lydia_Pintscher wrote:

Ok cool! Then let's go ahead.

Thanks @Majora, this is indeed bad. Apologies for the situation! I'm looking into the bug.

Thanks for explanation. I was just not sure if this was intended or not, as I do remember that the removal of ruby Wikibase stuff has already been claimed premature earlier. Having those jobs restored would be cool, but if it waits a few days that'd be cool.
For clarity, once resolved, we should probably close this task ,as it seems to bring more confusion than value.

So we've removed the daily Wikibase ruby job, although we're far from having migrated all ruby test to node.

@ArielGlenn have you maybe had a chance to discuss this topic in the SRE round?

waaat, I was looking at what could be wrong in the code, and missed the Depens thing. thanks for fixing.

Tests failing in a why that suggests something is not fully working, hence putting this back to Doing.

Tests failing in a why that suggests something is not fully working, hence putting this back to Doing.

Thanks for the update @sbassett. I've also received similar information via email from @Reedy, which is much appreciated. Given this information we've been confident we're on a way, and so we were not bugging you on this review for last few weeks.

In T221764#5267856, @Lea_Lacroix_WMDE wrote:

Update: we are about to start migrating property terms this week. The original planned date was 19th of June. Due to WMF US Staff holiday, there will be no deployments today. Therefore, we are going to switch the migration on (write both stage) of property terms one day later than planned, on Thursday the 20th of June. All other dates regarding migration remain the same for the moment.

https://gerrit.wikimedia.org/r/517836 seems to have solved the problem

I believe the only code change that Wikibase might be facing might then be a new to have a DB connection for wb_terms table server, and the DB connection for the regular mw tables server. This is not going to be rocket science, but also not something that would just work out of the box.

In T68108#5267982, @WMDE-leszek wrote:

In T68108#5262336, @jcrespo wrote:

The move would likely require some changes to the Wikibase code

Could you clarify why? As all other hosts seem to be ok with wikibase server for wikidata being on a separate database? Does MCR use wikibase differently or is it something else? Maybe having 2 wikibase services to use? Note again we notified of this need months in advance during planning phase, as new features require usually extra resources.

What I had in mind is wb_terms (and its successor being introduced currently) are also used in queries that do joins with Mediawiki's "standard" tables, e.g. page or revision. This code/queries will not continue to work if the wb_terms table of commons gets moved to another server than the one mw tables are.

In T68108#5262336, @jcrespo wrote:

The move would likely require some changes to the Wikibase code

Could you clarify why? As all other hosts seem to be ok with wikibase server for wikidata being on a separate database? Does MCR use wikibase differently or is it something else? Maybe having 2 wikibase services to use? Note again we notified of this need months in advance during planning phase, as new features require usually extra resources.

In T223270#5267696, @noarave wrote:

• Mediawiki Docker setup (or access to log files in other setup)

Adding to what @Addshore mentioned above, note that github hints that the "dbs" fork is quite behind the upstream, WMDE-provided, images: "This branch is 46 commits behind wmde:master. "
This makes it significantly harder to reason about, given we're talking some non-current version of images. Given the limited resources we all have, we'd rather focus on issues in the current upstream version. We hope you understand this being a reason to decline the bug report for us.

Thanks all!

https://gerrit.wikimedia.org/r/517385 helped and unblocked merging Wikibase patches. Thanks to @hashar and @zeljkofilipin for help. And special thanks to @Michael for discovering the root issue.
Leaving the ticket open until the backport patches get merged.

Pinging maintainers according to https://www.mediawiki.org/wiki/Developers/Maintainers: @hoo, @vvv, @csteipp, @Legoktm: any idea why mediawiki.org and wikidata.org might behave special (and arguably incorrect)?

@hashar mentioned on IRC that the value of $wgMainCacheType has been recently changed to no longer be CACHE_NONE

Seems to also happen for mediawiki.org.
Does not happen on test.wikidata.org.

@Niharika would you mind reminding us what is the planned timeline of the deployment of partial blocks? Just so that we know how urgently the fix is needed (I do anticipate that ASAP is preferred, but given you've "only" raised the priority up to "Normal" we would also only prioritize it accordingly)

Documentation should be updated once the bug reported in T192462 is fixed

Pinging @Ebe123, did you have chance to look into the issue described above (i.e. T218191#5180154)? It's been a while since the cache version has been bumped, but the score still does look wrong on https://www.wikidata.org/wiki/Q11980. Any ideas what could be an issue there?