I'm guessing that in Wikidata it's not expected behavior for an entity to have an assigned ID but no structured content, as appears to be the case in WikibaseMediaInfo before any structured data is entered, and so entityLoaded would never fail to fire for a valid entity under normal conditions. Is that correct?

In T228978#5375173, @Addshore wrote:

I haven't seen the hook in #2 used at all anywhere).

When reproducing locally, this error output appears in the console: https://phabricator.wikimedia.org/P8818

@Addshore do you feel like having a look at the above considerations and those patches, i.e. https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/526117/3 and https://gerrit.wikimedia.org/r/526174, and providing some guidance what would be the right (tm) way to achieve what we want to achieve?
It looks like both hooks would lead to a success, but there must be some reasons why two separate hooks exist. We've come up with some interpretations for our use, but still are not sure which direction we should continue in.
Thanks sir!

What I see on the recording is basically what this task describes, great stuff @Rosalie_WMDE !
One think I noticed, that it looks like the "Add qualifier" was clicked twice, i.e. there are two "Property" fields and two "remove" icons. This is due to accidental double clicking for reporting, right?

My two cents to the recent discussion:

1. The change https://gerrit.wikimedia.org/r/523176 is interesting, in particular the fact it has been accepted by ULS maintainers. In T222790 we've been asking them if the "old" behaviour was intended or a mistake, and haven't got the clear answer from the respective team yet. I also note (as an "interesting" observation, I do not claim it is correct and have no opinion on this whatsoever) that "reverse-interpreting" the changes to the said ULS method I referred to in T222790#5198012 seemed to lead to the conclusion that the "old" behaviour ("fil" included in result) was actually intended (at some point in time). It looks now the Language team might have identified the desired behaviour. In this case it would be great to have an answer posted in T222790 a well. Apologies from my side for not mentioning T222790 in the task description here, it would have been helpful.
2. Re fixing ULS, and not changing Wikibase. I do not oppose such approach. When this story has been being defined the approach considered was a bit different though. Please bear with me when I try to rephrase the approach to the issue here that I had in mind when writing the task. I do not claim it is better, just bringing it up for your consideration here @Ladsgroup and @alaa_wmde. One could look at Wikibase using ULS as a black box, which provides language codes (and possible other language information). In design terms one could think of the "language information source" interface, which Wikibase knows and uses. There could be multiple services serving as a "language information source" for Wikibase, one based on ULS could be easily imagined. Services like ULS should in my eyes be complete agnostic to the fact there is a software like Wikibase that uses them for the particular need. Wikibase does have its own definition of the "valid/allowed/correct" language code, which, again in my opinion, should stay inside Wikibase, i.e. other services, be it ULS, or anything else should not conform to Wikibase conditions, as they simple couldn't even know they exist. Regardless what "service" is used as a "language information provided" I'd argue Wikibase should not just accept its output but always apply its own filtering etc.

Caveat: The link is added by the FileExporter extension which currently does not have access to FileImporter's configuration.

merci beaucoup @hashar!

Thanks gentlemen!

@Ladsgroup: thanks. It seems to me that fixing the problem quickly first, and then spending some more time without a pressure how to resolve the issue "the right way" would be a very sensible approach.

Thanks @fsero!

Dear UniversalLanguageSelector folks, it's me again.
It looks that with https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/UniversalLanguageSelector/+/523176 being merged it looks we might have got an answer on the question?

Thanks for the work and your thoughts on this so far @akosiaris and @fsero!
Would you be able to estimate when the new version of the service could be available? We'd gladly take the bad news, no need for optimistic estimates :)

Thanks @Ladsgroup, @Pablo-WMDE and @Lydia_Pintscher. Some, hopefully useful, addition:

• The thought to re-consider the said module/widget has been triggered by the discovery that on client-only (in Wikibase terms) wikis only module/frontend parts of Wikibase related to the link item widget are used, although way more JS code is being loaded, which is suboptimal from the performance perspective. This task as it was originally created might be focusing too much on the possible solution instead of the problem that lead to it. This should be made more clear, thanks again to @Ladsgroup for taking time to discuss it today IRL.
• When tackled, the issue will be worked on by Wikidata engineering team at WMDE, given (apart from the obvious fact it is Wikibase software issue) we have the best understanding of Wikibase (front-end) code, and also have the overview of the ongoing front-end development happening around Wikibase, which might become relevant factor in making decisions and choosing the implementation details
• We will step back and look at the problem we intend to solve (see my first bullet point), and define it on Phabricator more precisely. This very task is likely going to be closed as it is now, in case we decide on different path of solving the underlying issue. The issue we focus on is indeed not related to the design of the widget.

@alaa_wmde @Ladsgroup: for the issue of WikibaseClient loading way too much stuff, and how to solve, could you please work together on defining the problem space, and the sensible first iteration of what we would like to achieve? It seems like a dedicated "epic"/"story"/however you want to label it, in any case a separate phabricator ticket, would make the most sense to not miss the focus of this broader ticket.

Trying the ping again: @hoo, @vvv, @csteipp, @Legoktm: any idea why mediawiki.org and wikidata.org might behave in "special" way (and arguably incorrect)?
Also, I dare to ping @Reedy as the second most active code contributor according to github, maybe he has an idea what the issue here, or know who would be able to say something about.

It might be that it is already handled as intended in all contexts (all API endpoints etc, all special pages) but then this task would at least about checking and ensuring it (e.g. automated tests) in a systematic way

Note that the change quoted above was premature and it is not true that all Wikibase browser tests have been migrated to nodejs. We still have quite some work there.
We intend to restore said daily job soon, but let's track this separately, as the status of this task has been clearly confusing.

I am an engineering manager at WMDE, and Jakob's line manager. By submitting this request I approve it at WMDE's end.

@sbassett minor update: we haven't deployed the service to use it on test.wikidata.org. There have been some additional technical issues on the way (with the deployment infrastructure, not with our service), and it is a middle of summer holiday season here at WMDE, so things move slow. We should do the test deployment to test.wikidata.org in a week or two. For the completeness, we'll update on this here when it has happened.

That sounds good @MoritzMuehlenhoff, thanks!

Re being in WMDE group being equivalent to being in NDA group, just for the record: there are/were members of WMDE non-engineering staff (Product Management, for example) who were in WMDE group but haven't signed the NDA ldap group's NDA. This could be seen e.g. with https://tools.wmflabs.org/wmde-access/ So technically there would be some users who shouldn't just go to the NDA group, and are not subject to this request.

Thanks @Ladsgroup for merging the quickfix.

Changed the priority to match the priority of the task this one is blocking.

@Steinsplitter apologies for the bug, it got overlooked when fixing T224303. The bug fix is on its way.

In T225004#5291475, @MoritzMuehlenhoff wrote:

Hi Leszek,
we have two ways to approach this: If you specifically only need Logstash access, we can extend the configuration for Logstash to check the membership of the wmde LDAP group in addition to wm/nda. Or if we add all WMDE engineers to the NDA group, then they'll also be able to access the additional services listed at https://wikitech.wikimedia.org/wiki/LDAP/Groups. Let me know your preference/intention.

In T207168#5290126, @Lydia_Pintscher wrote:

Ok cool! Then let's go ahead.

Thanks @Majora, this is indeed bad. Apologies for the situation! I'm looking into the bug.

Thanks for explanation. I was just not sure if this was intended or not, as I do remember that the removal of ruby Wikibase stuff has already been claimed premature earlier. Having those jobs restored would be cool, but if it waits a few days that'd be cool.
For clarity, once resolved, we should probably close this task ,as it seems to bring more confusion than value.

So we've removed the daily Wikibase ruby job, although we're far from having migrated all ruby test to node.

@ArielGlenn have you maybe had a chance to discuss this topic in the SRE round?

waaat, I was looking at what could be wrong in the code, and missed the Depens thing. thanks for fixing.

Tests failing in a why that suggests something is not fully working, hence putting this back to Doing.

Tests failing in a why that suggests something is not fully working, hence putting this back to Doing.

Thanks for the update @sbassett. I've also received similar information via email from @Reedy, which is much appreciated. Given this information we've been confident we're on a way, and so we were not bugging you on this review for last few weeks.

In T221764#5267856, @Lea_Lacroix_WMDE wrote:

Update: we are about to start migrating property terms this week. The original planned date was 19th of June. Due to WMF US Staff holiday, there will be no deployments today. Therefore, we are going to switch the migration on (write both stage) of property terms one day later than planned, on Thursday the 20th of June. All other dates regarding migration remain the same for the moment.

https://gerrit.wikimedia.org/r/517836 seems to have solved the problem

I believe the only code change that Wikibase might be facing might then be a new to have a DB connection for wb_terms table server, and the DB connection for the regular mw tables server. This is not going to be rocket science, but also not something that would just work out of the box.

In T68108#5267982, @WMDE-leszek wrote:

In T68108#5262336, @jcrespo wrote:

The move would likely require some changes to the Wikibase code

Could you clarify why? As all other hosts seem to be ok with wikibase server for wikidata being on a separate database? Does MCR use wikibase differently or is it something else? Maybe having 2 wikibase services to use? Note again we notified of this need months in advance during planning phase, as new features require usually extra resources.

What I had in mind is wb_terms (and its successor being introduced currently) are also used in queries that do joins with Mediawiki's "standard" tables, e.g. page or revision. This code/queries will not continue to work if the wb_terms table of commons gets moved to another server than the one mw tables are.

In T68108#5262336, @jcrespo wrote:

The move would likely require some changes to the Wikibase code

Could you clarify why? As all other hosts seem to be ok with wikibase server for wikidata being on a separate database? Does MCR use wikibase differently or is it something else? Maybe having 2 wikibase services to use? Note again we notified of this need months in advance during planning phase, as new features require usually extra resources.

In T223270#5267696, @noarave wrote:

• Mediawiki Docker setup (or access to log files in other setup)

Adding to what @Addshore mentioned above, note that github hints that the "dbs" fork is quite behind the upstream, WMDE-provided, images: "This branch is 46 commits behind wmde:master. "
This makes it significantly harder to reason about, given we're talking some non-current version of images. Given the limited resources we all have, we'd rather focus on issues in the current upstream version. We hope you understand this being a reason to decline the bug report for us.

Thanks all!

https://gerrit.wikimedia.org/r/517385 helped and unblocked merging Wikibase patches. Thanks to @hashar and @zeljkofilipin for help. And special thanks to @Michael for discovering the root issue.
Leaving the ticket open until the backport patches get merged.

Pinging maintainers according to https://www.mediawiki.org/wiki/Developers/Maintainers: @hoo, @vvv, @csteipp, @Legoktm: any idea why mediawiki.org and wikidata.org might behave special (and arguably incorrect)?