Thanks for the detailed responses, but I feel that one piece is still missing in the puzzle. As you can see, for instance here, this is literally a CGI. Ancient technology where the webserver was supposed to start the command-line executable, pass query string as a parameter and use output as a resulting http response.
Did I get it right that you want @MBH to rewrite his CGIs using .NET Core Minimal APIs?

I'm a colleague of @MBH, we are developing tools together. I apologize for a long text, but it looks like

• we know how things are working for us, but lack basic knowledge of docker/heroku (even after reading all links you provided)
• you know how infrastructure works for you, but have no idea how we (mis)use it.

Let's try to close this knowledge gap :)))

Per Ghuron, an alternative workflow would be:

Theoretically one can build a bot executable using C# compiler (mcs is mono 6.08), but I believe we need the latest 6.12 for many other reasons.
I was having problems with 6.08 connecting sites, that require TLS 1.3 (which is rare at the moment, but would not be so rare in the near future)

In T232565#8335248, @Mvolz wrote:

Can you make a separate API request for the corresponding property text based off the q number you receive?

Confirm that code executed in tf-mono68 can access mediawiki API
Will continue testing

Although the idea of building things INSIDE execution container seems awkward to me, upgrading mono to the latest stable (6.12.0.182) sounds like a right thing to do

I want to emphasize that right now https-connections cannot be established which makes tf-mono68 practically useless in toolforge environment. Steps to reproduce:

1. Create a test.cs text file with the following content:

class Test {
  public static void Main(){
    new System.Net.WebClient().DownloadString("https://en.wikipedia.org/");
  }
}

2. Execute mcs test.cs to compile it to test.exe
3. Run toolforge-jobs run test --command "mono test.exe" --image tf-mono68 and check test.err

When trying to run mono executable inside this image, I'm getting exception:

thanks, I did. Once container will be available, I will start testing it

mono68-sssd looks like a good starting point, please let me know when I can use toolforge-jobs run ... --image tf-mono68 thanks!

Some identifiers (such as P356) are case-insensitive, but others (such as P3083) are case-sensitive. We have no control over it, since those are identifiers in external sources. As a result, when I'm searching for haswbstatement:"P3083=* b Cen" I'm getting:

1. https://www.wikidata.org/wiki/Q2072696#P3083 (* b Cen)
2. https://www.wikidata.org/wiki/Q4836952#P3083 (* B Cen)

and there is no way I can limit my search to the first result only.
Either case-sensitivity parameter in search or corresponding property text in results (so I can filter second entity by myself) would work for me.

this tool was PoC successfully superseded by YurikBot and can be deleted

There are ~30 tools (web UI, bots, etc) written in C# using DotNetWikiBot
Most of them are actively used by ru-wiki community
I can see that there are no tf image with mono support
Please advise how we can proceed

When I was creating this ticket, I was not aware of T118935. Anyway, my proposal consists of 3 things:

1. Instead of Pxxx show label in the corresponding wikiproject language
2. Instead of Qxxx show label in the corresponding wikiproject language
3. Show an affected item with the label e.g. Cîteaux Abbey (Q843261) when it is different from the item, that corresponds to the article

I believe T118935 covers 1 and 2, but does not cover 3. The latter one becomes important in the situation when infobox utilizes various connected wikidata items (e.g. place of birth). It might not be a problem 7 years ago when T118935, but certainly a problem now at least in ru-wiki, where infoboxes involves processing dozens of items, e.g.

I want to describe how things are (almost) implemented in ru-wiki to give some context regarding how this functionality can help community.
The desired user story is pretty simple:

1. When I'm editing an article and have a red link, but knows that article exists in another language wikipedia, I'm leaving a hint there (e.g. {{Interlanguage link|Arthur Kaan|de||vertical-align=sup}} in en-wiki)
2. When someone else is clicking on the red link ...
   1. ... browser open an article creation page with some content pre-generated based on wikidata (e.g. correct infobox) and language sitelinks are already displayed
   2. ... preview shows infobox populated from the specified wikidata item
   3. ... and save article, it automatically placed as a sitelink in the corresponding wikidata item

In order to achieve this, in ru-wiki we have several components:

1. Most of infoboxes accept parameter from= (e.g. {{Infobox enzyme|from=Q1262082}}), so it can be used to display infobox for (yet) non-existing article
2. Infobox modules, that shows info from wikidata are able to show redlinks and associate it with wikidata element (both visually and via url parameter from=Q1117754) see
   
3. JS-gadget, that
   1. looks for interlanguage links such as {{Ill|HD 165493|it}}
   2. gets wikidata element (Q2399598) for https://it.wikipedia.org/wiki/HD_165493
   3. adds from=Q2399598 to red link url
4. JS-gadget, that invoked when a new article page is opened (e.g. url contains redlink=1) and url contains from=Qxxx This gadget
   1. finds an appropriate infobox and put it in the article text with from parameter
   2. generates human-readable article preamble based on wikidata item
   3. shows language links
      
5. Bot that monitors newly created articles and
   1. adds article as a sitelink to the corresponding wikidata item (e.g. https://www.wikidata.org/w/index.php?diff=1397217877)
   2. removes from= parameter from the infobox (e.g. https://ru.wikipedia.org/w/index.php?diff=113447669)

How native support of from=Q1234 (or itemid=Q1234 like proposed above) url parameter could simplify those five components?

1. If parser functions invoked on non-existing page will consider this url parameter, we won't need from= parameter in infoboxes. Ru-wiki already invested into that, but a lot of other communities does not, so this will save them a lot of effort
2. Nothing really changed here. Server-side generation of redlinks with new parameter is bad idea, so we recommend to keep it on js-gadget (code is trivial, but unfortunately depends on the formatting, that differs from project to project)
3. Same here, we will keep it in js-gadget, (wbgetentities + above)
4. We will still have to do A and B, but C will be done for us by engine
5. That piece can be done by engine easily. Existing bot runs once a day, so for majority of new articles sitelinks are added manually. We are planning to rewrite bot so it will monitor Page creation stream, but streams are not particularly reliable

So to summarise, this feature (with small additional contribution from community) can effectively helps editors to create new articles more efficiently and provides clear incentives to enhance wikipedia-wikidata integration.

Apparently implementation of wikibase.getSitelink() in client/includes/DataAccess/Scribunto/mw.wikibase.lua is utilizing trackUsageForTitleOrSitelink() in client/includes/DataAccess/Scribunto/WikibaseLanguageIndependentLuaBindings.php
Implementation of methodtable.getSitelink() in client/includes/DataAccess/Scribunto/mw.wikibase.entity.lua is much more brutal, it just fires addSiteLinksUsage on any access to sitelinks table

In 2019 based on discussion with @Yurik and @Iniquity (see https://www.mediawiki.org/wiki/Topic:V0hrxe3xa924u9xm) my intern created proof of concept: https://dspull.toolforge.org/
Main scenario is:

One of tools I'm maintaining (https://www.mediawiki.org/wiki/MW2SPARQL) is effectively WDQS federated endpoint. WDQS has white list of supported endpoints and http://tools.wmflabs.org/mw2sparql/sparql is there. Who can replace it to http://mw2sparql.toolforge.org/sparql ?

Just in case you need more queries to test this:

Yes, thank you. Corrected title

I can see that protecting of portion of item might be a problem for wbeditentity, one have to diff changes in order to understand what exactly is beeing updated. I think for the time beeing, we can only apply this mechanism for wbset* actions. Majority of vandalism comes from UI and other user-friendly helper tools, wbeditentity is almost exclusively used by bots or complicated instruments like SourceMD (for items creation).
It is better to have easier half of this feature in production this year, rather than wait for full feature for another 2 years...

The only reason I mention Special:WhatLinksHere is that I thought it would be easier to implement, which doesn't seems to be the case.
If some can expose reasonable subset of haswbstatement - I would be more than happy.

looks like its working again

Can reproduce all three links above on Chrome 76.0.3809.136 (Official Build) (64-bit) ChromeOS

Mine wdt:P disappear

I definitely can re-import mine, but existing wdt:Pxxx makes operating existing bots much more difficult than it should be

Still getting a few dozens of wdt:Pxxx, query:

@Anomie, you were kind to fix T215444, maybe you can take a look on this one? We have thousands of potentially affected categories and problem reveal itself only on small random subset of them. If you need any additional traces in lua - I'm here to help :)

Any category here except for Википедия:Страницы с ошибками скриптов, использующих Викиданные‎. The problem is that errors disappears after ?action=purge

Thank you very much!

@Krenair, I assume that only home directory was not created:

ghuron@tools-sgebastion-07:~$ ls /data/project/dspull/
ls: cannot access '/data/project/dspull/': No such file or directory
ghuron@tools-sgebastion-07:~$ id
uid=19300(ghuron) gid=500(wikidev) groups=500(wikidev),50380(project-tools),53381(tools.mw2sparql),53738(tools.wdml),54111(tools.dspull)

Look at the query that is used to get missing articles for "List of articles every Wikipedia should have" https://quarry.wmflabs.org/query/26700 There are 2 joins:

• For iwlinks/wb_items_per_site join I have to do either CONCAT ('Q', ips_item_id) or TRIM('Q' FROM iwl_title)
• For wb_items_per_site/page join I have to do REPLACE(ips_site_page, ' ', '_') or vice versa and potentially take care of non-default namespace

Isn't it ironic that wikidatawiki_p.wb_items_per_site has 2 columns that is potentially joinable with wikipedia tables and both of them requires format transformation?

thanks, no problems now!

Still experiencing problems with hewiki_p (and sometimes eswiki_p). The rest is ok

А чем конкретно Вас не устраивает дамп?

It looks like it depends whenever PHP7 setting is tuned on or off. When its off, problem disaperars, when its on, action=purge any article that uses it and you will get "false" when calling mw.ext.data.get()

works for chromeos 70.0.3538.110

Fixed for all dbs mentioned above

@Aklapper I've found it in my labs project mw2sparql (it was bgwiki_p initially 07:34 UTC today) but since that I'm repoducing this in quarry (see link above). Sounds like operation issue with db replicas

Currently it is impossible to add/remove sitelinks in Russian UI. I would appreciate if someone will look into that as soon as possible

I'd like to explain another scenario, where this functionality can be useful. Ru-wiki infoboxes sometimes add categories to the corresponding articles. Sometimes we do this based on wikidata, but we are pretty much limited to a few cases, defined in P1465, P1792, P1464 or very special usage of P910 (e.g. if astronomical object belongs to some constellation). If I want to automatically place categories like Q8454640 to the films, directed by James Marsh, I probably would need a designated property. Alternatively I should be able to do it via extracting items that has P971:Q2720543 and simultaneously P971:Q29017630. This may sound like reinventing SPARQL queries, but getting results of getBacklinks(Q2720543) and getBacklinks(Q29017630) and performing list intersection in lua would work for me.

In T185313#3915129, @Uzume wrote:

This seems like a more generalized version of T99899 (see also T149108).

Well, although I can some similarities between this one and T99899, I believe there are different use cases involved. Lookup for external identifier is mostly needed in javascript (e.g. notify user that wikidata instance with the same imdb-id already exists) and can be implemented either via Markus resolver or directly via sparql endpoint (e.g. https://ru.wikipedia.org/w/index.php?diff=84445799). I can imagine some use cases where it would be nice to have it in lua (e.g. for additional decoration of wikipdate articles), but its exotic stuff. In contrast, building lists andbased on wikidata is more straightforward case and must be pure severside (for performance reasons), so I'd see a need for backlinks in lua.

I'm personally missing ability to get "what links here" (see T185313)