I've submitted this to the community wishlist again this year: https://meta.wikimedia.org/wiki/Community_Wishlist_Survey_2021/Miscellaneous/Checking_if_a_page_exists

I'm not sure how useful this is for Wikidata, but if we do want to support JD we may want to have the option of modified Julian date, which is more commonly used in practice in astronomy. It's a simple offset between them.

In T261275#6497436, @Lydia_Pintscher wrote:

In T261275#6476146, @ChristianKl wrote:

I'm not sure this is the best way to frame the problem. The acceptance criteria don't cover the case where an article gets deleted but the redirect stays.

Hmmmm. Is it even possible to delete an article and leave a redirect? Where would it be?

Personally, I would like to see us going in a different direction - we should stop excluding namespaces. For the 'Draft' namespace in particular, I don't like it, but it's unlikely to go away any time soon. So if we want to encourage people to use Wikidata information in articles, without having to constantly define the QID, then we should allow sitelinks to the draft articles. They would only be temporary after all, since enwp seems to periodically delete drafts.

In T233520#6616386, @Ladsgroup wrote:

I think this is another issue then the one originally reported (the end-user issue is the same though). I think we should open a new ticket.

I've been adding infoboxes on Commons by finding the categories a different way (walking the category tree), when that edit is done it seems to fix the problem. However, I can't always do that, since I'm using page_props to find the candidates to add infoboxes to! See related discussion at https://commons.wikimedia.org/w/index.php?title=User_talk:Mike_Peel&oldid=506602467

Another example: https://www.wikidata.org/wiki/Q100535082 (created 18 October), linking to https://commons.wikimedia.org/wiki/Category:J%C3%BCdenstra%C3%9Fe_(Gotha) .

In T254280#6546354, @abian wrote:

In T254280#6545624, @Lydia_Pintscher wrote:

Is the page info just out of date? Does it get overwritten by namespace-wide protection and just doesn't reflect that?

Luckily or unfortunately this happens on all wikis (also Wikipedia) and all protected/semi-protected namespaces (also the MediaWiki namespace). The (dis?)information page only seems to take into account the individual protection status of the page.

In T254280#6545675, @Mike_Peel wrote:

Possibly related, if you try to protect a property (e.g., https://www.wikidata.org/wiki/Property:P7006 ) now you see this:

There's no longer an option to semi-protect, but 'allow all users' seems to be possible?

In T258354#6563989, @Lucas_Werkmeister_WMDE wrote:

Well, IIRC we’ve had problems with MassMessage being rate-limited in the past, so it’s probably better to be safe than sorry and still add it to the group. I’ve left a message on Ymblanter’s talk page.

@Xqt Thanks! I was trying title() (as per wd_item.title() ), but I didn't try .title. Is the introduction of SiteLink objects documented somewhere please?

Possibly related, if you try to protect a property (e.g., https://www.wikidata.org/wiki/Property:P7006 ) now you see this:

Raising this to high priority, this is really important to properly support coordinate systems for other planets without mistakenly implying that they are on Earth.

Yay! @Addshore @Lydia_Pintscher This configuration change can now be made.

I don't know how to move this forward now, I can't see how this close is compatible with $wgNamespaceProtection. Any thoughts?

The RfC was closed with " In this discussion, I see rough consensus to generally semi-protect all property pages. The arguments for protection are that properties are highly visible; they are rarely edited by IPs and new users, and most of these edits get reverted, so that there is no obvious benefit. The main oppose argument is that many editors in the projects, who are not autoconfirmed on Wikidata, will not be able to edit properties, and this may affect descriptions on the less common languages. These arguments are valid and clear, they have been discussed for a sufficient period of time, and the opposers are clearly in minority.

Possibly this is still happening, see https://commons.wikimedia.org/wiki/Category:Serangoon_Secondary_School - sitelink was added on Wikidata on 14 August, but the query in https://bitbucket.org/mikepeel/wikicode/src/master/query_infobox_candidates.py is not retrieving it. Could someone check this example please?

Also, at https://commons.wikimedia.org/w/index.php?title=Category:San_Francisco&action=info the page image was added via changing the image on Wikidata so it wasn't a panoramic one and then doing a null edit to the category - the actual edits to the page didn't change anything while the image was still a panoramic one.

I've tested this a bit, and it looks like null edits do update the page info, e.g. at https://commons.wikimedia.org/w/index.php?title=Category:Lovell_Telescope&action=info . I've also posted a notice about this at https://commons.wikimedia.org/wiki/Commons:Village_pump#PageImages_enabled_for_categories .

Fantastic, thank you! I'll check more cases in the next few days, and will post back here if I find any issues.

In T198716#6397827, @Jdlrobson wrote:

Note this is enabled but comes with a big caveat - a page image is only derived from images within a page, not the pages from categories.
The image for the page image must be inside the article itself
e.g.
https://commons.wikimedia.org/wiki/Category:Coats_of_arms_of_presidents_of_Chile?action=info

Just to let you know that I've started at RfC about bot-syncing Wikidata and enwp descriptions at https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(proposals)#Synchronising_short_descriptions_and_Wikidata_descriptions

Additional discussion at https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(proposals)#Synchronising_short_descriptions_and_Wikidata_descriptions

In T184000#6003787, @Mike_Peel wrote:

In T184000#6003783, @DannyS712 wrote:

What about a bot importing the local descriptions from enwiki to wikidata? That way enwiki still has control over the content, but wikidata still has accurate and updated descriptions (that are then provided to commons, etc)

I can write a bot that will do that. It would still suck, as it's duplicating descriptions rather than having a single place that the descriptions are updated at, and I expect that it would be opposed for that reason. But if you would support it, then I can write it up and propose it.

What would happen if a bot reached a rate limit? Would its edits just be delayed, the same as what happens currently when maxlag is reached, or would the edits be dropped/result in an error message? Hopefully it would be the former.

There is now an RfC at https://www.wikidata.org/wiki/Wikidata:Requests_for_comment/General_semi_protection_for_all_property_pages - once that has finished and been closed I'll update this ticket accordingly.

OK, back to pinging @Lydia_Pintscher and @Lea_Lacroix_WMDE then (please?). ;-)

@Jdlrobson I think part of it is removing P 373 from that config, and that alone would be a good step forward. However, the other part is to follow P 910/P 1754 items through to a different Wikidata item, and to fetch the Commons sitelink from that - and that's probably more complicated/requires code change rather than just a config change.

@Jdlrobson I saw you were working on the sidebar recently (for the language vs. other sites order), is this something you/your team would be interested in?

In T198716#6235577, @Jdlrobson wrote:

@Mike_Peel is this still needed? If so @Ramsey-WMF is this something your team or our team (@ovasileva ) could look into?

Seems so, but that was the phab ticket immediately after this one! ;-)

In T246970#6213955, @Marostegui wrote:

I ran that query on an idle host and it took 25 minutes, which is right on the edge of the 30 minutes killer Quarry has, so given that those hosts are shared, it is entirely possible that it goes over 30 minutes and gets killed by Quarry.

I am still getting the same error at https://quarry.wmflabs.org/query/40539 . Trying on toolforge now...

@Lydia_Pintscher Yes, editable by autoconfirmed users. The discussion is still ongoing, and I'm not quite sure how to close it, see https://www.wikidata.org/wiki/Wikidata:Project_chat#When/how_to_close_this?

@Lydia_Pintscher Yes, it's related to https://commons.wikimedia.org/wiki/Template_talk:Wikidata_Infobox#Performance_questions

@Addshore The property-term user right is something that would have to be granted so that people could edit property labels, right? I think that's going too far for now as it would have to be given to quite a few editors. $wgNamespaceProtection sounds good.

OK, I think I'm just wasting CPU time by trying to run this query at the moment. I'll pause {{Wikidata Infobox}} deployment on Commons until things are running better. Best of luck with the debugging, thank you for your work, and please comment on this ticket when the situation is better.

I'm now running my query (https://quarry.wmflabs.org/query/40539) directly on toolforge via the grid engine, and I'm still getting "pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')" (after running for ~2 hours). Should I open a separate ticket for that? Something doesn't seem right here still, though.

If you go to e.g., https://en.wikipedia.org/wiki/Raymond_McLeod (which isn't linked to Wikidata yet), there is a section 'Languages' with a button 'add links' in the bottom of the left-hand sidebar. That used to be present on Commons as well, but it seems to have vanished.

@Gapfall Sorry for the slow reply. Yes, see https://en.wikipedia.org/w/index.php?title=Template:Infobox_observatory&diff=855389083&oldid=855381712 - I've removed it again.

@Framawiki @bd808 Source/context for the issue, please? If there are only 3 servers in total here, are there plans to add more?

Thanks Tgr, that's good to know, and has answered my question. It sounds like a no-go, so I think this ticket can be closed as resolved.

In T249565#6037147, @DannyS712 wrote:

In T249565#6037136, @Bugreporter wrote:

Is it the time to unblock EmausBot?

I don't think so, at least for now; better to err on the side of caution

I think Roy was talking about enwp:
https://en.wikipedia.org/w/index.php?title=Special:Log&page=User%3AEmausBot&type=block

Anyone? There are quite a few commons links that aren't being displayed because the logic here hasn't been updated.

Hmm, Quarry is still not completing the query, and now paws has stopped working. :-(

In T248457#6004246, @Alsee wrote:

In T248457#6003779, @Mike_Peel wrote:

I really hope this never happens

There was repeated consensus on this, and if you recall you participated in one. The Foundation handled things badly, but committed to terminating wikidata descriptions when we reached 2 million local descriptions. I expect the result would be Bad all around if the Foundation were to renege.

In T184000#6004316, @TheDJ wrote:

I think we are forgetting that both systems have divergent copyright licenses, with WikiData the more liberal one. As such a bot automatically copy pasting would likely violate copyright licensing.

In T184000#6003788, @DannyS712 wrote:

In T184000#6003787, @Mike_Peel wrote:

In T184000#6003783, @DannyS712 wrote:

What about a bot importing the local descriptions from enwiki to wikidata? That way enwiki still has control over the content, but wikidata still has accurate and updated descriptions (that are then provided to commons, etc)

I can write a bot that will do that. It would still suck, as it's duplicating descriptions rather than having a single place that the descriptions are updated at, and I expect that it would be opposed for that reason. But if you would support it, then I can write it up and propose it.

It would suck, but its better than two different and diverging sources. We would need a way to tell the bot to skip certain items though...

It still means that we maintain two separate databases containing what should be the same information. Telling a bot to skip certain items is a key problem, but not the only one, we still have to figure out how to automatically resolve arguments that arise between an editor on enwp and an editor on wikidata, or have some sort of mediation page.

In T184000#6003783, @DannyS712 wrote:

What about a bot importing the local descriptions from enwiki to wikidata? That way enwiki still has control over the content, but wikidata still has accurate and updated descriptions (that are then provided to commons, etc)

I really hope this never happens, and that the locally-defined descriptions are merged back to Wikidata. There are 2.8+ million infoboxes on Commons that are using the English descriptions from Wikidata, and there is currently no way that they can access the enwp descriptions.

In T184000#5993141, @RexxS wrote:

Just a "heads-up" that a search on enwiki articles for hastemplate:"short description" yields 2,083,124 results.

May we assume that we will be moving on to Stage 2 in the near future?

I sort of have my query running on paws now (at https://paws.wmflabs.org/paws/user/Mike_Peel/notebooks/Query%20for%20Wikidata%20Infobox.ipynb if you can access that) - it fetches 100 results at a time, with an offset for the next query. I'm running through those results to add the infobox on commons, but it's a painful process (and it's probably costing more server time than usual). There must be a better way of doing this.

In T246970#5978191, @Marostegui wrote:

Not sure if you were looking at the right side, the host involved here is labsdb1011, which has a high CPU usage

I expect that there would be few requests. Phab would also work.

In T246970#5977568, @zhuyifei1999 wrote:

@zhuyifei1999 Perhaps there could be some sort of a trusted user set on quarry that can run things for longer?

How do you want such a list to be made? I obviously can't made it like RFA sort of thing, since quarry has no admin web interface.

Looking at CPU usage at https://grafana.wikimedia.org/d/000000607/cluster-overview?orgId=1&from=now-30d&to=now I can't see anything obvious that would explain things running longer. An exception might be an-launcher1001, which shows repeated CPU usage spikes since about the same start time.

@Lydia_Pintscher Is this within WMDE's scope, or WMF? It would be nice to have this actioned at some point.

In T246970#5952464, @Mike_Peel wrote:

I'm now getting the normal 'killed' message for going over 30 minutes, rather than the MySQL error. So perhaps things are now back to normal?

I get a different problem when trying to reproduce this, which may be related - typing "en" does not return English Wikipedia, just 'enwikibooks', see screenshot. Same for 'de', 'nl', etc.

I'm now getting the normal 'killed' message for going over 30 minutes, rather than the MySQL error. So perhaps things are now back to normal?

In T246970#5950773, @Marostegui wrote:

In T246970#5950772, @Mike_Peel wrote:

It is likely that the query is running for more than 4h then. I am executing it on an idle labs replica under root user, to see how long it takes to actually finish on a non loaded host.

It's a query I run regularly, it normally takes between 15 and 30 minutes depending on the server load - and sometimes longer if the servers are busy, but then it normally just gets killed by quarry's 30 minute limit.

Ah, I didn't know there was a limit on quarry itself. Then you've got your answer :-)
The query times will probably depend on how loaded the host is, so it can be different each time.
I am running it on a non loaded host, so we can see how long it takes there for real.

It is likely that the query is running for more than 4h then. I am executing it on an idle labs replica under root user, to see how long it takes to actually finish on a non loaded host.

In T246970#5949573, @Stashbot wrote:

Mentioned in SAL (#wikimedia-cloud) [2020-03-06T19:32:03Z] <zhuyifei1999_> changed to analytics replica for database queries and restarted celery workers T246970

Also under discussion at https://www.mediawiki.org/w/index.php?title=Topic:Vhw07swro9jqy4w0

In T244529#5860081, @Tarrow wrote:

We're rolling back to wmf.16 on all wikis. We hope this will resolve the issues but of course some bad data may now be cached. We'll look at this again on Monday.

This is also showing up on Commons in the infobox and other templates that use Wikidata, see:
https://commons.wikimedia.org/wiki/Template_talk:Wikidata_Infobox#Family_relations_show_up_as_Q_values
https://commons.wikimedia.org/wiki/Commons:Village_pump#Familytree_template_and_Wikidata_info
+1 to 'unbreak now' please!

If you search for ticket numbers (e.g., 'T243479'), then you can reach the corresponding ticket - but no free text searching it seems.

OK, I've tentatively closed this, if you start getting the notifications again then please reopen it!

OK, two test edits:

In T243130#5818856, @Dudley_Miles wrote:

They are all Commons categories so far I remember. Examples are Category:Sole Common Pond, Category:Westfield Farm Chalk Bank, Category:Easton Farm Meadow, Category:Boxford Water Meadows.

@Dudley_Miles Could you give some examples of articles/categories where you received notifications from me, please? One thought is that I am using a Python script to add these links rather than the editing interface, and in some cases I add both the enwp and commons link simultaneously to a new item linking the two. Perhaps that somehow means that the option blocks one notification but somehow lets the other through.

This is now live in the main version of the infobox.

Thank you, @Addshore, for investigating this issue!

Thanks! There's a possible new case from the last few days at https://commons.wikimedia.org/wiki/Category:Henrik_Harpestrengs_Vej / https://www.wikidata.org/wiki/Q81792970

If null edits work, and this isn't a problem where new cases will be created in the future, then I could run a bot through them to make null edits - is there is a list of all affected pages?

Added to {{Wikidata Infobox/sandbox}} for testing.

I haven't seen any live examples of this recently, so I'm marking this as resolved.

In T235811#5657800, @egardner wrote:

The widgets in the WikibaseMediaInfo extension use templates and render() methods to ensure that the visible UI remains in sync with the structured data as it changes. Existing DOM elements are often thrown out and re-rendered from scratch to ensure consistency. Our introduction of this workflow into the Captions widget is probably what initially broke this gadget.

I'll have a look around for some current examples, but I haven't seen any for the last week, so this may well have been solved with the deadlocks issue. Although T233520 (page_props missing links) is definitely still happening, and may be related to clearing out any old cases here as well.

This hack generally doesn't seem to work now. I think something has changed in the SDC code that means that the hook no longer works. I'm not sure if that can be fixed within the gadget, or if it requires a tweak to the SDC HTML. Perhaps @Jdforrester-WMF or @Abit could have a look?

I can currently normally find live examples of this by looking through the English Wikipedia for commons category links missing from Wikidata - so if you need live examples when debugging this, let me know and I can provide some.