I looked at the code in the HeaderCallback class. Questions:

1. Does the HTTP spec actually forbid headers without a colon? If this is the case, we should make violations visible.
2. Is it the job of the HeaderCallback class to validate all headers? If it is, it should properly log bad headers so we can see them. If not, my proposal in https://gerrit.wikimedia.org/r/491539 makes the class stop failing on headers it actually does not care about.

We know on what basis it is decided which type a reference has

I'm not sure I understand the idea presented in this tickets description. Is the suggestion to somehow force people to type hint all code? How would we set this up? Who would be in charge to update the code? Are we in a position to assign budget on a task as massive as updating multiple tens of thousand function headers in a thousand codebases? How do we communicate this new requirement to all the dev teams? How can we check if code conforms to this new standard? I mean, it's not like PHP 7 suddenly forbids to have function parameters that can be, for example int|string, string[]|string, or even mixed, just to name the most prominent examples that don't involve null.

I believe this was fixed via https://gerrit.wikimedia.org/r/404161 already, uploaded in January 2018, merged in October 2018.

Pinging @Raymond because we are renaming one message key in https://gerrit.wikimedia.org/r/489743. Maybe he is able to help?

Here are screenshots of how the design looks like after https://gerrit.wikimedia.org/r/488938:

For future reference: Originally this ticket mentioned "lock" icons, as described at https://en.wikipedia.org/wiki/Help:Citation_Style_1#Registration_or_subscription_required. This confused me for a long time. Very late I realized there are (to) many different ways such icons can be implemented:

1. Via [[File:Lock-….svg|9px|…]] in the wikitext. Such images are part of the content of a footnote. There is no way they can disappear, except some code intentionally removes them. Example: https://en.wikipedia.org/wiki/Template:Free_access
2. Via global styles in the MediaWiki namespace. These are typically generic enough to also be active in popups. Example: https://en.wikipedia.org/wiki/MediaWiki:Common.css
3. Via TemplateStyles. These are automatically limited to elements with the mw-parser-output class. Example: https://en.wikipedia.org/wiki/Module:Citation/CS1/styles.css

Some notes from today:

• The idea is to copy-paste the file gateway/rest.js and create a new – for example – referenceRest.js.
• The existing gateway/reference.js should stay as it is, to be used as a fallback for users of the extension that don't have a RESTbased endpoint.
• We could rename it to referenceScraper.js.
• Here is the documentation for the API currently used for the page previews: https://www.mediawiki.org/wiki/Page_Previews/API_Specification
• The RESTbased endpoint is self-documenting, including a sandbox mode. Awesome! https://en.wikipedia.org/api/rest_v1/#!/Page_content/get_page_references_title_revision

I explored a bunch of ideas to get a grip on this issue. Finding some solution isn't that hard, but I want a solution that is sustainable and guaranteed to not mess with regular page previews. It should especially not make T198652 worse.

• As far as I can tell the title parser was intentionally build as restricted as it is. It should only accept links it absolutely must accept.
• One idea I had was to create a whitelist of URL parameters the title parser is allowed to ignore. We would need a list of use-cases where we want reference previews to be shown, and a (hopefully) complete list of use-cases where we do not want page previews to be shown. I tried, but this gets somewhat bonkers and unmaintainable pretty quickly.
• I believe the gold solution is to detect self-links, as discussed in T198652.

I 100% agree. :-) Just to have this documented: the previous label was following the mocks as presented in T213415 and especially T213905:

Please note that this issue is most probably unrelated to any of the new code we introduced for Reference Previews, as it can happen on regular page previews as well:

@WMDE-Fisch, I can confirm this observation.

As far as I understand what you refer to as being "fixed" now was never broken.

The best idea I have so far is the ABANDON_END_DELAY, that defaults to 300 ms, and is used in a single place in the code: https://phabricator.wikimedia.org/diffusion/EPOP/browse/master/src/actions.js$266. I tried to set this to 1, and the issue seems to disappear.

Hm. Generally, the monkey testing page is not the best to do such tests. It contains extreme edge-cases of unknown relevance for real-world wiki pages. We never specified an expected behavior for most of these edge-cases. If a report refers to an example on the monkey testing page, it is important to let us know which one was used.

I was under the impression the font size of this link is derived from the existing styling of page previews. For example:

For your information: We realized that that a transfer rule containing subst:Изображение would trigger a bug in FileImporter. The template name after the colon would get lost and the resulting wikitext would contain something like {{subst|…}}. The bugfix https://gerrit.wikimedia.org/r/485761 should go live on February 5th.

Thanks a lot, that sounds good.

Is there a chance this gets fixed fast? Otherwise we will temporarily disable the test. This should buy you enough time. The only thing you need to take care of then is to not forget to enable it again.

Oh. I suggest to lift the limitation from this and the other ticket T214754, as both are not exploitable on any production server.

This might be related to T214710#4914267, or might become much less of a problem after T214710 is solved. Let's make sure the fix is merged before we test this again.

This happens when the name of a reference contains a : character. jQuery requires that almost every special character must be escaped to be able to use it as a plain text character in an ID or class name. See http://api.jquery.com/category/selectors/. We did not anticipated this. I guess we unintentionally limited our tests so far to the characters set used for identifiers in programming languages. ;-)

What's wrong with this bot?

Yes, disambiguation pages are handled by the regular page preview code. They are just presented a little different (with the icon and a fixed message), because it would not be terribly helpful to show a text extract from a disambiguation page.

Question to PM and possibly UX: I noticed that all existing icons are gray, not black. Well, "all existing" is more like a single icon for disambiguation pages:

Is this a change in behavior from the way references work today? For me personally, I usually control or middle-click when a new tab is wanted. […]

It looks like I also get these emails. I don't understand why. They are of zero use for me. How can I turn them off?

Thanks a lot for working on this!

Quick first story grooming:

• Converting dates is an extremely hard problem. See T87764, for example. Whatever we implement will most probably be a "good enough" solution that will stop working in many situations (e.g. Chinese).
• Deleting template parameters is certainly possible. It was not part of the original CommonsHelper2 specification, that's why we have not talked about it yet.
• Wrapping text in language templates is part of the original CommonsHelper2 specification. We decided this should not be part of an MVP. A separate ticket exists: T198607.
• I think the last bit is technically already resolved via T198316. I juts went to https://www.mediawiki.org/wiki/Extension:FileImporter/Data/de.wikipedia#Licensing and added this alternative headline.

Why is suppressing the preview better than showing a possibly redundant one?