This place for discussions is what I was looking for. I hope that it is a very active place.

Maybe this was not the right place to present the profiler. Indeed, I know, at this stage, it's not asking for anything else than comments on the kind of profiling that can be inferred from the output. If there is no interest, I will stop here. The code is not important. It's the idea that matters at this stage.

Thank you Reedy. The profiler I created produces a call graph similar to Kcachegrind. There is an added value to see the connection between the different processes. Also, the profiler I created is way more precise in terms of where the time is spent than what I see in the daily flame graph. For example, the flame graph mentions "braceSubstitution" whereas my profiler monitors "braceSubstitution_safesubst:#invoke:Annotated link", "braceSubstitution_#invoke:citation/CS1", etc. It may have considered 20 or more different calls to braceSubstitution - I did not count. These different calls to "braceSubstitution" behave completely differently, so it is pertinent information. At this stage, the profiler needs improvement: it is necessary to group small processes to have a graph that can be presented and look nice, but it is also necessary to be able to search these groups and to have more informative labels for them.

I was looking for a way to profile mediawiki with that level of details and could not find anything. So, I created my own profiler adapted to the situation. I have, of course, seen the profiling information available in the html source, but it is not comparable. It is an invitation for mediawiki to offer that kind of profiling info to developers.

I have uploaded a patch with the associated commit message for the addition of validations to the previous design.
{F41728944}
In the proposed design, the processing of the ref tags is independent of validation: all ref tags, with errors or not, are processed and presented as ref links or/and ref items. The validation simply attaches an error message to an erroneous ref item. In this way, the error messages are always close to their erroneous ref item and the erroneous ref item contains its backlinks when they exist. This is just an extra aspect of the design. The key point is to allow nested refs in references block.

I have uploaded a patch with the associated commit message for the new proposed design. It's only proposed to inform a discussion. {F41728941}

To show that it is possible to have nested refs, even in references blocks, without modifying the core parser, I have installed on this test site a recent master branch version of the core (not patched) with a new version of the Cite extension. Again, many wikipedians use nested refs and many prefer to put ref content in references blocks and, therefore, it is sad that these two features cannot be combined. I made sure that any reference error listed in the code (under a cite_error_<type> message), if there was any, is well displayed. There are none in that page, but I created this other page on the same site to illustrate the display of all these errors. I believe a good display of editor errors is important. I do not ignore the intension to move toward parsoid/PHP, but I believe that it is important to have a good point of reference when we do such a transition.

I reassigned myself to this task. I agree with those that say that it is not a bug, but a design decision. So, I have no intention to provide a patch that will correct that "bug". What is required is an entire new branch with an improved design. Currently, many wikipedians use the in list feature, that is, they prefer to put ref tags inside a references tag : <refererences> <ref> ... </ref> ... <ref> ... </ref> </references> (or the equivalent), because they feel that putting the ref tags directly in the body of the page makes the page hard to edit. It is also very common to see pages, even featured pages, that have notes with ref tags. Internally, notes are forms of ref tags. So, nested ref tags are very common. It is sad that it is not possible to nicely use these two common features together. Therefore, I have assigned myself the task of providing a new design that will allow these two features to coexist.

In retrospective, I think it would have been better that I squash every thing in a single commit with a long explanation that covers every step of the change. My hope was that by doing a single git review for all the commits, gerrit or git review would have understood that it is a single proposal that must be presented as such into a single page with links to the patches. For that reason, I did not understand why Aklapper was concerned with dependencies. I even wrote that there was no dependencies, which is false (and erased it quickly). The point is that, if it is a single proposal, the notion of dependencies is not important, because they are ordered and if we understand the big picture, the dependencies are obvious. So obvious that we don't think of that as being important. I was very disappointed when I saw that it created many separate requests for review. Then I realized that, considering the dependencies, this was a mess. This has to be reviewed as a single proposal and the separation in many ordered patches is just for convenience.

Looks good to me. You may be interested in https://www.mediawiki.org/wiki/Gerrit/Advanced_usage#Create_a_dependency if you plan to put the patches for review into Gerrit

Yes there are dependencies and did not realize it. However, the commits in the Cite extension will do a great job even without patching the bug in the parser. The only things is that if an editor creates nested ref tags that the parser cannot support because of this bug, the editor will receive an error message. Currently, the situation is just worst: even simpler nesting of tags is not supported.

I have done all my commits locally with short subject, body, Bug # and Change Id (generated by git review). There are seven of them. I would like to push them to a topic branch without squashing as illustrated here , because squashing does not seem appropriate in this particular use case. In particular, I would like to add a cover letter to the topic. There is not enough details in these linked explanations. I would like to be sure I will do that correctly. For information, I have uploaded the log of six commits for the CIte extension. There is another commit for the parser in the core.

In T348967#9431325, @Muzilon wrote:

You may have to log in to Oxford Academic via the the Wikipedia Library page again.
https://wikipedialibrary.idm.oclc.org/login?auth=production&url=https://academic.oup.com/

This old wikipedia library url forOxford Academic still does not work, but you have another one, perhaps a new one, that does work.

This used to work before. It still does not work. What it is that works for you now that makes you think that the problem is solved.

I have divided the whole thing in 8 commits. Each one has its separated purpose and the code is running after each commit. However, they are not at all separated logically. The effect is not seen until all of them is applied. In fact, though I checked that the code is still running and seem to be ok in the case of non nested tags, it may be that each of them separately creates problems (seen after careful testing) without providing alone any advantages. They are designed to work together. Therefore, I am not sure what is the best way to present these 8 patches. The big picture might be important to understand each patch. What I described above is for a single patch, but it is kind of central. Yet, looking at the diff of this patch alone would still not do much better than what I tried to explain above. I am afraid that an understanding of the problems must come first and the patches individually will not in themselves provide that understanding. All of them together start to be complicated and require some explanation for the big picture. I would feel much more comfortable, if the expectation of people in terms of what they can understand by looking at the patches themselves is not too high. Concretely, I decided to present the 8 commits into a single series of patch sets. I have a question. Locally, I only tested the code with the top of the master branch and 5 extensions, all running properly: Cite, ParserFunctions, Scribunto, TemplateStyles and VisualEditor. I also added many templates to test {{harv}} and {{refn}}. But, I have not tested the code in a realistic environment. I tried, but did not succeed. In particular, my IDE (Netbeans) is not stable when I have all the extensions. How can I test my code, not necessarily locally, in a more realistic environment?

I would like to assign myself to this task. I spent the last two weeks working on this. I found several bugs in the code and a patch for each of them. The result is a code that works with nested <ref> tags without the need of templates or anything else. Some are not really bugs, but more choices in the design that were not compatible with the recursive power of the parser. Yet, I did not need to change a lot the code. For example, the idea of recursive parsing is that, when the extension receives a tag, it does not do much: it just do the part that it needs to do and let the parser (with the help of other calls to extensions) do the rest. The extension just focuses on a small thing and the recursive power of the parser makes the whole thing work. So, making the code truly recursive makes it simpler. Currently, the code does not use the recursive power of the parser and it's more complicated and it does not really work. In particular, in the new code, there is a call to recursiveTagParse() inside guardedRef(). Without that call, it cannot be said to use the recursive power of the parser. The current code does instead two calls to recursiveTagParse() at the end in ReferencesFormatter.php, after the recursion is terminated. It is important to respect the logic of the parser. The parser recursively transforms wikitext into half-parsed html, but this haf-parsed htm actually contains a lot of fully parsed html stored in a stripState. The stripState is conceptually an associative array that matches the full html created by an extension with a marker which is also found in the half-parsed html. I might seem complicated the way I explain it now, but it is simple.: the half-parsed html contains markers and the stripState associates these markers with full html. So, despite the fact that half-parsed html is created by the parser, the job of the extension is to return fully parsed html to be stored in the stripState. When this is understood, it is easy to see that the job of formatListItem is to directly create its own html to wrap the text, which will already be html, because of recursion. There should be no need wrap the text with wikitex and then transform the whole tag in html using calls to recursiveTagParse(), which is what the current code does or rather tries to do. It does not really work anyway, because the href urls toward the cite_ref ids must be computed dynamically during the recursion. After, at the end, the information is lost. The new approach makes use of recursion and considers that the text is html (because of recursion) and only wraps it with its own html. This means that there is no need to call recursiveTagParse() again in FormatReferences() like the current code does. The new code gets rid of these two not really recursive calls at the end and replaces it by a single truly recursive call in guardedRef(). This does not require a lot of change in the code, but it is big change in the perspective: it is much cleaner. There are other aspects of a similar kind to consider. This is just an example.