In T163986#6015781, @RLazarus wrote:

Reopening this -- we had some alerts today from Citoid that turned out to be latency associated with only 404s (graph). We suspect the root cause was elevated latency from example.com.
We should probably remove that last external URL from the spec and replace it with something we host -- even if the uptime still isn't 100%, at least we'll be alerting on our own issues and not on example.com's.
Marielle, can I ask you to take a look?

In T248267#6010160, @AdhamKhatean wrote:

I'm not sure what you mean by auto-generate - do you mean suggesting the template parameters to use in the map?

Maps parameter is a way of JSON mapping the template and its parameters and their relationship with another set of parameters from another tool right? So, what I was thinking about is that some parameters can by TemplateData itself instead of leaving it for the user as it will not matter for the users to fill them by themselves. actually, it will make their workflow faster.

This is already done, by another tool. The user doesn't have to fill anything out by themselves already :).

In T248267#6008966, @AdhamKhatean wrote:

So, instead of directly storing the data inserted by the user from the GUI editor, we place them in the text editor using JS and it will be saved as usual we the user clicks "Save changes"? won't this make it harder for us to auto-generate things serial numbers, DOIs, ISSNs, etc? anyway if this does not seem achievable or unnecessary using Php, we can still do it with JS.

In T248267#6000424, @AdhamKhatean wrote:

And point two is, I'm not sure how much back-end Php work is needed - I expect this to be possible to do nearly all of it as front end (although I could be wrong about that!) So overall I would say add more time for front end work and break that down more, in the proposal.

Here comes the usage of Php as a back-end that is compatible with HTML, which means it can take the data inserted by the user and parsing it to JSON using a Hook. And also deals with the API(s) used to generate data like serial number, DOI, ISSN –will appear for the user but the input field will not be editable- and push it to the user.
And the way that this should work is when the user interacts with extension to make the parameter of type “map” it runs a Hook that is registered in “TemplateDataHooks.php” to add this data to the JSON.

This part should all be front-end, IMO. There's no reason to contact the back-end for this. It's easier to parse JSON with JavaScript than PHP anyway (and in fact our PHP JSON parser has a long standing bug!)

lowering to medium due to covid related switch in priorities.

In T248267#5997619, @AdhamKhatean wrote:

@Mvolz, @Tchanders, @Mooeypoo - Hello,
I am adham Khatean, an undergraduate student from Egypt pursuing degree in physics, I am interested in participating in "support maps parameter in TemplateData editor (GUI) " as a GSoC student.
That is my proposal for the project, and I was wondering if you have any advice on improving it? Especially on the timeline part? Do you think there is any thing that I miss understood about the idea or the extension TemplateData itself?
Would you any other micro tasks that you think might make me more understanding of the idea or the extension?
Thanks in advance.

In T208305#5987134, @AdhamKhatean wrote:

Hello, I am interested in participating in the project as part of GSoC. I have installed MediaWiki on my device in addition to some other extensions including the required ones for this project and now trying to work around to really get familiar with the codebase. However, I have a problem importing Template:cite_journal

as one of these two errors always appear:

Fatal error: Maximum execution time of 120 seconds exceeded in C:\xampp\htdocs\mediawiki\extensions\Scribunto\includes\engines\LuaStandalone\LuaStandaloneEngine.php on line 533

OR

Fatal error: Maximum execution time of 120 seconds exceeded in C:\xampp\htdocs\mediawiki\includes\Title.php on line 431

although these pointed lines do not seem to contain anything related to this issue even when I tried to track there parameters and returns.
If you are familiar with this issue it would be very helpful if you could give me a hint about it or what I am doing wrong.
Thanks in advance.

In T247560#5967818, @Theklan wrote:

Oopps! The user is me, so I get the notification. Thanks, I added the templatedata because I wasn't receiving it in the manual section. It may be a cache issue then, because now everything is working perfectly.
Thanks and take care!

A user added template data to the main template page, when there was already template data on the "dok" page. This overwrote the template data in the "dok" page, so consequently citoid wasn't fetching the correct configuration file and couldn't use the template at all.

In T247590#5966742, @Pchelolo wrote:

Request id seems to be only logged once here, on receiving the request:

Actually, it's not. As you can see here, the req.logger is a child logger that includes the request_id by default, so the intention is that all the logs made with req.logger will include the request_id field, as well as the 'request' field.
Example - log from mobile apps:

[2020-03-13T08:29:42.777Z]  INFO: mobileapps/391 on scb1002: Page or revision not found. (message="Page or revision not found.", status=404, detail="Page or revision not found.", request_id=1d999c24-091c-4a83-bdf6-87e2852b3e82, levelPath=info/404)
    type: https://mediawiki.org/wiki/HyperSwitch/errors/not_found
    --
    request: {
      "url": "/en.wikipedia.org/v1/page/summary/SHFNGH89M16Z336X",
      "headers": {
        "x-request-id": "1d999c24-091c-4a83-bdf6-87e2852b3e82",
        "content-length": "0"
      },
      "method": "GET",
      "params": {
        "0": "/en.wikipedia.org/v1/page/summary/SHFNGH89M16Z336X"
      },
      "query": {},
      "remoteAddress": "10.64.0.100",
      "remotePort": 34296
    }

The key is to use req.logger and not app.logger in your handlers. I can see that cited doesn't follow this rule - most of the handlers store a logger in the constructor and fetch it from app.logger. Making app.logger include details about the request is impossible - app is a global object and is shared between potentially multiple requests, so if we reassign the app.logger, the logs will all show the request_id of the latest received request, and not necessarily the request that a specific log belongs to.

In T214038#5362155, @JTannerWMF wrote:

Hey @Mvolz we probably won't work on this feature this year, but feel free to ping me if you think that is a mistake or you'd like to discuss that more.

In T243444#5947944, @akosiaris wrote:

I guess we aren't gonna find the source of this specific event. @Mvolz do you feel we are at least better prepared logging wise for the next time this happens?

In T245934#5910127, @Nemo_bis wrote:

This is not a bug. The VisualEditor "cite" dialog (which has nothing to do with the Cite extension) just calls Citoid, which relies on CrossRef data, which is just what publishers provide. In this case, the publisher has uppercase titles:
https://api.crossref.org/works/10.1097/00007890-197506000-00010
Citoid can and should avoid introducing its own junk, but it's not within its scope to clean up the garbage provided by publishers.

In T244953#5892050, @matthiasmullie wrote:

In T208213#5880462, @Samwalton9 wrote:

Are all aspects of this task now resolved? If not, what's outstanding?

I've opened a new translator request to Zotero for this one: https://github.com/zotero/translators/issues/2118

Reported upstream to Zotero as well here: https://github.com/zotero/translators/issues/2117

Reported two places:
https://github.com/zotero/translation-server/issues/111
https://github.com/zotero/translators/issues/2117

Change 566747 now deployed, except the information contained in the error is totally useless. 🤦 Example of uselessness: https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2020.02.11/citoid?id=AXA0MBAdh3Uj6x1zvZsc&_g=h@9a6393e

@Tchanders any ideas for good microtasks for this project?

In T208305#5843856, @Prondubuisi wrote:

After going through https://www.mediawiki.org/wiki/Content_handlers page, I was able to enable missing extensions and import the template page Successfully

However I see no content when I navigate to the imported page

Is there any other step to see page content

In T208305#5839990, @Prondubuisi wrote:

Hello @Mvolz thanks for the additional description! I am interested in working on this project,
Please when you say add some templates to your mediawiki extension, how do you mean?
Is there a Separate extension called Mediawiki? or are you referring to the TemplateData Extension?

In T243444#5829834, @akosiaris wrote:

In T243444#5829606, @Mvolz wrote:

In T243444#5826816, @akosiaris wrote:

Thanks. That's working now, but I've downloaded the log file and it's just what's already available on kibana, warn level or higher. There's no debug level or message (10/20) in the logs - I don't suppose we have those anywhere?

Nope. What's in there is what citoid is instructed to log. The config is at https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/master/charts/citoid/templates/config.yaml#16

Ah - well unfortunately without the trace and debug messages, and the missing err_body_internalURI in the warning log entries of interest, the logs seem largely useless in terms of figuring out which url was the culprit :/
I've put in a pr to bump up zotero misses up to warn, what do you think about logging trace and debug as well in the meantime? Too overkill?

info would be acceptable I guess. trace/debug might be an overkill though, especially given that citoid seems to log the body of the entire page (why does it do that again?). Also, we don't know when the next event will occur (it's months since we last got paged for zotero memory issues).

err_body_internalURI should ideally be in every single log message too, I guess I'll try to figure out why that isn't happening.

+1

In T243444#5826816, @akosiaris wrote:

Thanks. That's working now, but I've downloaded the log file and it's just what's already available on kibana, warn level or higher. There's no debug level or message (10/20) in the logs - I don't suppose we have those anywhere?

Nope. What's in there is what citoid is instructed to log. The config is at https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/master/charts/citoid/templates/config.yaml#16

(Note that this is not due to any change in the software, but rather that lots of websites seem to be using short urls in their metadata than ever before. So any change you're seeing is rather more distributed! )

For this specific case, the url seems to be coming from

<meta property="og:url" content="https://www.faz.net/1.6469341" />

It would probably be better to use <link rel="canonical" href="https://www.faz.net/aktuell/gesellschaft/menschen/verleumdungsklage-von-e-jean-carroll-gegen-trump-16469341.html"/>, in theory. But this is coming from Zotero, so we can't really change how they pick which metadata to use, except possibly to file a bug. One possible solution is the one I've closed this as a duplicate of.

In T243444#5826662, @akosiaris wrote:

-l app=citoid as that's the value for the app label, not citoid-production.
You can have a look at all the labels that are attached to pods with either https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/master/charts/citoid/templates/deployment.yaml#17 or just by doing kubectl describe deployment and look at the pod template/labels stanza.
What it did ask for is the name of the container since there are 3 containers in the pod

1. citoid-production (what you want)
2. production-metrics-exporter (the prometheus exporter for the grafana dashboards)
3. citoid-production-tls-proxy (the container that makes sure TLS is terminated before the app)

The exact call would be
kubectl logs -l app=citoid -c citoid-production

In T243444#5825779, @akosiaris wrote:

it should be in the raw logs

Not sure what this refers to. If it is zotero logs, note that those don't exist. We had to turn them off as they were in a really bad format that caused issues to our logging pipeline. Assuming https://github.com/zotero/translation-server/issues/2 is addressed, we could reenable them. The disabling was done back in 827891af3d49beefc4a32042ff0ee2907dc77f0b
If it's the citoid logs, running kubectl logs -l app=citoid after having sourced .hfenv from the correct cluster directory works. But they are going to be way more difficult to parse of course.

In T243444#5825779, @akosiaris wrote:

it should be in the raw logs

Not sure what this refers to. If it is zotero logs, note that those don't exist. We had to turn them off as they were in a really bad format that caused issues to our logging pipeline. Assuming https://github.com/zotero/translation-server/issues/2 is addressed, we could reenable them. The disabling was done back in 827891af3d49beefc4a32042ff0ee2907dc77f0b
If it's the citoid logs, running kubectl logs -l app=citoid after having sourced .hfenv from the correct cluster directory works. But they are going to be way more difficult to parse of course.
But I do see in logstash a err_body_internalURI attribute in some entries that seems to indeed have what we need?

Unfortunately I'm still not seeing the ability to +2 :/