Page MenuHomePhabricator
Create Task
Maniphest T356195

Request creation of topic-curator VPS project
Closed, InvalidPublic
Actions

Description

Project Name: topic-curator

Developer account usernames of requestors: so9q

Purpose: Curate subgraphs (e.g. the 85 mio scientific articles currently missing one) and add P921 main subject in batches based on matching.

Brief description: This is a simple python flask application. I intend to use a database to record use of the tool. mariadb should suffice.

How soon you are hoping this can be fulfilled: as soon as possible.

I installed the tool on toolforge but because of some configuration limit it does not work as expected.
see https://github.com/dpriskorn/WikidataTopicCurator/issues/5

I have a lot of experience with installing and maintaining linux servers. I expect to be able to handle everything myself. See https://meta.wikimedia.org/wiki/User:So9q

License: AGPLv3+

Related Objects

Event Timeline

So9q created this task.Jan 30 2024, 4:49 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 30 2024, 4:49 PM
So9q updated the task description. (Show Details)Jan 30 2024, 4:52 PM
So9q updated the task description. (Show Details)
bd808 subscribed.Jan 30 2024, 11:34 PM

@So9q Moving the tool to a dedicated project is reasonable if it will actually fix something, but if your guess about URL length is correct I'm not sure how that will be different in a dedicated project vs Toolforge assuming that your dedicated project will still need to use the https front proxy for Cloud VPS projects that is just a slightly different configuration than the Toolforge front proxy. It would be nice to have a better understanding of the problem you are facing before rushing to making a dedicated project the solution.

bd808 added a comment.EditedJan 30 2024, 11:57 PM

With the changes from https://github.com/dpriskorn/WikidataTopicCurator/commit/d532547c74c4ca156d712299f580bc72e50f645a now in place on your Toolforge tool I can't use something like https://topic-curator.toolforge.org/articles?qid=Q1949144&limit=100&prefix=haswbstatement%3AP31%3DQ13442814+-haswbstatement%3AP921%3DQ1334131&affix=-inlabel%3Asyndrome to try and help debug the problem as the live code aborts with a deliberate failure.

If your assumption about URL truncation by the Toolforge proxy is true, I'm not sure how your tool would work from anywhere as the long URL is a 302 redirect targeting https://quickstatements.toolforge.org. Or am I thinking about this wrong and the problem is somehow in emitting the redirect header from your tool?

bd808 added a comment.Jan 31 2024, 12:51 AM

Sending all 66 items from https://bd808-test2.toolforge.org/articles?qid=Q1949144&limit=100&prefix=haswbstatement%3AP31%3DQ13442814+-haswbstatement%3AP921%3DQ1334131&affix=-inlabel%3Asyndrome to quickstatements seems to work as expected. I have that tool running a git clone pointed at https://github.com/dpriskorn/WikidataTopicCurator/commit/940ec123a3ce82eb002331ae20da0cd3c95e5a96 with the Toolforge specific abort code removed. The current HEAD of your repo fails with some validation problem.

url to qs: https://quickstatements.toolforge.org/#/v1=Q124337936%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ124328031%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ58140711%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ120979760%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ123142554%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ123137148%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ114637338%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ122229150%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ86322627%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ87167347%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ46104761%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ34649470%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ58114146%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ56390204%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ55966557%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ38425136%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ121278371%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ60604143%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ114637339%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ58273690%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ47563201%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ121914239%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ122187100%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ60604151%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ121864561%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ121394250%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ58115000%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ122190923%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ47644319%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ92655359%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ92850628%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ121362469%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ89759655%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ58280896%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ91723138%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ123684929%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ121588162%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ58279417%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ61448909%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ55966558%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ47614919%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ122923515%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ123159070%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ47570414%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ47561330%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ60604138%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ49709069%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ123130475%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ45781056%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ47608217%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ58261704%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ58114912%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ56390199%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ47289146%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ122925945%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ47564580%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ58159207%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ50207875%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ47562375%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ41073104%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ34531796%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ59704352%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ47563201%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ121432870%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ121234103%7CP921%7CQ1949144%7CS887%7CQ69652283%7C%7CQ56672229%7CP921%7CQ1949144%7CS887%7CQ69652283

That is a 3492 character URL pointing to quickstatements. Per https://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers the default buffer size should be 8k, so there is a limit that would eventually be hit but a rather large number of parameters should fit. You could squeeze in a few more even by adjusting your URL encoding. There is technically no reason to encode | as %7C when contacting quickstatements. The example 3492 character URL is reduced to 2704 characters when using | directly. The large_client_header_buffers is applied in-bound to the quickstatements tool, so as I understand your stated problem this would not be changed in any way by moving your tool outside of Toolforge.

bd808 updated the task description. (Show Details)Jan 31 2024, 12:52 AM
So9q closed this task as Resolved.Feb 2 2024, 11:56 AM
So9q claimed this task.

Big thanks for taking a deeper look into this and for the suggestion to adjust the encoding. I'll close this as resolved for now, while I try diving deeper into the issue and try to make it work.
Locally on my machine everything works fine. I can send batches as large as 5000 items to QS with no problems which translates to 10k qs-lines to edit.

So9q added a comment.EditedFeb 2 2024, 12:07 PM
In T356195#9500686, @bd808 wrote:

With the changes from https://github.com/dpriskorn/WikidataTopicCurator/commit/d532547c74c4ca156d712299f580bc72e50f645a now in place on your Toolforge tool I can't use something like https://topic-curator.toolforge.org/articles?qid=Q1949144&limit=100&prefix=haswbstatement%3AP31%3DQ13442814+-haswbstatement%3AP921%3DQ1334131&affix=-inlabel%3Asyndrome to try and help debug the problem as the live code aborts with a deliberate failure.

I changed the tool, see https://topic-curator.toolforge.org/results?lang=en&qid=Q1949144&limit=100&prefix=haswbstatement%3AP31%3DQ13442814+-haswbstatement%3AP921%3DQ1334131&affix=-inlabel%3Asyndrome which now loads.

If your assumption about URL truncation by the Toolforge proxy is true, I'm not sure how your tool would work from anywhere as the long URL is a 302 redirect targeting https://quickstatements.toolforge.org. Or am I thinking about this wrong and the problem is somehow in emitting the redirect header from your tool?

Exactly that is my theory: nginx is denying to send the request from toolforge -> QS because of the header exceeds the max allowed size.

Thus if I get to control the nginx configuration in Cloud VPS of the instance topic-curator is running on, then I can tweak until it works for large batches also.

So9q added a comment.Feb 2 2024, 12:49 PM

To see the error in action try this url: https://topic-curator.toolforge.org/results?lang=en&subgraph=&qid=Q7889&limit=&prefix=&affix=&terms=videogame&terms=computer+games&terms=video+game&terms=computer+game&terms=videogames
Select all
Click submit
->

image.png (568×787 px, 78 KB)

JJMC89 changed the task status from Resolved to Invalid.Feb 2 2024, 4:05 PM
bd808 mentioned this in T356525: Very long response headers cause a 502 response from the Toolforge front proxy.Feb 2 2024, 5:26 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL