In case it is useful for botnet detection, looking at the Grafana chart I detect that each day, at 4 PM UTC o'clock, the traffic load changes significantly, and remains stable for the next 24 hours. For example, from yesterday at 4 PM UTC to today at 4 PM UTC, requests per second where at ~26. Today from 4 PM UTC the traffic is quite stable at ~18 req/sec. This exact pattern is identifiable since at least 15 days ago. https://grafana.wmcloud.org/d/TJuKfnt4z/tool-dashboard?from=now-30d&to=now&timezone=utc&var-namespace=tool-paulina

@Danielyepezgarces Good suggestion, thanks! I implemented it with https://gitlab.wikimedia.org/toolforge-repos/paulina/-/commit/d9d52342460f24356380d2297fa741307d34afad

@Afanyulionel I'm closing this task as it was not finally implemented. Thanks anyway!

@marfossatti Done. Basically, the Wikidata graph split made it impossible to access scholarly articles with a SPARQL query. The migration from SPARQL to the Action API method made the trick: https://gitlab.wikimedia.org/toolforge-repos/paulina/-/commit/742dbd7f35728c6c480263cfc0b79fb9ed5a3170

@Ademola I merged your contribution. Thank you!

Thank you! I set up robots.txt as you recommended and made progress with "nofollow". I'll continue improving as much as I can.

Yesterday around 12:00 AM UTC, CPU topped and the tool failed, so I set up 6 * (2 core + 1G). If things go fine, I think I can reduce the number of cores again to 1 or 1.5. Is there a maximum number of replicas?

I don't expect this will change the bot access rates much, but I would recommend changing your robots.txt to:

ROBOTS_TXT = "User-agent: *\nDisallow: /\n"

Thank you @bd808! I just tried your suggestion:

While tinkering with Grafana, I see that the application's memory usage is actually low (around 500 MB of 6GB allocated), which I find strange: https://grafana.wmcloud.org/d/TJuKfnt4z/tool-dashboard?orgId=1&from=now-7d&to=now&timezone=utc&var-cluster=P8433460076D33992&var-namespace=tool-paulina&viewPanel=panel-2

Thank you @bd808. The big step up on August 12 probably reflects the uwsgi.ini settings I added that day, while still using the legacy python 3.13 build:

@bd808 I migrated the Paulina tool to the new build service container, and started it with --mount=none

In T432878#12210217, @bd808 wrote:
The more I look at this, I think moving to a custom build service container and disabling NFS mounts would help you quite a bit. It looks like the main change you would need to make in the code is moving your config.py configuration into environment variables. https://wikitech.wikimedia.org/wiki/Help:Toolforge/Building_container_images/My_first_Buildpack_Python_tool lays out the basics of using the build service to make a container for a Flask app like yours. You might need to figure out how to map your uwsgi.ini customizations to gunicorn config in the container too.

@BLiviero-WMF @bd808 Thanks for you answers!

Implemented from a contribution by @Ademola: https://gitlab.wikimedia.org/toolforge-repos/paulina/-/commit/0076e743255dd0f8c324f996d48ecb5bbe76c11a

The issue is gone. Probably the cause was T425758. I suspect the server couldn't finish processing requests that contained a sparql query because the sparql query kept running indefinitely. This caused new requests to accumulate and remain unresolved until capacity was exceeded.

Implemented: https://gitlab.wikimedia.org/toolforge-repos/paulina/-/commit/72115507fbe2c9e549c76114958528a10f9021d6

Some screenshots of the features we were working on:

Great work @marfossatti! I'll implement the taxonomy as with T393157.

I implemented the feature as planned, for author and work search, and it works very well. See this commits:

• https://gitlab.wikimedia.org/toolforge-repos/paulina/-/commit/d80d30dbaa2dc3eb7d50d4f88b318eec0f169d23
• https://gitlab.wikimedia.org/toolforge-repos/paulina/-/commit/ba8336517ba8432e39401bb8e5eed9cb3f77e68a

I merged @System625's solution for results pagination. Great work!

This could be a possible query, but it times out in WQS: https://qlever.dev/wikidata/M2aKAs

First iteration of tests: https://gitlab.wikimedia.org/toolforge-repos/paulina/-/commit/fb222b8c07a0cd0c21f4e7b87b0ddd50056a9995

This feature is not necessary anymore, since the new Paulina editing features allow users to create items from within Paulina. Anyway, I don't close it until we add the links to create items in the different contexts detailed in the task description.