Cool! I like the new form. Apart from the encoding part:

If you find some time, could you please prioritize preparing and synching https://gitlab.wikimedia.org/toolforge-repos/request-review_bot ? There seems to be an issue with this part of the bot lately (see https://de.wikipedia.org/wiki/Benutzer_Diskussion:FNBot#keine_Aktualisierung_der_Sichtertabellen ), and I'd like to have a closer look.

The immediate issue is resolved: I updated the renderer so all pad titles from the current backup are now both displayed and linked correctly. As expected, I didn't need to rename any files. It was just a matter of rendering them correctly in index.html. Once you're sure that object storage can handle all characters including "*", "?", "\", "/", "%", ">" ,... and your python frontend handles them correctly, too, feel free to close this task.

Thanks for improving the handling of the external ressources. Regarding the size of the json list: This was a quick solution without optimization in mind. If the size becomes a problem in future, we can easily squeeze it in half by switching from JSON objects to JSON arrays (see https://datatables.net/manual/data/#Data-source-types) and reducing the date precision:

Done: I created a pad_data.json from the output of the directory listing and an index.html file to present the data in a searchable table, and put both in the public_html folder, accessible via https://etherpad-backup.toolforge.org/ .

I posted the list of links here, to keep this thread compact: https://meta.wikimedia.org/wiki/User:Sj/Etherpad_links

Oh, and in case you need it for whatever reason, I just officially made you co-maintainer of etherpad-backup. But there's really nothing in there other than the static HTML files.

Sounds good. Just as a heads up: If you plan to copy / mirror all NFS content to S3, you'll quickly hit the storage limit of 4096 files mentioned in https://wikitech.wikimedia.org/wiki/Help:Object_storage_user_guide#Quotas_and_other_limitations and might need to open a quota request ticket.

FYI: I didn't comment on the VPS project request, but I'm aware of it and already successfully logged on to the Horizon dashboard to check my access to etherpads3 - which works as expected. I've no expertise whatsoever with object storage, but I'm willing to learn, so let me know if there's anything I can do to support you.

@Novem_Linguae : It would probably be technically possible, but at the same time quite risky: I know users who consider Etherpads with a long enough random title as completely "private" (though they never really are), so your approach could accidentally unveil *a lot* of personal data, passwords and other unexpected content.

If keeping a read-only version at least for some time is not an option, can you *please* at least put a warning on the front page (https://etherpad.wikimedia.org/) and ideally also in the header of the editor page that everything will be deleted in 6 weeks? I see people still creating new pads and updating existing pads which were already backed-up (e.g. live vs. backup), making it close to impossible to keep track of what is saved already and what not.

But I think it is at least better than status quo.

Done: All pads mentioned / listed above (in total: 8337) are backed up on Toolforge: P89822

Ok, so I backed up all lists added by @Pppery yesterday, and just to be on the safe side, ran the "find links within pads" script again on the recently added pads. Another 125 popped up. 🙄 Working on those now.

Finally: After ~500 in round two and ~200 in round three, there're no more links from backed-up pads to other pads not saved before, so no other round is needed.

Quick update: ~1000 new pads added to the backup which were only linked from other pads and not included in previous lists. Currently in the second round (adding pads which were linked from pads which were linked from other pads). Identified ~700, but not sure yet how many of those actually exist, and which point to empty pads. Checking and backing them up right now.

Finding other etherpads linked in backed-up ones looks doable.

In T415237#11629990, @Pppery wrote:

In T415237#11617760, @MGChecker wrote:

If I see it correctly, we did not try to do a global search across Phabricator yet, is this correct? The standard search seems to be a bit lacking for that purpose, but I don't know whether there is something better. It certainly finds stuff.

Etherpads found on Phabricator

P88888 (notice the nice round number ...)

This was generated using Maniphest search rather than global search since the latter has no API, but I don't think there's anything interesting other than tasks on this instance.

That means if you need to run an event, you can easily check whether it'll fall under the purge time or not.
At any given point in time, there might be corruption of data or other issues that force us to remove pads or everything altogether.

Reminder to myself: After implementing the change, I also have to update all links to the old subpages, including Babel templates like https://de.wikipedia.org/wiki/Benutzer:Hannes_R%C3%B6st/Vorlage/Sichter/Tabelle .

All the pads mentioned by @Pppery above have now been exported as HTML to /mnt/nfs/labstore-secondary-tools-project/etherpad-backup/public_html/p/

Summary of the lists compiled by @Pppery above and the current status:

More random thoughts I had today:

In T415237#11613170, @Pppery wrote:

All etherpad.wikimedia.org URLs linked on-wiki

(...)

Generated using CloudElastic with minor manual cleanups:

There are a bunch of pages using templates that dynamically generate etherpad URLs, which I still need to process.

In T415237#11613667, @Bugreporter wrote:

Should we require user to login to Wikimedia wikis before modifying etherpad? This will reduce the risk for abusing etherpad for uses unrelated to Wikimedia (including spam).

The legacy instance could be deleted as planned by end of April. But without the second instance, where should pads be created in the next 10 weeks?!

All pads will be permanently deleted after 30 April, 2026.

In T417207#11609478, @bd808 wrote:

I pinged @Tkarcher's Wikitech talk page to see if there is reasonable overlap with his plans for https://toolsadmin.wikimedia.org/tools/id/etherpad-backup.

That's exactly what I did yesterday for the 927 pads linked from the German Wikipedia: All of them are now archived as text files in /mnt/nfs/labstore-secondary-tools-project/etherpad-backup/public_html/p/ , publicly available via https://etherpad-backup.toolforge.org/p/<title> (e.g. https://etherpad-backup.toolforge.org/p/Tech_on_Tour_Berlin ). But that was a one-time effort, and not a long-term solution for archiving future pads across all wikis.

The more I learn about this project, the less I want to adopt it. 🫣

not sure I’ll be able to get to it this weekend

it would seem reasonable to track it as a separate thing

@LucasWerkmeister : I still hesitate to complete the adoption, as most parts of the bots still run smoothly as of today and would immediately stop once you delete the confidential data, putting pressure on me to get FNBot2 up and running asap. But as the list of small bugs and new feature requests is growing: Would it be possible to share the code (without the confidental data) with me prior to the adoption, so I can have a look already and prepare the necessary changes before the actual adoption? (I'm aware that large parts are visible already, but some folders like /mnt/nfs/labstore-secondary-tools-project/request/FNBot/review_bot/ are completely hidden)

Possible alternative approach:

Pinging @M-J (to keep you in the loop on the planned changes)

I don't know why I didn't mention this when I subscribed to this task a couple of month ago, but now that it popped up again: I implemented a reminder bot for exactly this purpose in dewiki three years ago already[1], and we're currently discussing a rollout to other languages.[2]

Fixed.

Fixed in production.

Fixed in production.

For such a small chart, it would probably be best to have a cached static / non-interactive preview picture instead of rendering the full interactive chart live. This could possibly also be useful for T394262 (Treat Charts as media with previews, not as pages, in Commons category pages), which is currently more or less blocked due to the performance concerns related to rendering dozens or even hundreds of charts simultaneously. If we had a cached thumbnail picture of every chart for galleries and categories, accessible via parameter (like "mini"), this could reduce both rendering time and client side script executions:

I also like the toolbox idea, but would additionally recommend to include all charts and tabular data files used in a given article in the respective info page (e.g. https://en.wikipedia.org/w/index.php?title=Ammonia&action=info ), where we already list Templates, Modules and Wikidata items used in the article. (Thanks to @Kallichore for bringing up this idea in https://de.wikipedia.org/wiki/Hilfe_Diskussion:Chart#Seiteninformationen )

Allowing smaller sizes (not necessarily arbitrary, custom sizes - a fixed "thumbnail / preview" size with the option to expand it to fullscreen would be enough) would not only help with infoboxes or the example mentioned above, but also in cases where the chart is not meant to be studied in detail, but only included for illustrative purposes, like most charts in https://de.wikipedia.org/wiki/Stochastische_Analysis : Replacing those with the new charts in fullscreen mode would drastically reduce the readability of the article.

You can set format to auto to correct this

Doing this in the Ammonia production chart only adds a thousand separator to the numbers, making the overlap worse than before.

The axis title on the y-axis is also optional

Thanks! Didn't try this before, but yes, leaving it out completely is the best solution/workaround for the specific case of the Ammonia production chart.

I also stumbled across this problem today. Do you still plan to restore the old values? And (more out of curiosity): Why do I still see one entry from 2024 in this list when everything was overwritten in February? (Hm. This article was moved on the day of the migration. A race condition maybe?)

Perfect - thanks! (And also thanks for reviewing the open changes in the development branch and bringing them to production!)

Sorry for necro-posting, but I don't want to open a new task for this quick adoption-related question: I also requested access to https://gitlab.wikimedia.org/toolforge-repos/wosretbot a couple of days ago. Am I even supposed to get access to this repository as per the adoption policy? (Would be a convenient alternative to creating a duplicate "wosretbot2" repo) And if yes: Is anyone actively monitoring these requests in Gitlab, or should I use another channel to request access?

Thanks for following up on this, and I agree with creating a new bot for this as well. But as I just adopted another tool yesterday (T392781) which I'm analysing / repairing right now, I'd rather have this adoption being postponed for another couple of days until WosresBot works smoothly again so I can concentrate on restoring this one. I'll let you know (and reopen this task) once this is done.

In T392781#10773574, @LucasWerkmeister wrote:

Should be done, I think – please check :)

I see two pieces of secret information so far: [...] wosretbot/nonpublic.toml, with user name(s) to exclude from certain notifications? I don’t really understand this feature; @Tkarcher do you know what it does?

@LucasWerkmeister: I agree with setting up a new bot, but as Wosretbot is still active on a daily basis (only certain parts are buggy), I'd rather set up the new bot and password before the adoption and share the bot password with you via a secure channel, so you can replace the file instead of deleting it and ensure a smooth transition.

Glad you're OK! 😊 Take your time. I'm sure there will be plenty of work left (and/or to fix 😬) when you come back.

Bad news, I'm afraid. 😕 M-J was temporarily blocked for a couple of weeks in February in de-wiki and did not return after that, and also didn't react to my message on the talk page last week. I have a rough understanding of the plan outlined in the roadmap above and think I can implement it, but didn't want to start working on it yet as I was still hoping M-J would return and pick it up again - which probably will not happen anytime soon. So I'll start looking deeper into it in the next days.

Any update? Or is there anything I can do to expedite the process?

In T389540#10659244, @bd808 wrote:

Problem 0 here is that I can't find anything that looks remotely like a license inside of /data/project/request and the tool pre-dates the selection of a default license at tool creation time.

Merged on 2024-05-21. That's nearly a year ago. Maybe the view was there but dangling until recently?

The errors started shortly after that: Last successful run was on Jun 10th. Took us a while to notice it, and then some more time to actually start the adoption process.

I have no idea what the replacement for that is.

According to https://www.mediawiki.org/wiki/Extension:FlaggedRevs/flaggedpage_pending_table, we should probably use the flaggedpages table instead

@Mr_Tortue : Looks good, thanks! @M-J : Any update from your side already? Is there anything we can support you with?

Update available now in the development branch: https://gitlab.wikimedia.org/toolforge-repos/erinnermich/-/commit/10879f0216c0ed21c2614c6a44e085c441b4a843

Bot is renamed now:

The bot I created two weeks ago for the French Wikipedia is indeed called RappelleMoiBot (so grammatically correct), but it would be easy to globally rename it to Pense-Bôt. And I like the idea! :-) Any objections, @M-J ? Otherwise, I'd request the renaming.

Feedback from the user mentioned above: Did neither work with the primary browser (Safari) nor with a second one (not specified). But answering on my discussion page works, so the problem seems indeed to be limited to edits via Visual Editor.

Another report from the German Wikipedia:
https://de.wikipedia.org/wiki/Benutzer_Diskussion:Tkarcher#Frage_von_R2d345_(18:12,_31._Jan._2025)
(I just asked for more details)

Comment from @M-J copied over here from Gitlab:

Back to Phabricator after failed attempt to use Gitlab issues...

Moved to https://gitlab.wikimedia.org/toolforge-repos/erinnermich/-/issues/1

Sorry! I migrated Klexibot some time ago already, but forgot to close this task here. I see the bot running on Kubernetes:

I didn't even officially request access to the tool yet and already got my first task assigned. 😂 Thanks, I guess...