Create cron tasks for archiving aggregates
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	Scardenasmolinar
	Jun 11 2025, 11:17 PM

Description

There are 3 management commands created for T370980: Export aggregates to static files that archive aggregates. This is a further effort to keep the database on the small side and to improve performance in the UI. Once the two pending PRs of the mentioned ticket are merged, we should add the following management commands to the production cron tab:

'archive_link_aggregates' command
'archive_user_aggregates' command
'archive_pageproject_aggregates' command

These commands should be run on the 10th of every month.

Details

Other Assignee: Scardenasmolinar

Related Objects

Mentioned In: T399763: Create banners to indicate wrong data on Wikilink
T269844: Improve coverage of tests, particularly for views
T370980: Export aggregates to static files
Mentioned Here: T399746: Request to increase Object Storage capacity - Wikilink project
T370980: Export aggregates to static files

Event Timeline

Scardenasmolinar created this task.Jun 11 2025, 11:17 PM

Scardenasmolinar updated the task description. (Show Details)

jsn.sherman moved this task from Inbox to To be estimated on the Moderator-Tools-Team board.Jun 12 2025, 5:19 PM

Kgraessle mentioned this in T370980: Export aggregates to static files.Jun 17 2025, 4:16 PM

Kgraessle triaged this task as Medium priority.Jun 17 2025, 4:39 PM

Kgraessle moved this task from To be estimated to Kanban on the Moderator-Tools-Team board.

Kgraessle edited projects, added Moderator-Tools-Team (Kanban); removed Moderator-Tools-Team.

Kgraessle moved this task from Ready to Ready (Maintenance) on the Moderator-Tools-Team (Kanban) board.

Kgraessle set the point value for this task to 3.

Kgraessle mentioned this in T269844: Improve coverage of tests, particularly for views.Jun 24 2025, 8:09 PM

We also need to run the new fill commands as well so the program totals continue to render properly. How far back do we want to go for archiving aggregates? Would aggregates older than a year work?

Amdrel changed the task status from Open to In Progress.Jun 24 2025, 11:13 PM

Amdrel claimed this task.

Amdrel moved this task from Ready (Maintenance) to Eng review on the Moderator-Tools-Team (Kanban) board.

@Amdrel I'm looking through the order of operations for reviewing your PRs and just wanted to make sure I was reviewing them in the order you intended:

Is this the correct order?

Kgraessle moved this task from Eng review to Reviewed (waiting for changes) on the Moderator-Tools-Team (Kanban) board.Jun 25 2025, 2:23 PM

@Amdrel https://phabricator.wikimedia.org/p/Amdrel/ I'm looking through
the order of operations for reviewing your PRs and just wanted to make sure
I was reviewing them in the order you intended:

Is this the correct order?

https://github.com/WikipediaLibrary/externallinks/pull/429

https://github.com/WikipediaLibrary/externallinks/pull/431

https://github.com/WikipediaLibrary/externallinks/pull/440

https://github.com/WikipediaLibrary/externallinks/pull/439

Yes, that is correct!

In T396681#10947700, @Amdrel wrote:

@Amdrel https://phabricator.wikimedia.org/p/Amdrel/ I'm looking through
the order of operations for reviewing your PRs and just wanted to make sure
I was reviewing them in the order you intended:

Is this the correct order?

https://github.com/WikipediaLibrary/externallinks/pull/429

https://github.com/WikipediaLibrary/externallinks/pull/431

https://github.com/WikipediaLibrary/externallinks/pull/440

https://github.com/WikipediaLibrary/externallinks/pull/439

Yes, that is correct!

@Amdrel Ok perfect!

I went ahead and merged #429 yesterday.
It looks like #431 will depend on having the aggregates populated in object storage, but we don't actually start doing that until #440 is merged in.
My suggestion is to pull out the cron tab for the aggregates from #440 and create a separate PR for adding the cron commands for filling the top aggregates later.