Page MenuHomePhabricator

migrate previously archived events to object storage
Closed, ResolvedPublic5 Estimated Story Points

Description

Currently, when we serialize and compress events for archiving, json.gz files are dumped to the local filesystem and then manually rotated to external storage that is not publicly available. We got the go ahead to share these archives, so we should move the existing archives to the publicly-readable object store used in T386475: store newly archived events in object storage.

Most of our existing archives use an old schema that is incompatible with our current ORM. So we should migrate those before publication.

Acceptance criteria:

  • Previous archives use current event object schema
  • Archives hosted in a publicly readable object store
  • Archives have a predictable / logical path that allows for easy bulk download based on date without needing to interrogate an api or scrape any web pages; this should be in alignment with T386475

Event Timeline

Kgraessle set the point value for this task to 5.Mar 4 2025, 4:23 PM
Kgraessle removed the point value 5 for this task.
Mimurawil changed the task status from Open to In Progress.Mar 7 2025, 6:57 PM
Mimurawil claimed this task.

Regarding migrating the old schema from old archives, the fields below will have the treatment:

  • user_is_bot -> default to 0 (False)
  • domain -> not null field, we need a default value for old archives (maybe a simple n/a should work?)
  • hash_link_event_id -> computed (same way as in the migration 0010_data_link_event_id_hash.py)
  • content_type_id and object_id -> default to null
  • content_type_id and object_id -> default to null

Correction on those two fields: they both can be computed as in 0014_migrate_url_pattern_relationships.py

Regarding migrating the old schema from old archives, the fields below will have the treatment:

  • user_is_bot -> default to 0 (False)
  • domain -> not null field, we need a default value for old archives (maybe a simple n/a should work?)
  • hash_link_event_id -> computed (same way as in the migration 0010_data_link_event_id_hash.py)
  • content_type_id and object_id -> default to null

Regarding this comment above, that is incorrect since we have all these data from the creation of this model, except for the content_type and object_id which can be computed in migration 0014.

jsn.sherman set the point value for this task to 5.Mar 25 2025, 6:38 PM

I dropped the old PR considering we're already migrating the old archives, which would make this new script unnecessary.

I added a new action to the linkevents_archive script to only upload a file to Swift so it helps in case we want to upload individual files.

The PR is the same as T386475 considering it reuses the upload function in there https://phabricator.wikimedia.org/T386477.

jsn.sherman changed the task status from Stalled to In Progress.May 14 2025, 5:18 PM
Scardenasmolinar updated the task description. (Show Details)
Scardenasmolinar moved this task from QA to Done on the Moderator-Tools-Team (Kanban) board.

I have verified in the public Object Store all of the acceptance criteria.