Page MenuHomePhabricator
Create Task
Maniphest T392247

Setup object storage for wikilink horizon project
Closed, ResolvedPublic
Actions

Description

Set up a Wikimedia developer service account
by following the steps outlined here:
https://wikitech.wikimedia.org/wiki/Help:Create_a_Wikimedia_developer_account

  • Create a Wikimedia global account
  • Create a Wikimedia developer account

(Use a username other than wikilink as that is taken)

Add the service developer account to the Wikilink project,
create application credentials for the service account as described here:
https://wikitech.wikimedia.org/wiki/Help:Using_OpenStack_APIs#Application_Credentials
These credentials are what wikilink will use to access the Swift object store -- add the credential details to the Wikipedia library password vault

Request a quota increase for object storage space if needed.
By default, any project may store up to 4096 different files and use a total of 8GB of space. These files may be distributed over any number of containers.
If we require more storage space, open a quota request ticket:
https://phabricator.wikimedia.org/project/view/2880/

test object storage

  • Backup wikilink
  • Add application credentials to wikilink
  • Manually archive some link events and verify that they upload correctly
  • If there is data loss, restore the backup

Related Objects

Event Timeline

jsn.sherman created this task.Apr 17 2025, 4:59 PM
Restricted Application added a subscriber: Sadads. · View Herald TranscriptApr 17 2025, 4:59 PM
• Scardenasmolinar moved this task from Ready to In Progress on the Moderator-Tools-Team (Kanban) board.Apr 17 2025, 5:06 PM
• Scardenasmolinar changed the task status from Open to In Progress.Apr 17 2025, 5:32 PM
• Scardenasmolinar triaged this task as Medium priority.
• Scardenasmolinar updated the task description. (Show Details)Apr 18 2025, 4:15 AM
jsn.sherman mentioned this in T386477: migrate previously archived events to object storage.Apr 23 2025, 5:20 PM
• Kgraessle mentioned this in T386475: store newly archived events in object storage.Apr 23 2025, 5:20 PM
Groceryheist subscribed.Apr 23 2025, 5:24 PM
• Scardenasmolinar moved this task from In Progress to Eng review on the Moderator-Tools-Team (Kanban) board.Apr 24 2025, 2:11 AM

I can confirm that link events have been successfully uploaded to the Object Store. The data looks good upon downloading the files. I'm moving this on to review.

Screenshot 2025-04-23 at 20.09.05.png (1,473×384 px, 61 KB)

• Kgraessle subscribed.Apr 24 2025, 1:26 PM

@Scardenasmolinar I just took a look and I see the files in the Object Store, this looks good!

Just for clarification- are the credentials uploaded on the server or did you run the command manually to populate the files? Will the next linkevents_archive dump out to Object Store?

jsn.sherman closed this task as Resolved.Apr 24 2025, 5:06 PM
jsn.sherman moved this task from Eng review to Done on the Moderator-Tools-Team (Kanban) board.
In T392247#10764815, @Kgraessle wrote:

@Scardenasmolinar I just took a look and I see the files in the Object Store, this looks good!

Just for clarification- are the credentials uploaded on the server or did you run the command manually to populate the files? Will the next linkevents_archive dump out to Object Store?

@Kgraessle yes, the credentials are in the .env file

Thanks @Scardenasmolinar

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits