Page MenuHomePhabricator

request: s3 access for wdqs::alternatives
Closed, ResolvedPublic

Description

For temporary early testing of our data reload flow, it would be really helpful to be able to read directly from an S3 bucket from our test WDQS nodes (wdqs::alternatives role).

The ultimate plan is for the data copy from s3 to our qlever nodes to happen via an airflow DAG, but for now (while we get that set up, and while we do some initial more manual testing/experimenting) it would be helpful to be able to directly read and write to/from our s3 bucket. (My specific use case right now is building an index manually, which is too big to do on my laptop, so I'd like to use a wdqs node to do it. But then I need to get it into the bucket to make it readable from our other nodes.)

Thank you!

Event Timeline

bking changed the task status from Open to In Progress.May 28 2026, 7:59 PM
bking claimed this task.
bking triaged this task as Medium priority.

I think that what we need to do for this is to add the hostnames for the wdqs::alternatives nodes to the hieradata here:
https://github.com/wikimedia/operations-puppet/blob/production/hieradata/role/common/ceph/server.yaml#L31-L36

In particular, it's probably the profile::tlsproxy::envoy::firewall_srange parameter.

This will open the firewall port 443 on the ceph servers to the source hosts.

Change #1296636 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add the wdqs::alternative nodes to the S3/Ceph envoy firewall

https://gerrit.wikimedia.org/r/1296636

Change #1296636 merged by Ryan Kemper:

[operations/puppet@production] Add the wdqs::alternative nodes to the S3/Ceph envoy firewall

https://gerrit.wikimedia.org/r/1296636

Confirming that I can connect to our S3 bucket from wdqs1030.