@Ahoelzl has access to provision a service account.

We expect Data-Engineering to work on T405360, which can then be configured for this data transfer.

cc netops (i.e. @ayounsi and @cmooney)

@Antoine_Quhen

Our network can 100% handle that kind of load, but we have some questions to make sure the transfers runs smoothly and doesn't have side effects.

• What are the expected SLO for this process ?
• What happens if there is a failure ?
• What's the timeline for the project ?
• How the 300MB/s value was picked ?
• Is there a design doc on google doc or (better) Wikitech?
• Why use urldownloader and not the HTTP proxies ? - https://wikitech.wikimedia.org/wiki/HTTP_proxy
• What IP/fqdn are you connecting to ?
• How many source nodes will be making the requests? i.e. would rclone be running on just a single node? Or multiple in paralell?
• Is the rclone connectivity a single TCP flow or multiple in paralell? (this will affect how the load is spread across potential proxy servers, and internal network links).

To start with we would recommend considering the HTTP proxies, and lowering the transfer rate (eg. 150MB/s), we could consider increasing it later on

As prerequisites it would be best to make the HTTP proxies fully active/active (depending on expected level of service) as well as Reaching GCP through peering and not transit (to be confirmed once we have the endpoint IP/fqdn)

Please let us know if that makes sens and as usual we're happy discuss it further.

In T427457#12009615, @ayounsi wrote:

• What's the timeline for the project ?

Ideally by the end of the month (end of June 2026), but we'll probably slip a bit. (and please tell us if this is unrealistic on your side)

• Is there a design doc on google doc or (better) Wikitech?

No, not at the moment. This looks simple enough on our side that a design doc looks like unnecessary overhead. But if you need one, we can create something.

• Why use urldownloader and not the HTTP proxies ? - https://wikitech.wikimedia.org/wiki/HTTP_proxy

Because we have no idea :) And we need you to tell us what needs to be used.

To start with we would recommend considering the HTTP proxies, and lowering the transfer rate (eg. 150MB/s), we could consider increasing it later on

Sounds good

As prerequisites it would be best to make the HTTP proxies fully active/active (depending on expected level of service) as well as Reaching GCP through peering and not transit (to be confirmed once we have the endpoint IP/fqdn)

I suspect that we can deal with flakiness with retry on the client side as needed.

@Gehel Thanks for looping in traffic. If I'm reading this correctly, the Hadoop cluster will be handling the traffic, which will not be going through LVS. In that case, I'm not sure there's much of traffic domain that will be touched during this process.

Please do correct me if I'm wrong and there's something we need to look at!

In T427457#12011345, @BCornwall wrote:

If I'm reading this correctly, the Hadoop cluster will be handling the traffic, which will not be going through LVS. In that case, I'm not sure there's much of traffic domain that will be touched during this process.

Yes, Hadoop is going to initiate connections to Google Cloud Storage. This is still going through network, so we want to be sure that the additional traffic is not putting too much pressure on network. And that we're going through the right path (@ayounsi mentioned using a different proxy and not urldownloader). This is an area where we don't have a lot of expertise, maybe this is all trivial on your side, in which case, all good!

@Gehel thanks for the info.

Does anyone have any answers to the remaining questions from us?

• What IP/fqdn are you connecting to ?
• How many source nodes will be making the requests? i.e. would rclone be running on just a single node? Or multiple in paralell?
• Is the rclone connectivity a single TCP flow from each host or multiple in paralell? (this will affect how the load is spread across potential proxy servers, and internal network links).

In T427457#12010832, @Gehel wrote:

In T427457#12009615, @ayounsi wrote:

• Why use urldownloader and not the HTTP proxies ? - https://wikitech.wikimedia.org/wiki/HTTP_proxy

Because we have no idea :) And we need you to tell us what needs to be used.

Yep no problem there, urldownloader is more for a specific set of things but even I get mixed up. Either way I think for this - or in general - the HTTP proxies are better to use.

To support this I think we'll need to beef up the proxy service and provision some more of them though. Those other answers will help us size up that requirement properly. I can't commit to having that all ready this month but it's not a massive task we hopefully can get things in place reasonably quick for you.

Ben should be able to help answer those other questions on Monday.

Here are some answers to the best of my current knowledge.

• What IP/fqdn are you connecting to ?

We will likely be connecting to storage.us-east4.rep.googleapis.com which currently has an IPv4 address of 34.1.73.193

That will be the case if the bucket can be co-located there, as @Antoine_Quhen mentioned in T427457#11993782
If that's not possible, or not convenient, then it might just be the standard API endpoint of storage.googleapis.com and there is a range of different IPv4 and IPv6 addresses for that.

btullis@barracuda:~$ host storage.googleapis.com
storage.googleapis.com has address 142.250.140.207
storage.googleapis.com has address 142.251.29.207
storage.googleapis.com has address 142.250.129.207
storage.googleapis.com has address 142.250.117.207
storage.googleapis.com has address 192.178.223.207
storage.googleapis.com has address 142.251.30.207
storage.googleapis.com has address 142.250.151.207
storage.googleapis.com has IPv6 address 2a00:1450:4009:c13::cf
storage.googleapis.com has IPv6 address 2a00:1450:4009:c17::cf
storage.googleapis.com has IPv6 address 2a00:1450:4009:c04::cf
storage.googleapis.com has IPv6 address 2a00:1450:4009:c08::cf

I'm not sure yet how frequently these might change, or whether there is any variation to that list depending on the location of the request.

• How many source nodes will be making the requests? i.e. would rclone be running on just a single node? Or multiple in paralell?

I am not 100% sure yet, but my suspicion is that we will use:

• one rclone process for each daily download job, so one host or pod
• up to four rclone processes for the initial backfill operation, so up to four source nodes

• Is the rclone connectivity a single TCP flow from each host or multiple in paralell? (this will affect how the load is spread across potential proxy servers, and internal network links).

The rclone utility offers a few options for tuning this behaviour. The main one is --transfers <int> - The number of file transfers to run in parallel. (default 4)
unfortunately, we can't use --multi-thread-streams <int> with a GCP source and an HDFS target

There is another option called --checkers <int> which will affect the number of simultaneous TCP connections opened from each process, with the default being 8.

So, with the default options, we would expect a single rclone process to open 12 concurrent TCP streams to the source, but I suspect that we will want to increase the value for --transfers to a number greater than 4.

The parallelism values that we end up choosing might also vary depending on the size and number of the exported parquet files, but we will keep you in the loop once we start testing these transfers.

To support this I think we'll need to beef up the proxy service and provision some more of them though. Those other answers will help us size up that requirement properly. I can't commit to having that all ready this month but it's not a massive task we hopefully can get things in place reasonably quick for you.

I can also help with any necessary scaling of the HTTP Proxy service, if that would be useful.

In T427457#12010832, @Gehel wrote:

• Why use urldownloader and not the HTTP proxies ? - https://wikitech.wikimedia.org/wiki/HTTP_proxy

Because we have no idea :) And we need you to tell us what needs to be used.

I was actually probably wrong about this. The urldownloaders already have new VMs provisioned -- 4 in each cluster -- and mostly just need loadbalancing set up. (This was already happening for hcaptcha reasons.) Whereas, the only other role with the caching squid proxy provisioned rn is installservers, I think.

Long-term I still think it makes sense to split usages based on "batch" vs "user-facing", but, I think it'd be acceptable to have this ride on the urldownloaders temporarily, since it'd save a lot of short-term work.

Let's discuss with @MoritzMuehlenhoff at the Monday I/F meeting I think

In T427457#11984157, @Gehel wrote:

@Ahoelzl has access to provision a service account.

This is the next critical step, as far as I am concerned.
@Ahoelzl would you mind sharing how this is this achieved please, at a practical level?

Do we have any sort of configuration management around this kind of GCP access control?
How would we go about revoking or renewing these credentials, if and when that need arises?

Does anyone on the SRE side have the necessary rights to manage roles and serviceaccounts for Google Cloud Platform?
If not, would it be prudent to configure it?

Maybe these are already completely solved problems and nothing that I need worry about, but I would like to know that we have the right processes in place to be referred to, if needed. Thanks.

! In T427457#12015123, @CDanis wrote:
I was actually probably wrong about this. The urldownloaders already have new VMs provisioned -- 4 in each cluster -- and mostly just need loadbalancing set up. (This was already happening for hcaptcha reasons.) Whereas, the only other role with the caching squid proxy provisioned rn is installservers, I think.

Yes, I was under the impression that it was just the installservers that serve the HTTP Proxies, too.

I'm happy to help set up the active/active load-balancing across the urldownloader VMs too, if that would be better than scaling the HTTP proxy role..

We discussed the HTTP Proxy vs. urldownloader during the I/F meeting and you can go ahead with using the urldownloader.

On our side, we're going to look at next steps to improve the two services, but we don't want you to be blocked.

We will likely be connecting to storage.us-east4.rep.googleapis.com which currently has an IPv4 address of 34.1.73.193
If that's not possible, or not convenient, then it might just be the standard API endpoint of storage.googleapis.com and there is a range of different IPv4 and IPv6 addresses for that.

Connectivity properly goes through our peering port.

And T429175: Scaling urldownloaders by adding redundancy and load balancing has been opened for a active/active urldownloader. I don't think that's a blocker if you lower the transfer to 150MB/s, but it would be necessary on the medium term.

@Antoine_Quhen newly has access to Google Cloud and can configure granular data access going forward.
So far I created a service account with a credential file which I can share privately. rclone seems to be configurable with that information.

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/sync-utils/-/merge_requests/6

Bump rclone to 1.74.3 from upstream binary

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/sync-utils/-/merge_requests/6

Bump rclone to 1.74.3 from upstream binary

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2326

Draft: Add rclone GCS-to-HDFS operator and GSC DAG

Change #1305857 had a related patch set uploaded (by Aqu; author: Aqu):

[analytics/refinery@master] Add GSC site/url impression Hive tables

https://gerrit.wikimedia.org/r/1305857

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2364

Add gcloud-cli Airflow image build (blubber + ci)

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2364

Add gcloud-cli Airflow image build (blubber + ci)

Change #1307184 had a related patch set uploaded (by Aqu; author: Aqu):

[operations/deployment-charts@master] Airflow-main: Add GCP Airflow connection

https://gerrit.wikimedia.org/r/1307184

Change #1309204 had a related patch set uploaded (by Aqu; author: Aqu):

[operations/deployment-charts@master] airflow: add gcs-token-search-console secret

https://gerrit.wikimedia.org/r/1309204

Change #1307184 abandoned by Aqu:

[operations/deployment-charts@master] Airflow-main: Add GCP Airflow connection

Reason:

Airflow connections leaks into logs. Replaced by 1309204

https://gerrit.wikimedia.org/r/1307184

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2326

Add rclone GCS-to-HDFS operator and GSC DAG

Change #1309204 merged by jenkins-bot:

[operations/deployment-charts@master] airflow: add gcs-token-search-console secret

https://gerrit.wikimedia.org/r/1309204

Change #1305857 merged by Aqu:

[analytics/refinery@master] Add GSC site/url impression Hive tables

https://gerrit.wikimedia.org/r/1305857

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2470

gsc: remove one-off backfill DAG

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2470

gsc: remove one-off backfill DAG