In T381389#10376688, @CDanis wrote:

I think we need an-worker* source port 50010, which I am pretty sure is just the dataplane of HDFS and not the metadata service or control plane.

Yep I confirmed this in Turnilo from the sflow data also.

Change #1100166 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] New ferm rule to permit HDFS data flows and mark as low-prio for qos

https://gerrit.wikimedia.org/r/1100166

This change looks fine to me, but would it be OK to wait until the New Year to implement it? I'm just a bit cautious about making significant changes to the HDFS data flows at this time of year.

In T381389#10389706, @BTullis wrote:

This change looks fine to me, but would it be OK to wait until the New Year to implement it? I'm just a bit cautious about making significant changes to the HDFS data flows at this time of year.

Sure I think that is fine Ben caution is always best.

@cmooney, should we move forward with this patch sometime soon?

In T381389#10583616, @xcollazo wrote:

@cmooney, should we move forward with this patch sometime soon?

Sorry I'd somehow missed this comment. Yes we just need to merge it can happen any time, I spoke to Ben about it just now and we're gonna do it next Tuesday.

Change #1100166 merged by Btullis:

[operations/puppet@production] New ferm rule to permit HDFS data flows and mark as low-prio for qos

https://gerrit.wikimedia.org/r/1100166

Things are looking good after the application of the change, an-worker nodes are correctly marking Hadoop traffic with DSCP AF11 mark (see P74391) and the network is mapping those packets into the lower-priority queue across the core as expected.

So now, if someone runs a query which generates sufficient Hadoop traffic to flood the core network links, the switches will ensure those flows do not squeeze out our other production traffic.

Hello, I would like to exercise this rule by running a very heavy Presto query.

Is there any Grafana dashboard I could monitor so that I cut my query short in the event I start causing trouble?

In T381389#10695612, @xcollazo wrote:

Hello, I would like to exercise this rule by running a very heavy Presto query.

Is there any Grafana dashboard I could monitor so that I cut my query short in the event I start causing trouble?

Yes. You can use this presto UI to give you admin access to the coordinator, from which you can cancel running queries and other tasks.
https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Presto/Administration#View_the_Presto_UI

You can use an SSH port forward on an-coord1003 to give you this, by virtue of the fact that you are in the analytics-admins POSIX group.

Cathal and I also did a bit of monitoring of the traffic, so I can also point you in the direction of this dashboard. https://grafana.wikimedia.org/goto/MgLIUQoHg?orgId=1

You can view this in real time, too. The important parts of this dashboard are the metrics:

• Queue 3 tx packets
• Queue 3 dropped packets
• Queue 3 tx bits/sec

You can see that in the last hour we have had a large query, transmitting a large amount of traffic from Hadoop to an-presto1006.
At the time the rate was 1.5 Gbit/sec. The peak packet rate was 125 kbps (left axis). At this time, the switch was unable to buffer all of the incoming traffic, so it dropped around 450 packets/sec (right axis).

We only have these stats for some of the presto hosts, which are those in rows E and F, as the equipment in rows A-D don't emit the same detail of metrics.

Let us know how you get on with your testing.

Ok attempting the below query again now:

In T390623#10699223, @xcollazo wrote:

Here is the query I ran that made Presto fail, for completeness:

presto:wmf_content> select max(length(revision_content_slots['main'].content_body)) from mediawiki_content_history_v1 where wiki_id = 'commonswiki';

Query 20250331_200300_00561_jxz5w, FAILED, 15 nodes
Splits: 4,227 total, 3,995 done (94.51%)
[Latency: client-side: 3:39, server-side: 3:39] [908M rows, 285GB] [4.15M rows/s, 1.3GB/s]

Query 20250331_200300_00561_jxz5w failed: Could not communicate with the remote task. The node may have crashed or be under too much load. This is probably a transient issue, so please retry your query in a few minutes. (an-presto1017.eqiad.wmnet:8281)

It is a benchmark query, I was trying to flood the routers between HDFS Datanodes and Presto nodes.

Will try again after we merge this heap patch.

I've succesfully run the following query:

In T390623#10699616, @xcollazo wrote:

Success!

presto> select max(length(revision_content_slots['main'].content_body)) from wmf_content.mediawiki_content_history_v1 where wiki_id = 'commonswiki';
  _col0  
---------
 4901848 
(1 row)

Query 20250401_150139_00003_cdjjz, FINISHED, 12 nodes
Splits: 4,272 total, 4,272 done (100.00%)
[Latency: client-side: 4:12, server-side: 4:12] [974M rows, 308GB] [3.86M rows/s, 1.22GB/s]

...

This effectively moved 308GB from HDFS Datanodes, thru the routers, to Presto servers, with average throughput of 1.22GB/s.

I want to make sure we saw no production issues. If so, I will then attempt a query that will move ~20TBs, probably at about the same throughput.

We only have these stats for some of the presto hosts, which are those in rows E and F, as the equipment in rows A-D don't emit the same detail of metrics.

@BTullis is this a software, or hardware limitation? Also, this is just a matter of metrics, but the QoS rules still apply to all presto servers, yes?

This effectively moved 308GB from HDFS Datanodes, thru the routers, to Presto servers, with average throughput of 1.22GB/s.

I want to make sure we saw no production issues. If so, I will then attempt a query that will move ~20TBs, probably at about the same throughput.

Yeah we didn't saturate any interfaces across the core that I can see, so that one didn't cause us any issues Looking at some of the core network interface stats, and indeed the an-presto ones on the link Ben posted above, I can see that the profiling was correct. So had we maxed out any of those links the QoS would have helped.

You should be ok to run a larger query, but let's not go too mad. QoS will help if we max things out but it's not a magic bullet to make all impact go away (we'd still get latency increases, it would page SREs etc).

In T381389#10699693, @xcollazo wrote:

We only have these stats for some of the presto hosts, which are those in rows E and F, as the equipment in rows A-D don't emit the same detail of metrics.

@BTullis is this a software, or hardware limitation? Also, this is just a matter of metrics, but the QoS rules still apply to all presto servers, yes?

Yes the QoS configuration is applied everywhere and works across the board. The lack of visibility in those rows is due to older devices not supporting the gnmi-based telemetry pipeline we are using on newer switches to expose these stats. The live numbers are available from the CLI if we need to troubleshoot though.

FWIW the largest potential bottleneck in Ashburn are on the 10G interfaces (names starting 'xe') connect access switches from core routers, which can be seen here:

https://grafana.wikimedia.org/goto/QfJf_XoNg

https://grafana.wikimedia.org/goto/pTYElXoNg

You can see the spike in traffic over the past hour (all in queue number 1 which is the tell it's due to these queries), but we've no maxed out any of them thankfully.

Thanks for the pointers @cmooney.

Here are my heavy query results:

First, I attempted query failed due to transient error:

presto> select max(length(revision_content_slots['main'].content_body)) from wmf_content.mediawiki_content_history_v1 where wiki_id='enwiki';

Query 20250401_153238_00024_cdjjz, FAILED, 15 nodes
Splits: 57,360 total, 52,468 done (91.47%)
[Latency: client-side: 19:59, server-side: 19:59] [0 rows, 0B] [0 rows/s, 0B/s]

Query 20250401_153238_00024_cdjjz failed: Error reading from /wmf/data/wmf_content/mediawiki_content_history_v1/data/wiki_id=enwiki/01221-346327-f15044f1-ab5f-463a-bb6e-9eb664ad9e79-00003.parquet at position 128590371

Seems like it is easy for Iceberg tables to fail when Presto is really busy. This is unrelated to QoS, so I switched to the query that started this whole effort when I caused a minor prod outage with it before the QoS markings:

SELECT
    count(1) as count,
    user_id_categories
FROM (
        SELECT
            CASE
                WHEN user_id = 0 THEN 'user_id = 0'
                WHEN user_id = -1 THEN 'user_id = -1'
                WHEN user_id < -1 THEN 'user_id < -1'
                WHEN user_id IS NULL THEN 'user_id IS NULL'
                ELSE 'user_id > 0'
            END AS user_id_categories
        FROM wmf.mediawiki_wikitext_history
        WHERE snapshot = '2025-02'
)
GROUP BY user_id_categories
ORDER BY count DESC

Here is the run and results:

presto> SELECT
     ->     count(1) as count,
     ->     user_id_categories
     -> FROM (
     ->         SELECT
     ->             CASE
     ->                 WHEN user_id = 0 THEN 'user_id = 0'
     ->                 WHEN user_id = -1 THEN 'user_id = -1'
     ->                 WHEN user_id < -1 THEN 'user_id < -1'
     ->                 WHEN user_id IS NULL THEN 'user_id IS NULL'
     ->                 ELSE 'user_id > 0'
     ->             END AS user_id_categories
     ->         FROM wmf.mediawiki_wikitext_history
     ->         WHERE snapshot = '2025-02'
     -> )
     -> GROUP BY user_id_categories
     -> ORDER BY count DESC;
   count    | user_id_categories 
------------+--------------------
 6610858626 | user_id > 0        
  526779900 | user_id IS NULL    
    8437200 | user_id = 0        
     760449 | user_id = -1       
(4 rows)

Query 20250401_160931_00030_cdjjz, FINISHED, 15 nodes
Splits: 795,809 total, 795,809 done (100.00%)
[Latency: client-side: 71:16, server-side: 71:16] [7.15B rows, 42TB] [1.67M rows/s, 10.1GB/s]

Thus we moved 42TB of data from HDFS over to Presto. Grafana link for this event: https://grafana.wikimedia.org/goto/mDjTquTNR?orgId=1

Grafana link for the core router for this event: https://grafana.wikimedia.org/goto/ihY1qXoNg?orgId=1. Looks like the query had a sustained ~60GB/s load on the core router.

No one is yelling on IRC so I think I am happy with this. I am done from my side.

No one is yelling on IRC so I think I am happy with this. I am done from my side.

Ok cool. FWIW we did saturate some links on the core, similar to what caused problems the last time.

Unfortunately it was outbound on the devices where we don't have stats. We got paged for the link usage, but no other alarms fired - unlike last time. So I'm happy that the QoS is doing what we need of it and this didn't affect other applications across the DC.

Also to get a sense of total throughput this graph is good:

https://grafana.wikimedia.org/goto/dVbqeuTNR

Looks like it was fairly solid at ~125Gbps across the network.

In T381389#10700438, @cmooney wrote:

No one is yelling on IRC so I think I am happy with this. I am done from my side.

Ok cool. FWIW we did saturate some links on the core, similar to what caused problems the last time.

Unfortunately it was outbound on the devices where we don't have stats. We got paged for the link usage, but no other alarms fired - unlike last time. So I'm happy that the QoS is doing what we need of it and this didn't affect other applications across the DC.

Got it. I will copy this to T381416: Do performance testing of a big Hadoop Table hosted by Ceph to keep it in mind whenever we do that work.