Earlier today I applied the class-of-service classifer and scheduler config to interfaces on the Juniper gear in ulsfo. It went smoothly enough, however we have a slight niggle. Detail below.
Core Routers / MX204
Everything looks good on the core routers. For instance taking a transport link we can see that the correct queues are applied, traffic is appearing where we expect (in 'normal' and 'control' classes - control being the traffic the router itself is sending like ospf), and we can see the scheduler parameters in terms of the minimum bandwidth for each of these queues looks right:
cmooney@cr3-ulsfo> show interfaces xe-0/1/1 extensive
Physical interface: xe-0/1/1, Enabled, Physical link is Up
Interface index: 163, SNMP ifIndex: 535, Generation: 189
Description: Transport: cr2-eqord:xe-0/1/3 (Arelion, IC-313592 51ms 10Gbps wave) {#1062}
<<--- cut --->>
Egress queues: 8 supported, 4 in use
Queue counters: Queued packets Transmitted packets Dropped packets
0 1367897928 1367897928 0
1 0 0 0
2 0 0 0
3 19864366 19864366 0
<<--- cut --->>
CoS information:
Direction : Output
CoS transmit queue Bandwidth Buffer Priority Limit
% bps % usec
0 normal 50 5000000000 50 0 low none
1 low 10 1000000000 10 0 low none
2 high 35 3500000000 35 0 low none
3 control 5 500000000 5 0 low noneTop-of-racks / EX4600
On the asw2-ulsfo vc-stack there is an issue, however. Not having had physical units to test in anger I had only been able to validate the configuration was accepted previously, and research online to understand the various platform limits. But it seems on the EX4600 I overlooked that scheduler-maps cannot be applied directly to ports, after pushing the config we get this:
cmooney@asw2-ulsfo> show configuration class-of-service interfaces xe-1/0/0
/* cp4037 {#cp4037d} */
##
## Warning: statement ignored: unsupported platform (ex4600-40f)
##
scheduler-map wmf_map;Looking at the interface the queues in use are correct, but it is not using our scheduler-map (which defines the bandwidth percent for each queue). Instead it's using the default bandwidth percent for them:
cmooney@asw2-ulsfo> show interfaces xe-1/0/0 extensive
Physical interface: xe-1/0/0, Enabled, Physical link is Up
Interface index: 674, SNMP ifIndex: 530, Generation: 167
Description: cp4037 {#cp4037d}
<<--- cut --->>
Egress queues: 12 supported, 5 in use
Queue counters: Queued packets Transmitted packets Dropped packets
0 681254765850 681254765807 43
3 0 0 0
4 0 0 0
7 2131509 2131509 0
8 128739108 128739108 0
<<--- cut --->>
CoS information:
Direction : Output
CoS transmit queue Bandwidth Buffer Priority Limit
% bps % usec
0 normal 5 500000000 5 0 low none
3 low 35 3500000000 35 0 low none
4 high 35 3500000000 35 0 low none
7 control 5 500000000 5 0 low none
8 mcast 20 2000000000 20 0 low noneLimitation
It seems based on the table at the top of this Juniper page that "port scheduling" is not supported for the EX4600 platform. This page explains how the two forms of queueing/scheduling need to be configured. In brief we had planned to use 'port scheduling', configured in the way it describes:
- Mapping a scheduler to the forwarding class in a scheduler map.
- Applying the scheduler map to one or more interfaces.
However it looks like on the EX4600 we can only use ETS / hierarchical port scheduling, which it says needs to be configured as follows:
- Mapping a scheduler to the forwarding class in a scheduler map
- Including the forwarding class in a forwarding class set
- Associating the scheduler map with a traffic control profile
- Attaching the traffic control profile to a forwarding class set and an interface
So basically there are some layers of indirection whereby we can't apply our scheduler-map directly to the interface, but rather need to link it to a "traffic control profile" and attach that to the interface. Additionally we need to use a "forwarding class set" to define what forwarding classes (queues) should be enabled on the interface, rather than it getting that directly from the set of them in the scheduler-map.
Alternate config
Following the above methodology we can create a config like this, which the switch seems happy with
set class-of-service forwarding-class-sets wmf_classes class control set class-of-service forwarding-class-sets wmf_classes class high set class-of-service forwarding-class-sets wmf_classes class low set class-of-service forwarding-class-sets wmf_classes class normal set class-of-service traffic-control-profiles wmf_tc_profile scheduler-map wmf_map set class-of-service traffic-control-profiles wmf_tc_profile guaranteed-rate percent 100 set class-of-service interfaces xe-2/0/18 forwarding-class-set wmf_classes output-traffic-control-profile wmf_tc_profile
Validating the config is a little tricky. Doing "show interfaces xe-2/0/18 extensive" still shows the same queues in use, however the scheduler detail / "CoS information" section is gone from the output. So we can't see if the correct percentage minimum output rate is being applied to each of them. I guess we can assume they are, as it's using the trafifc-control-profile, which in turn uses our scheduler-map, where the percentages are defined. But it would be nice to see it shown.
Doing some basic tests it does seem to be working, for instance adding a few custom rules on another host at the site to mark some traffic into the 'high' priority class it is visible:
cmooney@ganeti4007:~# ping -c 4 -4 ganeti4006.ulsfo.wmnet PING ganeti4006.ulsfo.wmnet (10.128.0.38) 56(84) bytes of data. 64 bytes from ganeti4006.ulsfo.wmnet (10.128.0.38): icmp_seq=1 ttl=64 time=0.207 ms 64 bytes from ganeti4006.ulsfo.wmnet (10.128.0.38): icmp_seq=2 ttl=64 time=0.213 ms 64 bytes from ganeti4006.ulsfo.wmnet (10.128.0.38): icmp_seq=3 ttl=64 time=0.216 ms 64 bytes from ganeti4006.ulsfo.wmnet (10.128.0.38): icmp_seq=4 ttl=64 time=0.150 ms
cmooney@ganeti4007:~# sudo iptables -t mangle -L POSTROUTING -v --line -n Chain POSTROUTING (policy ACCEPT 177K packets, 705M bytes) num pkts bytes target prot opt in out source destination 1 4 336 DSCP icmp -- * * 0.0.0.0/0 10.128.0.38 DSCP set 0x12 2 4 336 RETURN icmp -- * * 0.0.0.0/0 10.128.0.38 3 177K 705M DSCP all -- * * 0.0.0.0/0 0.0.0.0/0 DSCP set 0x00
On the switch we see 4 packets having been sent from the high-priority queue afterwards:
cmooney@asw2-ulsfo> show interfaces xe-2/0/18 extensive
Physical interface: xe-2/0/18, Enabled, Physical link is Up
Interface index: 670, SNMP ifIndex: 603, Generation: 163
Description: ganeti4006 {#ganeti4006d}
<<--- cut --->>
Egress queues: 12 supported, 5 in use
Queue counters: Queued packets Transmitted packets Dropped packets
0 3078707 3078707 0
3 0 0 0
4 4 4 0
7 137537 137537 0
8 68134 68134 0
Queue number: Mapped forwarding classes
0 normal
3 low
4 high
7 control
8 mcastSo it appears to work as expected, and there is no reason to think it would do anything than apply the bandwidth percentages configured. To test that we would need to saturate the port however, not something we can really do in production. We could potentially get two test hosts, one at 10G and one at 1G, and do something, but it's a lot of work.
Alternately we could go with this config on this platform. Ultimately it's only going to matter if ports start saturating, and if the wrong percentages are applied things will be no worse than they were already. So I think overall my preference right now would be to use this config on the 4600s and see how it goes.