Page MenuHomePhabricator

ATS trying to set socket options SO_MARK / IP_TOS
Closed, ResolvedPublic

Description

I have noticed that ATS frequently attempts and fails to set the SO_MARK socket option. As we do not use packet marks, it would be nice to make setting SO_MARK configurable.

setsockopt(57001, SOL_SOCKET, SO_MARK, [0], 4) = -1 EPERM (Operation not permitted)

From socket(7):

SO_MARK (since Linux 2.6.25)
       Set the mark for each packet sent through this socket (similar  to  the  netfilter  MARK
       target  but socket-based).  Changing the mark can be used for mark-based routing without
       netfilter or for packet filtering.  Setting this option requires the CAP_NET_ADMIN capa‐
       bility.

In our environment, neither trafficserver.service nor trafficserver-tls.service have CAP_NET_ADMIN.

Similarly, ATS tries to set IP_TOS too. From ip(7):

IP_TOS (since Linux 1.0)
       Set or receive the Type-Of-Service (TOS) field that is sent with every IP packet  origi‐
       nating  from  this  socket.   It is used to prioritize packets on the network.  TOS is a
       byte.  There are some standard TOS flags defined: IPTOS_LOWDELAY to minimize delays  for
       interactive traffic, IPTOS_THROUGHPUT to optimize throughput, IPTOS_RELIABILITY to opti‐
       mize for reliability, IPTOS_MINCOST should be used for "filler data" where  slow  trans‐
       mission  doesn't  matter.  At most one of these TOS values can be specified.  Other bits
       are invalid and shall be cleared.  Linux sends IPTOS_LOWDELAY  datagrams  first  by  de‐
       fault, but the exact behavior depends on the configured queueing discipline.  Some high-
       priority levels may require superuser privileges (the CAP_NET_ADMIN capability).

Event Timeline

ema created this task.Oct 19 2020, 2:44 PM
Restricted Application added a project: Operations. · View Herald TranscriptOct 19 2020, 2:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Marostegui triaged this task as Medium priority.Oct 19 2020, 3:07 PM

Change 635842 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Release 8.0.8-1wm3

https://gerrit.wikimedia.org/r/635842

Change 635842 merged by Vgutierrez:
[operations/debs/trafficserver@master] Release 8.0.8-1wm3

https://gerrit.wikimedia.org/r/635842

Mentioned in SAL (#wikimedia-operations) [2020-10-26T10:29:47Z] <vgutierrez> upload trafficserver 8.0.8-1wm3 to apt.wm.org (buster) - T265911

Mentioned in SAL (#wikimedia-operations) [2020-10-26T11:11:10Z] <vgutierrez> upgrade trafficserver to 8.0.8-1wm3 on cp4032 - T265911

ema moved this task from Triage to Bug Reports on the Traffic board.Mon, Oct 26, 12:26 PM
ema added a comment.Mon, Oct 26, 12:32 PM

Mentioned in SAL (#wikimedia-operations) [2020-10-26T11:11:10Z] <vgutierrez> upgrade trafficserver to 8.0.8-1wm3 on cp4032 - T265911

Nice, this seems to have worked fine. See cp4032:

root@cp4032:~# timeout --foreground 1 strace -f -p 6649 -e trace=setsockopt 2>&1 | grep -c PERM
0

vs cp4031:

root@cp4031:~# timeout --foreground 1 strace -f -p 40232 -e trace=setsockopt 2>&1 | grep -c PERM
159

Mentioned in SAL (#wikimedia-operations) [2020-10-29T15:06:36Z] <vgutierrez> rolling restart of ATS to upgrade to trafficserver 8.0.8-1wm3 - T265911

Vgutierrez closed this task as Resolved.Thu, Oct 29, 3:46 PM
Vgutierrez claimed this task.
vgutierrez@cumin1001:~$ sudo -i cumin 'A:cp' 'apt-cache policy trafficserver|grep Installed'
72 hosts will be targeted:
cp[2027-2042].codfw.wmnet,cp[1075-1090].eqiad.wmnet,cp[5001-5012].eqsin.wmnet,cp[3050-3065].esams.wmnet,cp[4021-4032].ulsfo.wmnet
Confirm to continue [y/n]? y
===== NODE GROUP =====
(72) cp[2027-2042].codfw.wmnet,cp[1075-1090].eqiad.wmnet,cp[5001-5012].eqsin.wmnet,cp[3050-3065].esams.wmnet,cp[4021-4032].ulsfo.wmnet
----- OUTPUT of 'apt-cache policy...r|grep Installed' -----
  Installed: 8.0.8-1wm3
================