interface-rps crashes on lsv4007
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Vgutierrez
	Feb 8 2021, 9:49 AM

Description

Performing a rutinary system reboot on lvs4007 I've found a crash on interfaces-rps trying to bring up enp5s0f0:

vgutierrez@lvs4007:~$ journalctl -u "ifup@enp5s0f0.service"
-- Logs begin at Mon 2021-02-08 09:23:42 UTC, end at Mon 2021-02-08 09:48:01 UTC. --
Feb 08 09:23:44 lvs4007 systemd[1]: Started ifup for enp5s0f0.
Feb 08 09:23:45 lvs4007 sh[944]: Traceback (most recent call last):
Feb 08 09:23:45 lvs4007 sh[944]:   File "/usr/lib/python3.7/sre_parse.py", line 1021, in parse_template
Feb 08 09:23:45 lvs4007 sh[944]:     this = chr(ESCAPES[this][1])
Feb 08 09:23:45 lvs4007 sh[944]: KeyError: '\\d'
Feb 08 09:23:45 lvs4007 sh[944]: During handling of the above exception, another exception occurred:
Feb 08 09:23:45 lvs4007 sh[944]: Traceback (most recent call last):
Feb 08 09:23:45 lvs4007 sh[944]:   File "/usr/local/sbin/interface-rps", line 321, in <module>
Feb 08 09:23:45 lvs4007 sh[944]:     main()
Feb 08 09:23:45 lvs4007 sh[944]:   File "/usr/local/sbin/interface-rps", line 298, in main
Feb 08 09:23:45 lvs4007 sh[944]:     rx_irqs = get_rx_irqs(rss_pattern, rx_queues, q_offset)
Feb 08 09:23:45 lvs4007 sh[944]:   File "/usr/local/sbin/interface-rps", line 186, in get_rx_irqs
Feb 08 09:23:45 lvs4007 sh[944]:     rss_pat_asre = re.sub('%d', r'(\d+)', rss_pattern)
Feb 08 09:23:45 lvs4007 sh[944]:   File "/usr/lib/python3.7/re.py", line 192, in sub
Feb 08 09:23:45 lvs4007 sh[944]:     return _compile(pattern, flags).sub(repl, string, count)
Feb 08 09:23:45 lvs4007 sh[944]:   File "/usr/lib/python3.7/re.py", line 309, in _subx
Feb 08 09:23:45 lvs4007 sh[944]:     template = _compile_repl(template, pattern)
Feb 08 09:23:45 lvs4007 sh[944]:   File "/usr/lib/python3.7/re.py", line 300, in _compile_repl
Feb 08 09:23:45 lvs4007 sh[944]:     return sre_parse.parse_template(repl, pattern)
Feb 08 09:23:45 lvs4007 sh[944]:   File "/usr/lib/python3.7/sre_parse.py", line 1024, in parse_template
Feb 08 09:23:45 lvs4007 sh[944]:     raise s.error('bad escape %s' % this, len(this))
Feb 08 09:23:45 lvs4007 sh[944]: re.error: bad escape \d at position 1
Feb 08 09:23:45 lvs4007 sh[944]: ifup: failed to bring up enp5s0f0
Feb 08 09:23:45 lvs4007 systemd[1]: ifup@enp5s0f0.service: Main process exited, code=exited, status=1/FAILURE
Feb 08 09:23:45 lvs4007 systemd[1]: ifup@enp5s0f0.service: Failed with result 'exit-code'.

Details

	Subject	Repo	Branch	Lines +/-
	interface: fix regex in interface-rps.py	operations/puppet	production	+1 -1

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
					Restricted Task
		Resolved		jijiki	T274103 interface-rps crashes on lsv4007

Event Timeline

Vgutierrez created this task.Feb 8 2021, 9:49 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 8 2021, 9:49 AM

Vgutierrez triaged this task as Unbreak Now! priority.Feb 8 2021, 9:50 AM

Vgutierrez moved this task from Backlog to LoadBalancer on the Traffic board.

It seems to me that this is related to https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/interface/files/interface-rps.py#186

>>> a = 'foo %d'
>>> b = re.sub('%d', r'(\d+)', a)
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/sre_parse.py", line 1039, in parse_template
    this = chr(ESCAPES[this][1])
KeyError: '\\d'
...

https://phabricator.wikimedia.org/T273918 was filed by Effie last week, it also happened for some MW servers.

Change 662637 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] interface: fix regex in interface-rps.py

https://gerrit.wikimedia.org/r/662637

gerritbot added a project: Patch-For-Review.Feb 8 2021, 10:02 AM

I did just a quick look but I think that if you use \\d it should do the right thing:

>>> a = 'foo %d bar'
>>> b = re.sub('%d', r'(\\d+)', a)
>>> c = re.compile(r'^\s*([0-9]+):.*\s' + b + r'\n$')
>>> c
re.compile('^\\s*([0-9]+):.*\\sfoo (\\d+) bar\\n$')

RhinosF1 subscribed.Feb 8 2021, 10:15 AM

Change 662637 merged by Effie Mouzeli:
[operations/puppet@production] interface: fix regex in interface-rps.py

https://gerrit.wikimedia.org/r/662637