ifup@eno1.service fails on mc* hosts after 4.19.171-2 upgrade
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jijiki
	Feb 4 2021, 6:43 PM

Description

After booting a few memcached servers in codfw, we got some errors when bringing up the network interface (that do not affect server functionality per se)

jiji@mc2019:~$ systemctl --failed
  UNIT              LOAD   ACTIVE SUB    DESCRIPTION
● ifup@eno1.service loaded failed failed ifup for eno1

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

1 loaded units listed. Pass --all to see loaded but inactive units, too.

-- Logs begin at Thu 2021-02-04 14:42:03 UTC, end at Thu 2021-02-04 16:22:52 UTC. --
Feb 04 14:42:04 mc2019 systemd[1]: Started ifup for eno1.
Feb 04 14:42:04 mc2019 sh[751]: Traceback (most recent call last):
Feb 04 14:42:04 mc2019 sh[751]:   File "/usr/lib/python3.7/sre_parse.py", line 1021, in parse_template
Feb 04 14:42:04 mc2019 sh[751]:     this = chr(ESCAPES[this][1])
Feb 04 14:42:04 mc2019 sh[751]: KeyError: '\\d'
Feb 04 14:42:04 mc2019 sh[751]: During handling of the above exception, another exception occurred:
Feb 04 14:42:04 mc2019 sh[751]: Traceback (most recent call last):
Feb 04 14:42:04 mc2019 sh[751]:   File "/usr/local/sbin/interface-rps", line 321, in <module>
Feb 04 14:42:04 mc2019 sh[751]:     main()
Feb 04 14:42:04 mc2019 sh[751]:   File "/usr/local/sbin/interface-rps", line 298, in main
Feb 04 14:42:04 mc2019 sh[751]:     rx_irqs = get_rx_irqs(rss_pattern, rx_queues, q_offset)
Feb 04 14:42:04 mc2019 sh[751]:   File "/usr/local/sbin/interface-rps", line 186, in get_rx_irqs
Feb 04 14:42:04 mc2019 sh[751]:     rss_pat_asre = re.sub('%d', r'(\d+)', rss_pattern)
Feb 04 14:42:04 mc2019 sh[751]:   File "/usr/lib/python3.7/re.py", line 192, in sub
Feb 04 14:42:04 mc2019 sh[751]:     return _compile(pattern, flags).sub(repl, string, count)
Feb 04 14:42:04 mc2019 sh[751]:   File "/usr/lib/python3.7/re.py", line 309, in _subx
Feb 04 14:42:04 mc2019 sh[751]:     template = _compile_repl(template, pattern)
Feb 04 14:42:04 mc2019 sh[751]:   File "/usr/lib/python3.7/re.py", line 300, in _compile_repl
Feb 04 14:42:04 mc2019 sh[751]:     return sre_parse.parse_template(repl, pattern)
Feb 04 14:42:04 mc2019 sh[751]:   File "/usr/lib/python3.7/sre_parse.py", line 1024, in parse_template
Feb 04 14:42:04 mc2019 sh[751]:     raise s.error('bad escape %s' % this, len(this))
Feb 04 14:42:04 mc2019 sh[751]: re.error: bad escape \d at position 1
Feb 04 14:42:05 mc2019 sh[751]: ifup: failed to bring up eno1
Feb 04 14:42:05 mc2019 systemd[1]: ifup@eno1.service: Main process exited, code=exited, status=1/FAILURE
Feb 04 14:42:05 mc2019 systemd[1]: ifup@eno1.service: Failed with result 'exit-code'.

Details

	Subject	Repo	Branch	Lines +/-
	interface: fix regex in interface-rps.py	operations/puppet	production	+1 -1

Customize query in gerrit

Related Objects

Mentioned In: T274103: interface-rps crashes on lsv4007

Event Timeline

jijiki created this task.Feb 4 2021, 6:43 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 4 2021, 6:43 PM

jijiki moved this task from Inbox 🐅 to In Progress 🏋️‍♀️ on the User-jijiki board.Feb 5 2021, 4:44 PM

elukey@mc2029:~$ python3
Python 3.7.3 (default, Jul 25 2020, 13:03:44) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
>>> re.sub('%d', r'(\d+)', "test")
Traceback (most recent call last):
  File "/usr/lib/python3.7/sre_parse.py", line 1021, in parse_template
    this = chr(ESCAPES[this][1])
KeyError: '\\d'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.7/re.py", line 192, in sub
    return _compile(pattern, flags).sub(repl, string, count)
  File "/usr/lib/python3.7/re.py", line 309, in _subx
    template = _compile_repl(template, pattern)
  File "/usr/lib/python3.7/re.py", line 300, in _compile_repl
    return sre_parse.parse_template(repl, pattern)
  File "/usr/lib/python3.7/sre_parse.py", line 1024, in parse_template
    raise s.error('bad escape %s' % this, len(this))
re.error: bad escape \d at position 1

>>> re.sub('%d', r'(\\d+)', "test")
'test'

Maybe interface_rps.py needs extra escape on Python3.7 (since the nodes are running buster) ?

elukey mentioned this in T274103: interface-rps crashes on lsv4007.Feb 8 2021, 9:57 AM

Change 662637 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] interface: fix regex in interface-rps.py

https://gerrit.wikimedia.org/r/662637

gerritbot added a project: Patch-For-Review.Feb 8 2021, 10:02 AM

RhinosF1 subscribed.Feb 8 2021, 10:15 AM

Change 662637 merged by Effie Mouzeli:
[operations/puppet@production] interface: fix regex in interface-rps.py

https://gerrit.wikimedia.org/r/662637

Maintenance_bot removed a project: Patch-For-Review.Feb 8 2021, 11:10 AM