Page MenuHomePhabricator

Deploy Scap version 4.0.2
Closed, ResolvedPublic

Description

Dear SRE team,

Please build and deploy the version 4.0.2 of scap using the instructions in https://wikitech.wikimedia.org/wiki/Scap/Release

This version of Scap requires Python 3 and drops Python 2 support.

Thanks in advance!

Event Timeline

Thank @dancy, I will try to get it done this week with @Arnoldokoth

Mentioned in SAL (#wikimedia-operations) [2021-09-28T08:57:16Z] <effie> upgrade scap on eqiad and codfw - T291095

@Ladsgroup run into this error:

10:39:22 Running '/usr/local/sbin/check-and-restart-php php7.2-fpm 100' on 352 host(s)
10:39:30 Unhandled error:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/scap/cli.py", line 352, in run
    exit_status = app.main(app.extra_arguments)
  File "/usr/lib/python3/dist-packages/scap/main.py", line 928, in main
    return super(SyncFile, self).main(*extra_args)
  File "/usr/lib/python3/dist-packages/scap/main.py", line 153, in main
    self._after_lock_release()
  File "/usr/lib/python3/dist-packages/scap/main.py", line 975, in _after_lock_release
    utils.human_duration(self.get_duration()),
  File "/usr/lib/python3/dist-packages/scap/cli.py", line 150, in announce
    self._announce_logger.info(*args)
  File "/usr/lib/python3.7/logging/__init__.py", line 1383, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/lib/python3.7/logging/__init__.py", line 1519, in _log
    self.handle(record)
  File "/usr/lib/python3.7/logging/__init__.py", line 1529, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.7/logging/__init__.py", line 1591, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.7/logging/__init__.py", line 905, in handle
    self.emit(record)
  File "/usr/lib/python3/dist-packages/scap/log.py", line 135, in emit
    sock.sendall(message)
TypeError: a bytes-like object is required, not 'str'
10:39:30 sync-file failed: <TypeError> a bytes-like object is required, not 'str'

So I rolled back scap on deploy* hosts, until a fix is ready and we can repackage.

Change 724396 had a related patch set uploaded (by Majavah; author: Majavah):

[mediawiki/tools/scap@master] log: Fix socket logging with Python 3

https://gerrit.wikimedia.org/r/724396

Mentioned in SAL (#wikimedia-operations) [2021-09-28T11:05:25Z] <effie> downgrading scap to 3.17.1 on deploy1002 - T291095

Change 724396 merged by jenkins-bot:

[mediawiki/tools/scap@master] log: Fix socket logging with Python 3

https://gerrit.wikimedia.org/r/724396

So I rolled back scap on deploy* hosts, until a fix is ready and we can repackage.

The fix has been merged (thanks @Majavah and @Joe!) and tagged 4.0.1.

dancy renamed this task from Deploy Scap version 4.0.0 to Deploy Scap version 4.0.1.Tue, Sep 28, 3:18 PM
dancy updated the task description. (Show Details)
dancy changed the task status from Open to Stalled.Tue, Sep 28, 7:00 PM

Holding for additional fixes.

@dancy it would be lovely if we can speed this up, right now we have deploy1002 and maps* on version 3.17.1, and the rest on version 4.0.0.

@dancy it would be lovely if we can speed this up, right now we have deploy1002 and maps* on version 3.17.1, and the rest on version 4.0.0.

Sorry @jijiki I was out sick yesterday so I lost a day getting things fixed up.

In the future if there's a problem w/ a new scap release, go ahead and roll back on all machines so we don't leave anybody blocked.

dancy renamed this task from Deploy Scap version 4.0.1 to Deploy Scap version 4.0.2.Fri, Oct 1, 4:44 PM
dancy changed the task status from Stalled to Open.
dancy updated the task description. (Show Details)

@jijiki 4.0.2 is tagged and ready for a retry.

@dancy it would be lovely if we can speed this up, right now we have deploy1002 and maps* on version 3.17.1, and the rest on version 4.0.0.

Sorry @jijiki I was out sick yesterday so I lost a day getting things fixed up.

In the future if there's a problem w/ a new scap release, go ahead and roll back on all machines so we don't leave anybody blocked.

Will do!

@jijiki 4.0.2 is tagged and ready for a retry.

Packaged and ready, will roll out tomorrow.

jijiki claimed this task.