Thank @dancy, I will try to get it done this week with @Arnoldokoth

Mentioned in SAL (#wikimedia-operations) [2021-09-24T11:58:35Z] <effie> upgrading scap on canaries - T291095

Mentioned in SAL (#wikimedia-operations) [2021-09-28T08:57:16Z] <effie> upgrade scap on eqiad and codfw - T291095

@Ladsgroup run into this error:

10:39:22 Running '/usr/local/sbin/check-and-restart-php php7.2-fpm 100' on 352 host(s)
10:39:30 Unhandled error:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/scap/cli.py", line 352, in run
    exit_status = app.main(app.extra_arguments)
  File "/usr/lib/python3/dist-packages/scap/main.py", line 928, in main
    return super(SyncFile, self).main(*extra_args)
  File "/usr/lib/python3/dist-packages/scap/main.py", line 153, in main
    self._after_lock_release()
  File "/usr/lib/python3/dist-packages/scap/main.py", line 975, in _after_lock_release
    utils.human_duration(self.get_duration()),
  File "/usr/lib/python3/dist-packages/scap/cli.py", line 150, in announce
    self._announce_logger.info(*args)
  File "/usr/lib/python3.7/logging/__init__.py", line 1383, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/lib/python3.7/logging/__init__.py", line 1519, in _log
    self.handle(record)
  File "/usr/lib/python3.7/logging/__init__.py", line 1529, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.7/logging/__init__.py", line 1591, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.7/logging/__init__.py", line 905, in handle
    self.emit(record)
  File "/usr/lib/python3/dist-packages/scap/log.py", line 135, in emit
    sock.sendall(message)
TypeError: a bytes-like object is required, not 'str'
10:39:30 sync-file failed: <TypeError> a bytes-like object is required, not 'str'

So I rolled back scap on deploy* hosts, until a fix is ready and we can repackage.

Change 724396 had a related patch set uploaded (by Majavah; author: Majavah):

[mediawiki/tools/scap@master] log: Fix socket logging with Python 3

https://gerrit.wikimedia.org/r/724396

Mentioned in SAL (#wikimedia-operations) [2021-09-28T11:05:25Z] <effie> downgrading scap to 3.17.1 on deploy1002 - T291095

Change 724396 merged by jenkins-bot:

[mediawiki/tools/scap@master] log: Fix socket logging with Python 3

https://gerrit.wikimedia.org/r/724396

In T291095#7383145, @jijiki wrote:

So I rolled back scap on deploy* hosts, until a fix is ready and we can repackage.

The fix has been merged (thanks @Majavah and @Joe!) and tagged 4.0.1.

@dancy it would be lovely if we can speed this up, right now we have deploy1002 and maps* on version 3.17.1, and the rest on version 4.0.0.

There are some more Python 3 related issues that need to be addressed such as T291990: Scap error when deploying kartotherian - https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/724527

In T291095#7390898, @jijiki wrote:

@dancy it would be lovely if we can speed this up, right now we have deploy1002 and maps* on version 3.17.1, and the rest on version 4.0.0.

Sorry @jijiki I was out sick yesterday so I lost a day getting things fixed up.

In the future if there's a problem w/ a new scap release, go ahead and roll back on all machines so we don't leave anybody blocked.

Mentioned in SAL (#wikimedia-operations) [2021-10-04T14:46:16Z] <effie> uploading scap 4.0.2 - T291095

In T291095#7395126, @dancy wrote:

In T291095#7390898, @jijiki wrote:

@dancy it would be lovely if we can speed this up, right now we have deploy1002 and maps* on version 3.17.1, and the rest on version 4.0.0.

Sorry @jijiki I was out sick yesterday so I lost a day getting things fixed up.

In the future if there's a problem w/ a new scap release, go ahead and roll back on all machines so we don't leave anybody blocked.

Will do!

In T291095#7395257, @dancy wrote:

@jijiki 4.0.2 is tagged and ready for a retry.

Packaged and ready, will roll out tomorrow.

Mentioned in SAL (#wikimedia-operations) [2021-10-05T11:15:08Z] <effie> upgrade scap to 4.0.2 - T291095

Yay! thanks @jijiki !