Update Debian Package for Scap3
Closed, ResolvedPublic

Description

A few bug fixes came out of the AQS deployment. Would like to get them packaged up and deployed on the servers. Newest deployment tag at: https://phabricator.wikimedia.org/diffusion/MSCA/browse/master/;3.0.2

There are a very large number of changes, so older changes are hidden. Show Older Changes
fgiunchedi closed this task as Resolved.Sep 23 2016, 3:16 PM

scap uploaded and deployed, resolving

thcipriani reopened this task as Open.Nov 13 2016, 11:48 PM

@fgiunchedi could you update the scap packages to v.3.3.1-1?

Changelog:

scap (3.3.1-1) unstable; urgency=low

  * Scap learned how to lock all deployments at the server level. Previously
    it only knew how to lock deployments at the repo level.

  * Scap now tells users who holds the lock file (when deploys are locked).
    Permissions on the lockfile have changed to 600, making it harder to
    overwrite (Fixes T140914).

  * "scap sync-file" and "scap pull" got new flags, "--beta-only-change" and
    "--no-touch", respectively. When used, this prevents the normal action
    taken by "scap pull" to touch the wmf-config/InitialiseSettings.php file
    which invalidate local caches.  This allows some syncs to avoid a
    potential problem with HHVM servers exhausting local cache when reading
    new files from disk. The option is now also always passed to `scap pull`
    by `scap sync-l10n` (fixes T149872 -- thank you Bryan Davis).

  * Scap learned to announce scap3 deployments in IRC. This should mean that
    deployers will no longer have to announce a deployment manually in
    *-operations -- scap will do it for them.

  * Scap3 updated the way it limits hosts and can now limit hosts from all
    groups, not just the default group (Fixes T149128).

  * HHVM restarts now happen via the /usr/local/bin/restart-hhvm script. This
    means that an individual server is now depooled via confctl before being
    restarted (thank you Giuseppe Lavagetto).

  * Scap3 learned about empty checks in the checks.yml file. This allows you
    to overwrite global checks in an environment specific checks.yml. These
    checks will be logged, but nothing will run (fixes T149668).

  * Scap3 added a "finalize" stage of deployment. This will allow for "promote"
    checks to execute before the final state is recorded and clean up of old
    rev directories is performed, and overall result in more consistent
    rollback behavior (Fixes T150267)

  * Scap learned how to use sub-sub commands, i.e., scap subcommand
    subsubcommand. None are yet implemented.

  * scap l10n-purge works once again, and restart_hhvm was restored (Fixes
    T146656).

  * Internally, scap simplified its usage of sudo calls. This means the
    internals are a bit more sane, unnecessary sudo calls, i.e., sudoing as
    yourself is less common throughout the code.

 -- Tyler Cipriani <tcipriani@wikimedia.org>  Thu, 10 Nov 2016 13:41:28 -0800

Thanks!

Gerrit patch for puppet (since it didn't link) https://gerrit.wikimedia.org/r/#/c/321339,edit/

thcipriani changed the task status from Open to Stalled.Nov 21 2016, 4:54 PM

Marking task as stalled. Want to get a new version with fix for T150897: Env vars being overwritten in it.

thcipriani changed the task status from Stalled to Open.Nov 28 2016, 4:36 PM

@fgiunchedi could you update the scap packages to v.3.4.0-1?

Changelog:

1​scap (3.4.0-1) unstable; urgency=low
2
3​ * BREAKING CHANGE: Old scap bin stubs (e.g., sync-file, sync-dir,
4​ mwversionsinuse, etc) will now exit 1, subcommands are now the only way to
5​ interact with scap, i.e., sync-file is now scap sync-file.
6
7​ * Scap linter now ignores auto autoload_static.php which was known to fail
8​ lint checks (Fixes T136009 - thank you Paladox)
9
10​ * Scap adds the sha1 of the commit being deployed to the announce log --
11​ this change affects scap3 repos.
12
13​ * This release fixes a bug where variables specified in vars.yaml were being
14​ overwritten in the opposite order (Fixes T150897)
15
16​ * Scap lockfiles now contain the message passed in the deployment, i.e. scap
17​ sync-dir wmf-config 'Update config' will create a lockfile with the
18​ message 'Update config'. If anyone else attempts to deploy while that
19​ lockfile is in place, they will be shown the lockfile message.
20
21​ -- Tyler Cipriani <tcipriani@wikimedia.org> Mon, 28 Nov 2016 09:03:50 -0700
22
23​scap (3.3.1-1) unstable; urgency=low
24
25​ * Scap learned how to lock all deployments at the server level. Previously
26​ it only knew how to lock deployments at the repo level.
27
28​ * Scap now tells users who holds the lock file (when deploys are locked).
29​ Permissions on the lockfile have changed to 600, making it harder to
30​ overwrite (Fixes T140914).
31
32​ * "scap sync-file" and "scap pull" got new flags, "--beta-only-change" and
33​ "--no-touch", respectively. When used, this prevents the normal action
34​ taken by "scap pull" to touch the wmf-config/InitialiseSettings.php file
35​ which invalidate local caches. This allows some syncs to avoid a
36​ potential problem with HHVM servers exhausting local cache when reading
37​ new files from disk. The option is now also always passed to `scap pull`
38​ by `scap sync-l10n` (fixes T149872 -- thank you Bryan Davis).
39
40​ * Scap learned to announce scap3 deployments in IRC. This should mean that
41​ deployers will no longer have to announce a deployment manually in
42​ *-operations -- scap will do it for them.
43
44​ * Scap3 updated the way it limits hosts and can now limit hosts from all
45​ groups, not just the default group (Fixes T149128).
46
47​ * HHVM restarts now happen via the /usr/local/bin/restart-hhvm script. This
48​ means that an individual server is now depooled via confctl before being
49​ restarted (thank you Giuseppe Lavagetto).
50
51​ * Scap3 learned about empty checks in the checks.yml file. This allows you
52​ to overwrite global checks in an environment specific checks.yml. These
53​ checks will be logged, but nothing will run (fixes T149668).
54
55​ * Scap3 added a "finalize" stage of deployment. This will allow for "promote"
56​ checks to execute before the final state is recorded and clean up of old
57​ rev directories is performed, and overall result in more consistent
58​ rollback behavior (Fixes T150267)
59
60​ * Scap learned how to use sub-sub commands, i.e., scap subcommand
61​ subsubcommand. None are yet implemented.
62
63​ * scap l10n-purge works once again, and restart_hhvm was restored (Fixes
64​ T146656).
65
66​ * Internally, scap simplified its usage of sudo calls. This means the
67​ internals are a bit more sane, unnecessary sudo calls, i.e., sudoing as
68​ yourself is less common throughout the code.
69
70​ -- Tyler Cipriani <tcipriani@wikimedia.org> Thu, 10 Nov 2016 13:41:28 -0800

  • Thanks!

Change 323852 had a related patch set uploaded (by Filippo Giunchedi):
scap: bump version to 3.4.0-1

https://gerrit.wikimedia.org/r/323852

Change 323852 merged by Filippo Giunchedi:
scap: bump version to 3.4.0-1

https://gerrit.wikimedia.org/r/323852

fgiunchedi closed this task as Resolved.Nov 29 2016, 1:02 AM

Package built and deployed.

demon reopened this task as Open.Dec 1 2016, 9:20 PM

Can we please get 3.4.1-1 built and uploaded? Thanks! Changelog:

scap (3.4.1-1) unstable; urgency=low
  * "scap deploy" no longer reports local commits when logging, now relies on
    last common ancestor with upstream. Prevents leaking security patches and
    instead reports public info.

 -- Chad Horohoe <chad@wikimedia.org>  Thu, 1 Dec 2016 21:03:19 -0000
fgiunchedi closed this task as Resolved.Dec 1 2016, 10:10 PM

@demon yep that's done (built+uploaded)

thcipriani reopened this task as Open.Dec 14 2016, 6:31 PM

I have a small bugfix release 3.4.2-1 that (aside from the necessary debian/packaging wrangling) only contains the fix for T152390: scap sync-l10n AttributeError: 'Namespace' object has no attribute 'message'

@fgiunchedi can you update the package on carbon?

/me makes puppet patch.

Change 327256 had a related patch set uploaded (by Thcipriani):
Bump scap version to 3.4.2-1

https://gerrit.wikimedia.org/r/327256

Change 327256 merged by Filippo Giunchedi:
Bump scap version to 3.4.2-1

https://gerrit.wikimedia.org/r/327256

Hiya @fgiunchedi -- I've tagged a debian/3.5.0-1 that is ready for release (puppet patch coming shortly). We've switched to using the release branch in the scap repo. The head of that branch should be where the new release tag is. All the gbp config should work for that branch as it has in the past -- let me know if you find anything configured weirdly.

Here is the full changelog for the new version (it's a biggy):

1​scap (3.5.0-1) unstable; urgency=low
2
3​ * MediaWiki: Old stub entry points for scap (e.g., sync-file, sync-dir,
4​ mwversionsinuse, etc) are gone. Formerly old binstubs simply exited with a
5​ non-zero exit code. Subcommands are now the only way to scap.
6
7​ * MediaWiki: the canary logstash check for MediaWiki deploys now supports an
8​ explicit service name (via the `canary_service` configuration variable).
9​ This will allow us to monitor HHVM errors as well as MediaWiki errors on
10​ the canary hosts before a deployment. (Fixes T154646, T142784)
11
12​ * Scap3: Scap's rollback behavior has been greatly improved. Scap supports a
13​ global `failure_limit` and a per-group `failure_limit` -- if a deployment
14​ exceeds the number or percentage of failures specified by this limit a
15​ deploy will fail and you will be prompted to rollback. Also, if you opt to
16​ *not* continue a deployment on remaining deploy groups, you will receive
17​ the option to rollback. (Fixes T149008)
18
19​ * Scap3: This scap release has some rollback logic fixes. First, if there is
20​ initial ssh failure for a host, scap will no longer attempt a rollback on
21​ that host (since the same ssh failure will likely cause a rollback
22​ failure). Next, all previously deployed groups of servers will now be
23​ rolled-back -- not just the group of servers that had failures. (Fixes
24​ T150267 T145460)
25
26​ * MediaWiki: /srv/mediawiki is now a flattened git directory. This is the
27​ first step towards moving MediaWiki deployments to Scap3. This is not a
28​ change for deployers, but is a needed internal change.
29
30​ * MediaWiki: `scap sync-file` and `scap sync-dir` are the same command
31​ internally. `scap sync-dir` is deprecated.
32
33​ * MediaWiki: scap now checks if proxies and canaries are listed among the
34​ pooled servers (in mediawiki-installation) before attempting to deploy to
35​ those hosts.
36
37​ * MediaWiki/Scap3: A fancier progress bar is available. Simply set
38​ `fancy_progress: True` in your config file to use it.
39
40​ * MediaWiki: The `scap l10n-purge` subcommand is no more. Removed as it was
41​ largely unused and was broken without notice for quite some time.
42
43​ * Scap3: Old scap-created tags will now be removed from the deployment host
44​ as part of the deployment. The number of tags to keep is controlled by the
45​ `tags_to_keep` configuration variable. By default, scap keeps 20 tags.
46
47​ * Scap3: in a shameless attempt to promote the use of messages for the
48​ Server Admin Log for non-MediaWiki deployments the default Server Admin
49​ Log message has been changed from "(no message)" to "(no justification
50​ provided)"
51
52​ * Internally, scap now uses wildly experimental Docker tests for CI and
53​ local testing. All tests are now being run by CI (which was not true in
54​ the past).
55
56​ * Scap is moving closer to supporting python3, small changes in this release
57​ bigger changes are yet to come.
58
59​ * Scap's scap say command is now more compatible with cowsay(6). It now
60​ supports passing messages on stdin as well as the `--eyes` flag.
61
62​ -- Tyler Cipriani <tcipriani@wikimedia.org> Wed, 25 Jan 2017 15:21:06 -0700

Change 334677 had a related patch set uploaded (by Thcipriani):
Scap: Bump version to 3.5.0-1

https://gerrit.wikimedia.org/r/334677

Mentioned in SAL (#wikimedia-operations) [2017-01-30T18:36:31Z] <godog> upload scap 3.5.0-1 - T127762

Change 334677 merged by Filippo Giunchedi:
Scap: Bump version to 3.5.1-1

https://gerrit.wikimedia.org/r/334677

@mobrovac ^ fyi, new scap package is tagged

Change 338138 had a related patch set uploaded (by Filippo Giunchedi):
scap: upgrade to 3.5.2-1

https://gerrit.wikimedia.org/r/338138

Change 338138 merged by Filippo Giunchedi:
scap: upgrade to 3.5.2-1

https://gerrit.wikimedia.org/r/338138

fgiunchedi closed this task as Resolved.Feb 17 2017, 8:48 AM
thcipriani reopened this task as Open.Feb 27 2017, 7:04 PM

Hiya @fgiunchedi I just tagged a new Scap version (3.5.3-1) this morning rMSCA7a2395a559ee: Bump debian version 3.5.3-1 — could I get you to rebuild the scap package?

Puppet patch for the update incoming.

Change 340159 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
Scap: update version to 3.5.3-1

https://gerrit.wikimedia.org/r/340159

Change 340159 merged by Filippo Giunchedi:
Scap: update version to 3.5.3-1

https://gerrit.wikimedia.org/r/340159

fgiunchedi closed this task as Resolved.Mar 2 2017, 2:57 PM

Package is out

thcipriani reopened this task as Open.Apr 5 2017, 5:54 PM

Just tagged scap debian/3.5.4-1: rMSCAf78caf877875: Update changelog for Debian version 3.5.4-1

@fgiunchedi could you update the package on carbon, please?

Change 346579 had a related patch set uploaded (by Thcipriani):
[operations/puppet@production] Scap: update version to 3.5.4-1

https://gerrit.wikimedia.org/r/346579

Just tagged scap debian/3.5.4-1: rMSCAf78caf877875: Update changelog for Debian version 3.5.4-1

@fgiunchedi could you update the package on carbon, please?

@fgiunchedi I tagged and pushed debian/3.5.5-1 that has the additional change of merging the global scap.cfg with more environment-specific scap.cfg rather than completely overriding. Could you update to 3.5.5 on carbon? I updated the puppet patch to reflect this change.

Mentioned in SAL (#wikimedia-operations) [2017-04-11T15:19:04Z] <godog> upload scap 3.5.5-1 - T127762

Change 346579 merged by Filippo Giunchedi:
[operations/puppet@production] Scap: update version to 3.5.5-1

https://gerrit.wikimedia.org/r/346579

fgiunchedi closed this task as Resolved.Apr 11 2017, 3:27 PM
thcipriani reopened this task as Open.Apr 24 2017, 11:34 PM

@fgiunchedi I tagged and pushed debian/3.5.6-1. Could you update to 3.5.6-1 on carbon? Puppet patch Soon™.

Thanks!

@thcipriani package built and updated in reprepro

Hi @fgiunchedi I just tagged and pushed debian/3.5.7-1. It has a fix for T163671: LocalisationUpdate not working since 2017-04-11. Could you update scap on carbon please. Puppet patch coming shortly.

Change 350757 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[operations/puppet@production] Scap: update version to 3.5.7-1

https://gerrit.wikimedia.org/r/350757

Change 350757 merged by Filippo Giunchedi:
[operations/puppet@production] Scap: update version to 3.5.7-1

https://gerrit.wikimedia.org/r/350757

fgiunchedi closed this task as Resolved.Apr 28 2017, 1:53 PM

@thcipriani yep, all done!

thcipriani reopened this task as Open.Jun 5 2017, 7:24 PM

Hi @fgiunchedi just tagged debian/3.5.8-1. Could you update scap on carbon please? And thank you :)

Change 357239 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[operations/puppet@production] Scap: Bump version to 3.5.8-1

https://gerrit.wikimedia.org/r/357239

Change 357239 merged by Filippo Giunchedi:
[operations/puppet@production] Scap: Bump version to 3.5.8-1

https://gerrit.wikimedia.org/r/357239

Mentioned in SAL (#wikimedia-operations) [2017-06-06T12:51:25Z] <godog> upgrade scap to 3.5.8 - T127762

fgiunchedi closed this task as Resolved.Jun 7 2017, 8:12 AM
thcipriani reopened this task as Open.Jul 25 2017, 8:17 PM

Hi @fgiunchedi just tagged debian/3.6.0-1 on scap's release branch. Could you update scap on carbon please?

Thanks!

Change 367749 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[operations/puppet@production] Scap: bump version to 3.6.0-1

https://gerrit.wikimedia.org/r/367749

Mentioned in SAL (#wikimedia-operations) [2017-07-27T08:21:34Z] <godog> upload scap 3.6.0-1 - T127762

Change 367749 merged by Filippo Giunchedi:
[operations/puppet@production] Scap: bump version to 3.6.0-1

https://gerrit.wikimedia.org/r/367749

fgiunchedi closed this task as Resolved.Jul 27 2017, 8:29 AM

@thcipriani all done!

thcipriani reopened this task as Open.Aug 31 2017, 5:13 PM

Pong :)

Hi @fgiunchedi just tagged debian/3.7.0-1 on scap's release branch. Could you update scap on carbon please? Puppet patch on the way!

Thanks!

Change 375029 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[operations/puppet@production] scap: upgrade to 3.7.0-1

https://gerrit.wikimedia.org/r/375029

Paladox added a subscriber: Paladox.Sep 1 2017, 6:32 PM

Mentioned in SAL (#wikimedia-operations) [2017-09-04T08:33:39Z] <godog> upload scap 3.7.0-1 to apt.w.o - T127762

Change 375029 merged by Filippo Giunchedi:
[operations/puppet@production] scap: upgrade to 3.7.0-1

https://gerrit.wikimedia.org/r/375029

fgiunchedi closed this task as Resolved.Sep 4 2017, 8:40 AM

@thcipriani all done!

thcipriani reopened this task as Open.Oct 12 2017, 4:43 PM

Hiya @fgiunchedi could I trouble you to update the scap package to version 3.7.1-1? It's tagged in the repo and only contains a packaging fix for T178039

Puppet patch incoming.

Change 383879 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[operations/puppet@production] scap: upgrade to 3.7.1-1

https://gerrit.wikimedia.org/r/383879

Change 383879 merged by Filippo Giunchedi:
[operations/puppet@production] scap: upgrade to 3.7.1-1

https://gerrit.wikimedia.org/r/383879

Mentioned in SAL (#wikimedia-operations) [2017-10-13T12:57:29Z] <godog> upload scap 3.7.1-1 - T127762

fgiunchedi closed this task as Resolved.Oct 13 2017, 1:02 PM

Yup all done @thcipriani !

New scap version 3.7.2-1 has been tagged. @fgiunchedi: Care to update the package, at your convenience? Puppet patch is https://gerrit.wikimedia.org/r/390355

@mmodell yup! I'll run the upgrade first thing next week

Mentioned in SAL (#wikimedia-operations) [2017-11-14T10:07:27Z] <godog> upload scap 3.7.2-1 - T127762

fgiunchedi closed this task as Resolved.Nov 14 2017, 10:19 AM

Merged!

Change 391219 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[operations/puppet@production] scap: upgrade to 3.7.3-1

https://gerrit.wikimedia.org/r/391219

Change 391219 merged by Filippo Giunchedi:
[operations/puppet@production] scap: upgrade to 3.7.3-1

https://gerrit.wikimedia.org/r/391219