Update Debian Package for Scap3
Closed, ResolvedPublic

Description

A few bug fixes came out of the AQS deployment. Would like to get them packaged up and deployed on the servers. Newest deployment tag at: https://phabricator.wikimedia.org/diffusion/MSCA/browse/master/;3.0.2

There are a very large number of changes, so older changes are hidden. Show Older Changes
dduvall moved this task from Needs triage to Adoption on the Scap board.Feb 22 2016, 11:27 PM

I've uploaded scap 3.0.2-1 to carbon

This time was trickier than usual because master isn't at 3.0.2 afaics?

anyway to convince git-buildpackage to DTRT in this case you have to be in a branch so I git checkout -b debian 3.0.2

also going forward we should make more obvious what's the process to bump versions, ATM setup.py version is never changed, IOW we're assuming scap3 will be only distributed via debian packages.

fgiunchedi closed this task as "Resolved".Feb 23 2016, 11:45 AM
hashar added a subscriber: hashar.Feb 23 2016, 12:38 PM
* 91cdbe4 - (HEAD -> master, origin/master, origin/HEAD) Add coverage for current link replacement (Mon Feb 22 17:13:56 2016 -0800) <Dan Duvall>
| * e786a9b - (tag: 3.0.2) Update changelog for bugfix package bump (Mon Feb 22 13:25:00 2016 -0800) <Tyler Cipriani>
|/  
* 2332ad0 - Treat linked dirs as link target as normal file (Mon Feb 22 12:49:16 2016 -0800) <Tyler Cipriani>
* 603ef0c - Only sudo when actually necessary (Mon Feb 22 11:54:54 2016 -0800) <Dan Duvall>
* a593285 - Compare cdb mtimes less granularly (Fri Feb 19 14:07:32 2016 -0800) <Tyler Cipriani>
* 2a2a5b5 - Default to the effective user in scap.Context (Mon Feb 15 10:50:56 2016 -0800) <Dan Duvall>
* 8289e30 - (tag: debian/3.0.1-1) I incorrectly tagged 3.0.1, updating changelog again (Mon Feb 15 09:12:47 2016 -0600) <Mukunda Modell>
* 31727c0 - updated changelog. (Mon Feb 15 06:03:47 2016 -0600) <Mukunda Modell>
* c3d4de8 - (tag: 3.0.1) Add refreshCdbJsonFiles to the scap.deb package (Fri Feb 12 12:10:42 2016 -0600) <Mukunda Modell>
* 2d4b633 - Socket has no fqdn method (Thu Feb 11 14:42:31 2016 -0800) <Tyler Cipriani>
* d836f57 - Canonicalize host names when excluding hosts (Thu Feb 11 12:52:29 2016 -0800) <Dan Duvall>
* 459247b - (tag: debian/3.0-1, tag: 3.0) Remove `git_repo_user` configuration (Fri Jan 29 10:35:23 2016 -0800) <Dan Duvall>

Yup 3.0.2 tag is a fork of master apparently.

Looking at scap, it lacks a configuration file for git-buildpackage debian/gbp.conf so it will end up using the master for the Debian branch but build the tarball from the tag. Note how there is a debian/3.0.1-1 which is potentially recognized by gbp out of the box. It has:

--git-debian-tag=DEBIAN_TAG
                    Format string for debian tags, default is
                    'debian/%(version)s'
thcipriani reopened this task as "Open".Feb 29 2016, 11:29 PM

@fgiunchedi could I bug you to do this again for me? Had problem that's having some affect in beta cluster (see T128414).

New tag should be at 3.0.3:

scap master[git]✔ 14s
(ノ^ヮ^)ノ*:・゚✧ git lol
* 697f198 (HEAD -> master, tag: 3.0.3, origin/master, origin/HEAD) Update changelog for bugfix package bump
fgiunchedi closed this task as "Resolved".Mar 2 2016, 3:57 PM

@thcipriani I've uploaded 3.0.3-1 with a couple of minor changes to debian/changelog

root@carbon:~# reprepro list jessie-wikimedia scap
jessie-wikimedia|main|amd64: scap 3.0.3-1
jessie-wikimedia|main|i386: scap 3.0.3-1
jessie-wikimedia|main|source: scap 3.0.3-1
root@carbon:~# reprepro list trusty-wikimedia scap
trusty-wikimedia|main|amd64: scap 3.0.3-1
trusty-wikimedia|main|i386: scap 3.0.3-1
trusty-wikimedia|main|source: scap 3.0.3-1
thcipriani reopened this task as "Open".Jul 29 2016, 11:55 PM

@fgiunchedi could I get you to update/upload to v.3.2.1-1?

I created tags for the upstream version (3.2.1) and the debian version with the changelog (debian/3.2.1-1)—lemme know if there are any issues.

Thanks!

thcipriani lowered the priority of this task from "High" to "Normal".Jul 29 2016, 11:58 PM

Mentioned in SAL [2016-08-02T09:33:03Z] <godog> upload scap 3.2.1-1 to carbon T127762

Mentioned in SAL [2016-08-02T09:39:08Z] <godog> upload scap 3.2.1-1 to carbon T127762 (at 9:33)

fgiunchedi closed this task as "Resolved".Aug 2 2016, 10:41 AM

this is completed!

thcipriani reopened this task as "Open".Aug 2 2016, 9:43 PM

Thanks for the upload @fgiunchedi

Bug was discovered on Beta this morning :(

I fixed and bumped the minor version—new version on Beta seems to have fixed the issue.

Could I get you to build/upload: debian/3.2.2-1?

Mentioned in SAL [2016-08-03T08:43:13Z] <godog> upload scap 3.2.2-1 to carbon T127762

I fixed and bumped the minor version—new version on Beta seems to have fixed the issue.

Could I get you to build/upload: debian/3.2.2-1?

for sure, that's done!

Hi @fgiunchedi

I have a bugfix release for scap. Tagged debian/3.2.3-1 and pushed up to the repo. This one should fix T142792: Failed to rollback scap3 deployment and T142364: TypeError: unsupported operand type(s) for %: 'dict' and 'tuple'

Thanks for all your help!

Mentioned in SAL [2016-08-17T09:45:28Z] <godog> upload scap3 3.2.3-1 to carbon T127762

@thcipriani no worries! I've uploaded the new version now

hashar closed this task as "Resolved".Aug 25 2016, 10:05 AM

tin / mira and a few random mw servers have scap 3.2.3-1 so looks like it is fixed.

thcipriani reopened this task as "Open".Aug 26 2016, 11:38 PM

Hiya @fgiunchedi could I get you to upload scap_3.2.4-1 to carbon?

This has the fix for T142990: Sequential execution should be per-deployment, not per-phase which unblocks Parsoid(!)

fgiunchedi closed this task as "Resolved".Aug 29 2016, 8:36 AM

@thcipriani for sure! package is built/uploaded

Change 309635 had a related patch set uploaded (by Thcipriani):
Bump scap version to 3.2.5-1

https://gerrit.wikimedia.org/r/309635

thcipriani reopened this task as "Open".Sep 9 2016, 11:51 PM

@fgiunchedi could I get you to upload scap_3.2.5-1 to carbon?

New release should fix up T145194: Scap fails to force-deploy the config, T144244: Tab completion doesn't work well for directories, and add a few other backwards-compaitble compatible niceties :)

fgiunchedi closed this task as "Resolved".Sep 12 2016, 8:55 AM

@thcipriani yup! I've updated the package on carbon to 3.2.5-1

Change 309635 merged by Giuseppe Lavagetto:
Bump scap version to 3.2.5-1

https://gerrit.wikimedia.org/r/309635

thcipriani reopened this task as "Open".Sep 23 2016, 12:41 AM
thcipriani added a subscriber: mobrovac.

Change 312480 had a related patch set uploaded (by Filippo Giunchedi):
scap: update to 3.3.0-1

https://gerrit.wikimedia.org/r/312480

Change 312480 merged by Filippo Giunchedi:
scap: update to 3.3.0-1

https://gerrit.wikimedia.org/r/312480

fgiunchedi closed this task as "Resolved".Sep 23 2016, 3:16 PM

scap uploaded and deployed, resolving

thcipriani reopened this task as "Open".Nov 13 2016, 11:48 PM

@fgiunchedi could you update the scap packages to v.3.3.1-1?

Changelog:

scap (3.3.1-1) unstable; urgency=low

  * Scap learned how to lock all deployments at the server level. Previously
    it only knew how to lock deployments at the repo level.

  * Scap now tells users who holds the lock file (when deploys are locked).
    Permissions on the lockfile have changed to 600, making it harder to
    overwrite (Fixes T140914).

  * "scap sync-file" and "scap pull" got new flags, "--beta-only-change" and
    "--no-touch", respectively. When used, this prevents the normal action
    taken by "scap pull" to touch the wmf-config/InitialiseSettings.php file
    which invalidate local caches.  This allows some syncs to avoid a
    potential problem with HHVM servers exhausting local cache when reading
    new files from disk. The option is now also always passed to `scap pull`
    by `scap sync-l10n` (fixes T149872 -- thank you Bryan Davis).

  * Scap learned to announce scap3 deployments in IRC. This should mean that
    deployers will no longer have to announce a deployment manually in
    *-operations -- scap will do it for them.

  * Scap3 updated the way it limits hosts and can now limit hosts from all
    groups, not just the default group (Fixes T149128).

  * HHVM restarts now happen via the /usr/local/bin/restart-hhvm script. This
    means that an individual server is now depooled via confctl before being
    restarted (thank you Giuseppe Lavagetto).

  * Scap3 learned about empty checks in the checks.yml file. This allows you
    to overwrite global checks in an environment specific checks.yml. These
    checks will be logged, but nothing will run (fixes T149668).

  * Scap3 added a "finalize" stage of deployment. This will allow for "promote"
    checks to execute before the final state is recorded and clean up of old
    rev directories is performed, and overall result in more consistent
    rollback behavior (Fixes T150267)

  * Scap learned how to use sub-sub commands, i.e., scap subcommand
    subsubcommand. None are yet implemented.

  * scap l10n-purge works once again, and restart_hhvm was restored (Fixes
    T146656).

  * Internally, scap simplified its usage of sudo calls. This means the
    internals are a bit more sane, unnecessary sudo calls, i.e., sudoing as
    yourself is less common throughout the code.

 -- Tyler Cipriani <tcipriani@wikimedia.org>  Thu, 10 Nov 2016 13:41:28 -0800

Thanks!

Gerrit patch for puppet (since it didn't link) https://gerrit.wikimedia.org/r/#/c/321339,edit/

thcipriani changed the task status from "Open" to "Stalled".Nov 21 2016, 4:54 PM

Marking task as stalled. Want to get a new version with fix for T150897: Env vars being overwritten in it.

thcipriani changed the task status from "Stalled" to "Open".Nov 28 2016, 4:36 PM

@fgiunchedi could you update the scap packages to v.3.4.0-1?

Changelog:

1​scap (3.4.0-1) unstable; urgency=low
2
3​ * BREAKING CHANGE: Old scap bin stubs (e.g., sync-file, sync-dir,
4​ mwversionsinuse, etc) will now exit 1, subcommands are now the only way to
5​ interact with scap, i.e., sync-file is now scap sync-file.
6
7​ * Scap linter now ignores auto autoload_static.php which was known to fail
8​ lint checks (Fixes T136009 - thank you Paladox)
9
10​ * Scap adds the sha1 of the commit being deployed to the announce log --
11​ this change affects scap3 repos.
12
13​ * This release fixes a bug where variables specified in vars.yaml were being
14​ overwritten in the opposite order (Fixes T150897)
15
16​ * Scap lockfiles now contain the message passed in the deployment, i.e. scap
17​ sync-dir wmf-config 'Update config' will create a lockfile with the
18​ message 'Update config'. If anyone else attempts to deploy while that
19​ lockfile is in place, they will be shown the lockfile message.
20
21​ -- Tyler Cipriani <tcipriani@wikimedia.org> Mon, 28 Nov 2016 09:03:50 -0700
22
23​scap (3.3.1-1) unstable; urgency=low
24
25​ * Scap learned how to lock all deployments at the server level. Previously
26​ it only knew how to lock deployments at the repo level.
27
28​ * Scap now tells users who holds the lock file (when deploys are locked).
29​ Permissions on the lockfile have changed to 600, making it harder to
30​ overwrite (Fixes T140914).
31
32​ * "scap sync-file" and "scap pull" got new flags, "--beta-only-change" and
33​ "--no-touch", respectively. When used, this prevents the normal action
34​ taken by "scap pull" to touch the wmf-config/InitialiseSettings.php file
35​ which invalidate local caches. This allows some syncs to avoid a
36​ potential problem with HHVM servers exhausting local cache when reading
37​ new files from disk. The option is now also always passed to `scap pull`
38​ by `scap sync-l10n` (fixes T149872 -- thank you Bryan Davis).
39
40​ * Scap learned to announce scap3 deployments in IRC. This should mean that
41​ deployers will no longer have to announce a deployment manually in
42​ *-operations -- scap will do it for them.
43
44​ * Scap3 updated the way it limits hosts and can now limit hosts from all
45​ groups, not just the default group (Fixes T149128).
46
47​ * HHVM restarts now happen via the /usr/local/bin/restart-hhvm script. This
48​ means that an individual server is now depooled via confctl before being
49​ restarted (thank you Giuseppe Lavagetto).
50
51​ * Scap3 learned about empty checks in the checks.yml file. This allows you
52​ to overwrite global checks in an environment specific checks.yml. These
53​ checks will be logged, but nothing will run (fixes T149668).
54
55​ * Scap3 added a "finalize" stage of deployment. This will allow for "promote"
56​ checks to execute before the final state is recorded and clean up of old
57​ rev directories is performed, and overall result in more consistent
58​ rollback behavior (Fixes T150267)
59
60​ * Scap learned how to use sub-sub commands, i.e., scap subcommand
61​ subsubcommand. None are yet implemented.
62
63​ * scap l10n-purge works once again, and restart_hhvm was restored (Fixes
64​ T146656).
65
66​ * Internally, scap simplified its usage of sudo calls. This means the
67​ internals are a bit more sane, unnecessary sudo calls, i.e., sudoing as
68​ yourself is less common throughout the code.
69
70​ -- Tyler Cipriani <tcipriani@wikimedia.org> Thu, 10 Nov 2016 13:41:28 -0800

  • Thanks!

Change 323852 had a related patch set uploaded (by Filippo Giunchedi):
scap: bump version to 3.4.0-1

https://gerrit.wikimedia.org/r/323852

Change 323852 merged by Filippo Giunchedi:
scap: bump version to 3.4.0-1

https://gerrit.wikimedia.org/r/323852

fgiunchedi closed this task as "Resolved".Nov 29 2016, 1:02 AM

Package built and deployed.

demon reopened this task as "Open".Dec 1 2016, 9:20 PM

Can we please get 3.4.1-1 built and uploaded? Thanks! Changelog:

scap (3.4.1-1) unstable; urgency=low
  * "scap deploy" no longer reports local commits when logging, now relies on
    last common ancestor with upstream. Prevents leaking security patches and
    instead reports public info.

 -- Chad Horohoe <chad@wikimedia.org>  Thu, 1 Dec 2016 21:03:19 -0000
fgiunchedi closed this task as "Resolved".Dec 1 2016, 10:10 PM

@demon yep that's done (built+uploaded)

thcipriani reopened this task as "Open".Dec 14 2016, 6:31 PM

I have a small bugfix release 3.4.2-1 that (aside from the necessary debian/packaging wrangling) only contains the fix for T152390: scap sync-l10n AttributeError: 'Namespace' object has no attribute 'message'

@fgiunchedi can you update the package on carbon?

/me makes puppet patch.

Change 327256 had a related patch set uploaded (by Thcipriani):
Bump scap version to 3.4.2-1

https://gerrit.wikimedia.org/r/327256

Change 327256 merged by Filippo Giunchedi:
Bump scap version to 3.4.2-1

https://gerrit.wikimedia.org/r/327256

Hiya @fgiunchedi -- I've tagged a debian/3.5.0-1 that is ready for release (puppet patch coming shortly). We've switched to using the release branch in the scap repo. The head of that branch should be where the new release tag is. All the gbp config should work for that branch as it has in the past -- let me know if you find anything configured weirdly.

Here is the full changelog for the new version (it's a biggy):

1​scap (3.5.0-1) unstable; urgency=low
2
3​ * MediaWiki: Old stub entry points for scap (e.g., sync-file, sync-dir,
4​ mwversionsinuse, etc) are gone. Formerly old binstubs simply exited with a
5​ non-zero exit code. Subcommands are now the only way to scap.
6
7​ * MediaWiki: the canary logstash check for MediaWiki deploys now supports an
8​ explicit service name (via the `canary_service` configuration variable).
9​ This will allow us to monitor HHVM errors as well as MediaWiki errors on
10​ the canary hosts before a deployment. (Fixes T154646, T142784)
11
12​ * Scap3: Scap's rollback behavior has been greatly improved. Scap supports a
13​ global `failure_limit` and a per-group `failure_limit` -- if a deployment
14​ exceeds the number or percentage of failures specified by this limit a
15​ deploy will fail and you will be prompted to rollback. Also, if you opt to
16​ *not* continue a deployment on remaining deploy groups, you will receive
17​ the option to rollback. (Fixes T149008)
18
19​ * Scap3: This scap release has some rollback logic fixes. First, if there is
20​ initial ssh failure for a host, scap will no longer attempt a rollback on
21​ that host (since the same ssh failure will likely cause a rollback
22​ failure). Next, all previously deployed groups of servers will now be
23​ rolled-back -- not just the group of servers that had failures. (Fixes
24​ T150267 T145460)
25
26​ * MediaWiki: /srv/mediawiki is now a flattened git directory. This is the
27​ first step towards moving MediaWiki deployments to Scap3. This is not a
28​ change for deployers, but is a needed internal change.
29
30​ * MediaWiki: `scap sync-file` and `scap sync-dir` are the same command
31​ internally. `scap sync-dir` is deprecated.
32
33​ * MediaWiki: scap now checks if proxies and canaries are listed among the
34​ pooled servers (in mediawiki-installation) before attempting to deploy to
35​ those hosts.
36
37​ * MediaWiki/Scap3: A fancier progress bar is available. Simply set
38​ `fancy_progress: True` in your config file to use it.
39
40​ * MediaWiki: The `scap l10n-purge` subcommand is no more. Removed as it was
41​ largely unused and was broken without notice for quite some time.
42
43​ * Scap3: Old scap-created tags will now be removed from the deployment host
44​ as part of the deployment. The number of tags to keep is controlled by the
45​ `tags_to_keep` configuration variable. By default, scap keeps 20 tags.
46
47​ * Scap3: in a shameless attempt to promote the use of messages for the
48​ Server Admin Log for non-MediaWiki deployments the default Server Admin
49​ Log message has been changed from "(no message)" to "(no justification
50​ provided)"
51
52​ * Internally, scap now uses wildly experimental Docker tests for CI and
53​ local testing. All tests are now being run by CI (which was not true in
54​ the past).
55
56​ * Scap is moving closer to supporting python3, small changes in this release
57​ bigger changes are yet to come.
58
59​ * Scap's scap say command is now more compatible with cowsay(6). It now
60​ supports passing messages on stdin as well as the `--eyes` flag.
61
62​ -- Tyler Cipriani <tcipriani@wikimedia.org> Wed, 25 Jan 2017 15:21:06 -0700

Change 334677 had a related patch set uploaded (by Thcipriani):
Scap: Bump version to 3.5.0-1

https://gerrit.wikimedia.org/r/334677

Mentioned in SAL (#wikimedia-operations) [2017-01-30T18:36:31Z] <godog> upload scap 3.5.0-1 - T127762

Change 334677 merged by Filippo Giunchedi:
Scap: Bump version to 3.5.1-1

https://gerrit.wikimedia.org/r/334677

@mobrovac ^ fyi, new scap package is tagged

Change 338138 had a related patch set uploaded (by Filippo Giunchedi):
scap: upgrade to 3.5.2-1

https://gerrit.wikimedia.org/r/338138

Change 338138 merged by Filippo Giunchedi:
scap: upgrade to 3.5.2-1

https://gerrit.wikimedia.org/r/338138

fgiunchedi closed this task as "Resolved".Feb 17 2017, 8:48 AM
thcipriani reopened this task as "Open".Feb 27 2017, 7:04 PM

Hiya @fgiunchedi I just tagged a new Scap version (3.5.3-1) this morning rMSCA7a2395a559ee: Bump debian version 3.5.3-1 — could I get you to rebuild the scap package?

Puppet patch for the update incoming.

Change 340159 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
Scap: update version to 3.5.3-1

https://gerrit.wikimedia.org/r/340159

Change 340159 merged by Filippo Giunchedi:
Scap: update version to 3.5.3-1

https://gerrit.wikimedia.org/r/340159

fgiunchedi closed this task as "Resolved".Mar 2 2017, 2:57 PM

Package is out

thcipriani reopened this task as "Open".Wed, Apr 5, 5:54 PM

Just tagged scap debian/3.5.4-1: rMSCAf78caf877875: Update changelog for Debian version 3.5.4-1

@fgiunchedi could you update the package on carbon, please?

Change 346579 had a related patch set uploaded (by Thcipriani):
[operations/puppet@production] Scap: update version to 3.5.4-1

https://gerrit.wikimedia.org/r/346579

Just tagged scap debian/3.5.4-1: rMSCAf78caf877875: Update changelog for Debian version 3.5.4-1

@fgiunchedi could you update the package on carbon, please?

@fgiunchedi I tagged and pushed debian/3.5.5-1 that has the additional change of merging the global scap.cfg with more environment-specific scap.cfg rather than completely overriding. Could you update to 3.5.5 on carbon? I updated the puppet patch to reflect this change.

Mentioned in SAL (#wikimedia-operations) [2017-04-11T15:19:04Z] <godog> upload scap 3.5.5-1 - T127762

Change 346579 merged by Filippo Giunchedi:
[operations/puppet@production] Scap: update version to 3.5.5-1

https://gerrit.wikimedia.org/r/346579

fgiunchedi closed this task as "Resolved".Tue, Apr 11, 3:27 PM