Page MenuHomePhabricator

scap package installed by CI breaks apt and thus puppet
Open, NormalPublic

Description

Spotted on deployment-mediawiki04.deployment-prep.eqiad.wmflabs , puppet bails out because apt tries to downgrade scap from 3.7.4-3 to 3.7.4-1~20180103034049.266. The later comes from CI and has a smaller version than the version in apt.wikimedia.org/prod.

apt-cache policy scap
scap:
  Installed: 3.7.4-3
  Candidate: 3.7.4-1~20180103034049.266
  Version table:
 *** 3.7.4-3 0
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/main amd64 Packages
        100 /var/lib/dpkg/status
     3.7.4-1~20180103034049.266 0
       1500 http://deployment-tin.deployment-prep.eqiad.wmflabs/repo/ jessie-deployment-prep/main amd64 Packages
Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install scap' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  freeipmi-common libfreeipmi16 libipmiconsole2 libipmidetect0 libnfsidmap2
  libpgm-5.1-0 libsodium13 libzmq3 linux-image-4.4.0-3-amd64 python-dateutil
  python-m2crypto python-msgpack python-semver python-zmq
Use 'apt-get autoremove' to remove them.
The following packages will be DOWNGRADED:
  scap
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 60 not upgraded.
Need to get 112 kB of archives.
After this operation, 3072 B of additional disk space will be used.
E: There are problems and -y was used without --force-yes
Error: /Stage[main]/Scap/Package[scap]/ensure: change from 3.7.4-3 to 3.7.4-1~20180103034049.266 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install scap' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  freeipmi-common libfreeipmi16 libipmiconsole2 libipmidetect0 libnfsidmap2
  libpgm-5.1-0 libsodium13 libzmq3 linux-image-4.4.0-3-amd64 python-dateutil
  python-m2crypto python-msgpack python-semver python-zmq
Use 'apt-get autoremove' to remove them.
The following packages will be DOWNGRADED:
  scap
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 60 not upgraded.
Need to get 112 kB of archives.
After this operation, 3072 B of additional disk space will be used.
E: There are problems and -y was used without --force-yes

So I guess the package generated by CI needs a version bump that match whatever is in production?

I think we talked about it on a Differential diff related to stripping ou the debian directory from the master branch. That is probably related.

Event Timeline

hashar created this task.Jan 3 2018, 8:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 3 2018, 8:10 PM
mmodell added a subscriber: mmodell.Jan 4 2018, 5:46 PM

indeed... our scap ci is now broken, it's not building the latest thing anymore.

greg triaged this task as High priority.Jan 9 2018, 9:04 PM
greg added a subscriber: greg.

This is border line UBN!, but it'd be UBN! only if it keeps spitting out non-working scap packages on Beta Cluster.

@greg: this isn't an issue currently, however, it does need to be made more robust.

greg added a comment.Jan 9 2018, 11:28 PM

ack, thanks

thcipriani closed this task as Resolved.Jan 10 2018, 4:54 PM
thcipriani claimed this task.

FWIW, I pushed https://phabricator.wikimedia.org/rMSCA58f5eca67156879a0999422c4662b1d1c13acd96 on the 3rd that should have fixed the cause of this immediate task. Overall beta deployment could use some rethinking, but this specific task is effectively resolved, I think.

thcipriani reopened this task as Open.Jan 12 2018, 4:47 PM

happening again but maybe with a different cause. Are beta packages being built from the release branch now?

Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install scap' returned 100: Reading package lists...                     
Building dependency tree...                                                                                                                                               
Reading state information...                                                                                                                                              
The following packages were automatically installed and are no longer required:                                                                                           
  linux-image-3.16.0-4-amd64 linux-image-4.9.0-0.bpo.4-amd64                                                                                                              
Use 'apt-get autoremove' to remove them.                                                                                                                                  
The following packages will be DOWNGRADED:                                                                                                                                
  scap                                                                                                                                                                    
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not upgraded.                                                                                              
Need to get 112 kB of archives.                                                                                                                                           
After this operation, 0 B of additional disk space will be used.                                                                                                          
E: There are problems and -y was used without --force-yes                                                                                                                 
Error: /Stage[main]/Scap/Package[scap]/ensure: change from 3.8.0-1~20180108201547.277 to 3.7.5-1+0~20180111222813.280~1.gbpbf2589 failed: Could not update: Execution of '
/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install scap' returned 100: Reading package lists...                                                            
Building dependency tree...                                                                                                                                               
Reading state information...                                                                                                                                              
The following packages were automatically installed and are no longer required:                                                                                           
  linux-image-3.16.0-4-amd64 linux-image-4.9.0-0.bpo.4-amd64                                                                                                              
Use 'apt-get autoremove' to remove them.                                                                                                                                  
The following packages will be DOWNGRADED:                                                                                                                                
  scap                                                                                                                                                                    
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not upgraded.                                                                                              
Need to get 112 kB of archives.                                                                                                                                           
After this operation, 0 B of additional disk space will be used.                                                                                                          
E: There are problems and -y was used without --force-yes
thcipriani removed thcipriani as the assignee of this task.Jan 12 2018, 4:47 PM

For whatever reason beta debs got built from the release branch? https://integration.wikimedia.org/ci/job/phabricator-jessie-debs/280/console

From that output:

+ set +x
HEAD is now at f8dc5a3... Support "Bug: T123" in commit messages
[detached HEAD bf25898] Support "Bug: T123" in commit messages
 Author: Mukunda Modell <mmodell@wikimedia.org>
 Date: Thu Jan 11 16:03:33 2018 -0600
 3 files changed, 6 insertions(+)

But that commit was made on the release branch:

scap (master u=) 1m 25s                                                                                                                                           [24/169]
(/^ヮ^)/*:・゚✧ git show f8dc5a3                                                                                                                                           
commit f8dc5a30060768406c727e5be7dfbb303cd20204 (tag: debian/3.7.5-1, origin/release)                                                                                     
Author: Mukunda Modell <mmodell@wikimedia.org>                                                                                                                            
Date:   Thu Jan 11 16:03:33 2018 -0600                                                                                                                                    
                                                                                                                                                                          
    Support "Bug: T123" in commit messages                                                                                                                                
                                                                                                                                                                          
    This is a feature of `gbp dch`, added appropriate regex to gbp.conf                                                                                                   
                                                                                                                                                                          
diff --git a/debian/gbp.conf b/debian/gbp.conf                                                                                                                            
index 4fedd54..19b6df0 100644                                                                                                                                             
--- a/debian/gbp.conf                                                                                                                                                     
+++ b/debian/gbp.conf                                                                                                                                                     
@@ -2,3 +2,7 @@                                                                                                                                                           
 upstream-tag = %(version)s                                                                                                                                               
 upstream-branch = release                                                                                                                                                
 debian-branch = release                                                                                                                                                  
+                                                                                                                                                                         
+[dch]                                                                                                                                                                    
+meta-closes-bugnum = (?:bug)?\s*T\d+                                                                                                                                     
+meta-closes = Bug                                                                                                                                                        
:...skipping...

The branch for which packages are built is controlled by the Autoclose setting in the repo - here the list includes master and release.

thcipriani lowered the priority of this task from High to Normal.Jan 12 2018, 5:18 PM

Changing priority because this is currently "fixed" because I pushed a new commit onto scap's master branch.

Lowering priority instead of closing this time because this will keep breaking until we change the way scap's package is built in beta. This will be tracked in T184628 so I'll make that task this task's parent.

thcipriani added a comment.EditedJan 12 2018, 5:37 PM

For reference, this is why this is happening:

  • The generic job phabricator-jessie-commit is triggered for every branch that is in autoclose (as mentioned in T184118#3897171) for any repo in differential tagged ci-meta-jessie (scap is such a repo)
  • phabricator-jessie-debs is triggered by phabricator-jessie-commit which builds debs for scap and pushes them out on beta
  • For beta, currently, this means when a commit is pushed to the release branch or the master branch, a new package is built and pushed out on beta. Because the version numbers are correctly different between the master and release branch, anytime a new commit is pushed to release these jobs push out a version of scap to beta with a lower version number, causing apt to think we're trying to downgrade, which causes puppet (this task) to freak out.

minor edit "autoclase -> autoclose"

Aside: It would be really swell if puppet wouldn't freak out and break in this situation.

mmodell moved this task from Needs triage to Debt on the Scap board.Feb 1 2018, 12:21 AM