thcipriani (Tyler Cipriani)
Rappa Ternt RelEng

Projects (17)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
Feb 9 2015, 10:04 PM (153 w, 5 d)
Availability
Available
IRC Nick
thcipriani
LDAP User
Unknown
MediaWiki User
TCipriani (WMF)

Recent Activity

Thu, Jan 18

thcipriani added a comment to T185231: Can't insert templates in the visual and 2017 wikitext editor.

forgot to add the bug to the scap message so it'd show up here, but this is now deployed: https://tools.wmflabs.org/sal/log/AWEK1UGSkkJ8OkTwh3Dt

Thu, Jan 18, 7:55 PM · MW-1.31-release-notes (WMF-deploy-2018-01-16 (1.31.0-wmf.17)), Patch-For-Review, VisualEditor-MediaWiki-2017WikitextEditor, VisualEditor
thcipriani added a subtask for T183956: 1.31.0-wmf.17 deployment blockers: T185231: Can't insert templates in the visual and 2017 wikitext editor.
Thu, Jan 18, 5:35 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
thcipriani added a parent task for T185231: Can't insert templates in the visual and 2017 wikitext editor: T183956: 1.31.0-wmf.17 deployment blockers.
Thu, Jan 18, 5:35 PM · MW-1.31-release-notes (WMF-deploy-2018-01-16 (1.31.0-wmf.17)), Patch-For-Review, VisualEditor-MediaWiki-2017WikitextEditor, VisualEditor
thcipriani closed T182643: cache_git_info (from e.g. scap sync-file) is way way too verbose as Resolved.

Fix should be live now

Thu, Jan 18, 3:14 PM · Scap
Liuxinyu970226 awarded T180749: 1.31.0-wmf.16 deployment blockers a Barnstar token.
Thu, Jan 18, 4:15 AM · Release-Engineering-Team (Kanban), Release, Train Deployments

Wed, Jan 17

thcipriani triaged T183956: 1.31.0-wmf.17 deployment blockers as Normal priority.
Wed, Jan 17, 4:41 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
thcipriani claimed T183956: 1.31.0-wmf.17 deployment blockers.
Wed, Jan 17, 4:41 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
thcipriani closed T184749: Every edit (including rollback) distorts non-ASCII text as Resolved.

This fix was deployed to wmf.16 wikis yesterday and verified by @Addshore, closing.

Wed, Jan 17, 4:40 PM · MW-1.31-release-notes (WMF-deploy-2018-01-09 (1.31.0-wmf.16)), User-notice, User-Addshore, Patch-For-Review, Regression, Multi-Content-Revisions, MediaWiki-Page-editing
thcipriani closed T184749: Every edit (including rollback) distorts non-ASCII text, a subtask of T180749: 1.31.0-wmf.16 deployment blockers, as Resolved.
Wed, Jan 17, 4:40 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
thcipriani lowered the priority of T184882: Scap: TypeError: error: (not all arguments converted during string formatting); format string: (Passed unrecognized git_binary_manager {}); arguments: ((u'fat',)) from High to Low.

Leaving this task open for all the git_fat -> git_binary_manager changes I made, but since the new version of scap was released yesterday, that doesn't need to happen as desperately anymore.

Wed, Jan 17, 4:36 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Scap

Tue, Jan 16

mmodell awarded T180749: 1.31.0-wmf.16 deployment blockers a Barnstar token.
Tue, Jan 16, 10:02 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
thcipriani closed T180749: 1.31.0-wmf.16 deployment blockers as Resolved.

1.31.0-wmf.16 is now live everywhere.

Tue, Jan 16, 10:02 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
thcipriani updated the task description for T180749: 1.31.0-wmf.16 deployment blockers.
Tue, Jan 16, 10:01 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
thcipriani added a comment to T95839: CAS update failed on user_touched for user ID.

Noticed this happening in 1.31.0-wmf.16 just now.

Tue, Jan 16, 9:40 PM · Wikimedia-log-errors, Availability, MediaWiki-User-preferences
thcipriani added a comment to T185037: Fatal error: Class undefined: PageAssessmentsBody.

Full stack trace:

Tue, Jan 16, 9:20 PM · Community-Tech-Sprint, MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Patch-For-Review, Wikimedia-log-errors, MediaWiki-extensions-PageAssessments
thcipriani added a project to T185037: Fatal error: Class undefined: PageAssessmentsBody: Wikimedia-log-errors.
Tue, Jan 16, 9:17 PM · Community-Tech-Sprint, MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Patch-For-Review, Wikimedia-log-errors, MediaWiki-extensions-PageAssessments
thcipriani triaged T185037: Fatal error: Class undefined: PageAssessmentsBody as High priority.

This started when I pushed out wmf.16 to the wikipedia wikis.

Tue, Jan 16, 9:13 PM · Community-Tech-Sprint, MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Patch-For-Review, Wikimedia-log-errors, MediaWiki-extensions-PageAssessments
thcipriani created T185037: Fatal error: Class undefined: PageAssessmentsBody.
Tue, Jan 16, 9:07 PM · Community-Tech-Sprint, MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Patch-For-Review, Wikimedia-log-errors, MediaWiki-extensions-PageAssessments
thcipriani created T185024: HHVM 3.18.5+dfsg-1+wmf3 changes parse_url causing unit tests to fail.
Tue, Jan 16, 7:07 PM · Operations, Continuous-Integration-Infrastructure, HHVM
thcipriani added a comment to T184749: Every edit (including rollback) distorts non-ASCII text.

Change 403930 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/core@wmf/1.31.0-wmf.16] RevisionStore, fix loadSlotContent with no $blobFlags

https://gerrit.wikimedia.org/r/403930

Tue, Jan 16, 5:44 PM · MW-1.31-release-notes (WMF-deploy-2018-01-09 (1.31.0-wmf.16)), User-notice, User-Addshore, Patch-For-Review, Regression, Multi-Content-Revisions, MediaWiki-Page-editing
thcipriani added a comment to T180749: 1.31.0-wmf.16 deployment blockers.

Status update: The fix is merged into master (with a backport to wmf.16) and thus on the Beta Cluster for testing. Given it is a Friday we will attempt a continuation of the train on Monday (per https://wikitech.wikimedia.org/wiki/Deployments/Holding_the_train#What_happens_next?), to be coordinated with @mmodell / @thcipriani and @Addshore.

Tue, Jan 16, 5:08 PM · Release-Engineering-Team (Kanban), Release, Train Deployments

Mon, Jan 15

thcipriani committed rOSCW39dcc6e54014: Scap: git_fat -> git_binary_manager (authored by thcipriani).
Scap: git_fat -> git_binary_manager
Mon, Jan 15, 4:45 AM
thcipriani committed rOSPJc04052e63b6a: Scap: git_fat -> git_binary_manager (authored by thcipriani).
Scap: git_fat -> git_binary_manager
Mon, Jan 15, 3:50 AM

Sun, Jan 14

thcipriani lowered the priority of T184882: Scap: TypeError: error: (not all arguments converted during string formatting); format string: (Passed unrecognized git_binary_manager {}); arguments: ((u'fat',)) from Unbreak Now! to High.

Current status is that repo owners depending on git_fat either have to wait for scap 3.7.6-1 to be released, or merge the patch I made against their repository from this ticket in order to deploy.

Sun, Jan 14, 11:52 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Scap
thcipriani committed rMSCAbc3d2eea42c6: Workaround for T184118 (authored by thcipriani).
Workaround for T184118
Sun, Jan 14, 10:25 PM
thcipriani triaged T184882: Scap: TypeError: error: (not all arguments converted during string formatting); format string: (Passed unrecognized git_binary_manager {}); arguments: ((u'fat',)) as Unbreak Now! priority.

This error is currently in scap version 3.7.5-1 that was released to production on Friday and that means that any repo currently using the:

Sun, Jan 14, 10:11 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Scap
thcipriani committed rMSCA0cfb201e5eda: Bumping Debian version to 3.7.6-1 (authored by thcipriani).
Bumping Debian version to 3.7.6-1
Sun, Jan 14, 10:04 PM
thcipriani committed rMSCAb716e877e08a: Bump version to 3.7.6 (authored by thcipriani).
Bump version to 3.7.6
Sun, Jan 14, 10:04 PM
thcipriani committed rMSCAee80537fac6c: Merge branch 'master' into release (authored by thcipriani).
Merge branch 'master' into release
Sun, Jan 14, 10:04 PM
thcipriani reopened T127762: Update Debian Package for Scap3 as "Open".

Hello, I just pushed new scap tags for version 3.7.6/debian/3.7.6-1. This has the fix for T184882.

Sun, Jan 14, 10:04 PM · Patch-For-Review, Scap
thcipriani committed rMSCAc95c5c8282a4: Fix unrecognized git_binary_manager compatibility (authored by thcipriani).
Fix unrecognized git_binary_manager compatibility
Sun, Jan 14, 9:55 PM
thcipriani closed D945: Fix unrecognized git_binary_manager compatibility.
Sun, Jan 14, 9:55 PM · Release-Engineering-Team
thcipriani requested review of D945: Fix unrecognized git_binary_manager compatibility.
Sun, Jan 14, 9:44 PM · Release-Engineering-Team
thcipriani added a revision to T184882: Scap: TypeError: error: (not all arguments converted during string formatting); format string: (Passed unrecognized git_binary_manager {}); arguments: ((u'fat',)): D945: Fix unrecognized git_binary_manager compatibility.
Sun, Jan 14, 9:42 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Scap
thcipriani created T184882: Scap: TypeError: error: (not all arguments converted during string formatting); format string: (Passed unrecognized git_binary_manager {}); arguments: ((u'fat',)).
Sun, Jan 14, 9:41 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Scap
thcipriani added inline comments to D921: Add git-lfs support.
Sun, Jan 14, 7:37 PM · Release-Engineering-Team

Fri, Jan 12

thcipriani added inline comments to D921: Add git-lfs support.
Fri, Jan 12, 10:37 PM · Release-Engineering-Team
thcipriani added a comment to T184628: Rethink beta scap deployment.
  1. Things I like
Fri, Jan 12, 5:51 PM · Release-Engineering-Team (Kanban), Scap
thcipriani added a comment to T184118: scap package installed by CI breaks apt and thus puppet.

For reference, this is why this is happening:

Fri, Jan 12, 5:37 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure, Scap
thcipriani triaged T184628: Rethink beta scap deployment as Normal priority.
Fri, Jan 12, 5:22 PM · Release-Engineering-Team (Kanban), Scap
thcipriani added a comment to T184628: Rethink beta scap deployment.

Looking at the phabricator-jessie-commits job on T184118#3897095 I had to refresh my memory on what was happening.

Fri, Jan 12, 5:22 PM · Release-Engineering-Team (Kanban), Scap
thcipriani added a parent task for T184118: scap package installed by CI breaks apt and thus puppet: T184628: Rethink beta scap deployment.
Fri, Jan 12, 5:18 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure, Scap
thcipriani added a subtask for T184628: Rethink beta scap deployment: T184118: scap package installed by CI breaks apt and thus puppet.
Fri, Jan 12, 5:18 PM · Release-Engineering-Team (Kanban), Scap
thcipriani lowered the priority of T184118: scap package installed by CI breaks apt and thus puppet from High to Normal.

Changing priority because this is currently "fixed" because I pushed a new commit onto scap's master branch.

Fri, Jan 12, 5:18 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure, Scap
thcipriani committed rMSCA85a30ba4d447: Update README to point to correct phab project (authored by thcipriani).
Update README to point to correct phab project
Fri, Jan 12, 5:03 PM
thcipriani added a comment to T184118: scap package installed by CI breaks apt and thus puppet.

For whatever reason beta debs got built from the release branch? https://integration.wikimedia.org/ci/job/phabricator-jessie-debs/280/console

Fri, Jan 12, 4:52 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure, Scap
thcipriani placed T184118: scap package installed by CI breaks apt and thus puppet up for grabs.
Fri, Jan 12, 4:47 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure, Scap
thcipriani reopened T184118: scap package installed by CI breaks apt and thus puppet as "Open".

happening again but maybe with a different cause. Are beta packages being built from the release branch now?

Fri, Jan 12, 4:47 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure, Scap
thcipriani closed T184781: Docker won't start on integration-slave-docker-1005 as Resolved.

to horizon project puppet for the integration project and docker is running again. Bringing intergration-slave-docker-1005 back online in jenkins and calling this done.

Fri, Jan 12, 1:48 AM · Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure
thcipriani added a comment to T184781: Docker won't start on integration-slave-docker-1005.

I think I found the problem:

Fri, Jan 12, 1:43 AM · Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure
thcipriani added a comment to T184781: Docker won't start on integration-slave-docker-1005.

I disconnected this machine in jenkins for now.

Fri, Jan 12, 1:18 AM · Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure
thcipriani created T184781: Docker won't start on integration-slave-docker-1005.
Fri, Jan 12, 1:16 AM · Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure

Thu, Jan 11

thcipriani claimed T183999: Scap canary has a shifting baseline.
Thu, Jan 11, 1:33 AM · Patch-For-Review, Release-Engineering-Team (Kanban), Scap

Wed, Jan 10

thcipriani added inline comments to D921: Add git-lfs support.
Wed, Jan 10, 5:24 PM · Release-Engineering-Team
thcipriani created T184628: Rethink beta scap deployment.
Wed, Jan 10, 5:11 PM · Release-Engineering-Team (Kanban), Scap
thcipriani closed T184118: scap package installed by CI breaks apt and thus puppet as Resolved.

FWIW, I pushed https://phabricator.wikimedia.org/rMSCA58f5eca67156879a0999422c4662b1d1c13acd96 on the 3rd that should have fixed the cause of this immediate task. Overall beta deployment could use some rethinking, but this specific task is effectively resolved, I think.

Wed, Jan 10, 4:56 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure, Scap
thcipriani added a task to rMSCA58f5eca67156: Bump development version to 3.8.0: T184118: scap package installed by CI breaks apt and thus puppet.
Wed, Jan 10, 4:54 PM
thcipriani added a commit to T184118: scap package installed by CI breaks apt and thus puppet: rMSCA58f5eca67156: Bump development version to 3.8.0.
Wed, Jan 10, 4:54 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure, Scap

Mon, Jan 8

thcipriani closed T183952: Investigate deployment that caused high error-rate and was prevented from going past canaries by Scap as Resolved.

Dug a little deeper on this today and realized that this didn't actually make it out to production afaict (thanks to @zeljkofilipin), although the error rate on the canaries was high enough to climb to spike the overall error rate. The task I filed a few comments back is still a thing we need to do. I'm going to call this investigation complete though.

Mon, Jan 8, 11:07 PM · Release-Engineering-Team (Kanban), Wikimedia-Incident, Scap
thcipriani renamed T183952: Investigate deployment that caused high error-rate and was prevented from going past canaries by Scap from Investigate deployment that caused high error-rate but wasn't prevented by Scap to Investigate deployment that caused high error-rate and was prevented from going past canaries by Scap.
Mon, Jan 8, 11:04 PM · Release-Engineering-Team (Kanban), Wikimedia-Incident, Scap
thcipriani added a comment to T183952: Investigate deployment that caused high error-rate and was prevented from going past canaries by Scap.

Scap did fail during deployment. Since the commit that caused the failure was already merged, I had to revert it and deploy the revert. Is there something else I should have done?

Mon, Jan 8, 10:57 PM · Release-Engineering-Team (Kanban), Wikimedia-Incident, Scap
thcipriani triaged T184457: Installation method for Minikube on CI for k8s testing as Normal priority.
Mon, Jan 8, 4:47 PM · Release-Engineering-Team (Kanban), Release Pipeline
thcipriani added a parent task for T184457: Installation method for Minikube on CI for k8s testing: T183165: Verify functionality of the 'production' image in the context of an isolated k8s deployment.
Mon, Jan 8, 4:47 PM · Release-Engineering-Team (Kanban), Release Pipeline
thcipriani added a subtask for T183165: Verify functionality of the 'production' image in the context of an isolated k8s deployment: T184457: Installation method for Minikube on CI for k8s testing.
Mon, Jan 8, 4:47 PM · Release-Engineering-Team (Kanban), Release Pipeline
thcipriani created T184457: Installation method for Minikube on CI for k8s testing.
Mon, Jan 8, 4:46 PM · Release-Engineering-Team (Kanban), Release Pipeline
thcipriani moved T183165: Verify functionality of the 'production' image in the context of an isolated k8s deployment from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.
Mon, Jan 8, 4:43 PM · Release-Engineering-Team (Kanban), Release Pipeline

Fri, Jan 5

thcipriani added a comment to D943: Tag HEAD in the submodules not the commit hash from parent repo.
18:08:10 ['/vagrant/scap/bin/scap', 'deploy-local', '-v', '--repo', 'mockbase/deploy', '--force', '-g', 'default', 'fetch', '--refresh-config'] on scap-target-01 returne$
 [70]: OpenSSH_7.4p1 Debian-10+deb9u1, OpenSSL 1.0.2l  25 May 2017                                                                                                        
debug1: Reading configuration data /dev/null                                                                                                                              
debug1: Connecting to scap-target-01 [192.168.122.133] port 22.                                                                                                           
debug1: Connection established.                                                                                                                                           
debug1: identity file /home/vagrant/.ssh/id_rsa type 1                                                                                                                    
debug1: key_load_public: No such file or directory                                                                                                                        
debug1: identity file /home/vagrant/.ssh/id_rsa-cert type -1                                                                                                              
debug1: key_load_public: No such file or directory                                                                                                                        
debug1: identity file /home/vagrant/.ssh/id_dsa type -1                                                                                                                   
debug1: key_load_public: No such file or directory                                                                                                                        
debug1: identity file /home/vagrant/.ssh/id_dsa-cert type -1                                                                                                              
debug1: key_load_public: No such file or directory                                                                                                                        
debug1: identity file /home/vagrant/.ssh/id_ecdsa type -1                                                                                                                 
debug1: key_load_public: No such file or directory                                                                                                                        
debug1: identity file /home/vagrant/.ssh/id_ecdsa-cert type -1                                                                                                            
debug1: key_load_public: No such file or directory                                                                                                                        
debug1: identity file /home/vagrant/.ssh/id_ed25519 type -1                                                                                                               
debug1: key_load_public: No such file or directory                                                                                                                        
debug1: identity file /home/vagrant/.ssh/id_ed25519-cert type -1                                                                                                          
debug1: Enabling compatibility mode for protocol 2.0                                                                                                                      
debug1: Local version string SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u1                                                                                                       
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u1                                                                               
debug1: match: OpenSSH_7.4p1 Debian-10+deb9u1 pat OpenSSH* compat 0x04000000                                                                                              
debug1: Authenticating to scap-target-01:22 as 'vagrant'                                                                                                                  
debug1: SSH2_MSG_KEXINIT sent                                                                                                                                             
debug1: SSH2_MSG_KEXINIT received                                                                                                                                         
debug1: kex: algorithm: curve25519-sha256                                                                                                                                 
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:XdTvHeZ/g7GyMh5Y2UylYB6/sNKetKlxvRKNIJ18OSA
debug1: Host 'scap-target-01' is known and matches the ECDSA host key.
debug1: Found key in /etc/ssh/ssh_known_hosts:2
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/vagrant/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: Authentication succeeded (publickey).
Authenticated to scap-target-01 ([192.168.122.133]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Sending command: /vagrant/scap/bin/scap deploy-local -v --repo mockbase/deploy --force -g default fetch --refresh-config
http://192.168.122.1/mockbase/deploy/.git
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 1 clearing O_NONBLOCK
Transferred: sent 2828, received 5516 bytes, in 1.4 seconds
Bytes per second: sent 1994.8, received 3890.9
debug1: Exit status 70
Fri, Jan 5, 6:11 PM · Release-Engineering-Team

Thu, Jan 4

thcipriani added a comment to P6522 (An Untitled Masterwork).
0 pages to fix, 0 were resolvable.
Thu, Jan 4, 10:55 PM
thcipriani closed T184175: beta-scap-eqiad FAILED - deployment-snapshot01.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed. as Resolved.

beta-scap-eqiad seems happy. Calling this closed :)

Thu, Jan 4, 10:01 PM · Release-Engineering-Team, Beta-Cluster-Infrastructure, Continuous-Integration-Infrastructure
thcipriani added a comment to D943: Tag HEAD in the submodules not the commit hash from parent repo.

Now I'm getting

Thu, Jan 4, 8:00 PM · Release-Engineering-Team
thcipriani added a comment to D943: Tag HEAD in the submodules not the commit hash from parent repo.

Tried locally, seems to yield:

Thu, Jan 4, 7:48 PM · Release-Engineering-Team
thcipriani triaged T184175: beta-scap-eqiad FAILED - deployment-snapshot01.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed. as Normal priority.

for the time being I reverted the patch that added deployment-snapshot01 to mediawiki deployment targets in beta just so beta-scap-eqiad will stop being mad alerting. Whenever deployment-snapshot01 is all setup we can remove:

Thu, Jan 4, 6:21 PM · Release-Engineering-Team, Beta-Cluster-Infrastructure, Continuous-Integration-Infrastructure
thcipriani added a comment to T184176: Scap not working in Beta.

This one seems related to https://github.com/wikimedia/scap/commit/5acb672bea5beecf677e791eb0ca332793412201 and T182865 @mmodell do you have some time to check this out?

Thu, Jan 4, 4:29 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure, Scap
thcipriani closed T183046: scap 3.7.4-2 is broken as Resolved.

This one was resolved with the release of scap 3.7.4-3

Thu, Jan 4, 4:26 PM · Patch-For-Review, Operations, Scap
thcipriani closed T183046: scap 3.7.4-2 is broken, a subtask of T182347: SCAP: Upload debian package version 3.7.4-3, as Resolved.
Thu, Jan 4, 4:26 PM · Patch-For-Review, Operations, Packaging, Scap
thcipriani added a comment to T184175: beta-scap-eqiad FAILED - deployment-snapshot01.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed..

I merged this; https://gerrit.wikimedia.org/r/#/c/400237/ this morning. Do I need to add anything else for scap of mw to work?

Thu, Jan 4, 3:34 PM · Release-Engineering-Team, Beta-Cluster-Infrastructure, Continuous-Integration-Infrastructure

Wed, Jan 3

thcipriani committed rMSCA58f5eca67156: Bump development version to 3.8.0 (authored by thcipriani).
Bump development version to 3.8.0
Wed, Jan 3, 11:38 PM
thcipriani accepted D941: Fix path to scap in tasks.py.
Wed, Jan 3, 11:13 PM · Release-Engineering-Team
thcipriani created P6522 (An Untitled Masterwork).
Wed, Jan 3, 8:30 PM

Tue, Jan 2

thcipriani added a comment to T183952: Investigate deployment that caused high error-rate and was prevented from going past canaries by Scap.

I filed T183999: Scap canary has a shifting baseline to address the main problem I see here, which is that a deployment that spikes the error rate and that's canary check fails but is subsequently redeployed results in the canary check running with a new baseline.

Tue, Jan 2, 9:24 PM · Release-Engineering-Team (Kanban), Wikimedia-Incident, Scap
thcipriani created T183999: Scap canary has a shifting baseline.
Tue, Jan 2, 9:21 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Scap
thcipriani lowered the priority of T182266: Composer\Downloader\TransportException in mwext-testextension-hhvm-jessie from Unbreak Now! to Normal.

It looks like this change eventually merged and that this test eventually passed (https://integration.wikimedia.org/ci/job/mediawiki-core-phpcs-docker/3352/console is a run of the same change). Lowering the priority from UBN accordingly.

Tue, Jan 2, 7:44 PM · Release-Engineering-Team (Kanban), RelEng-Archive-FY201718-Q2, Patch-For-Review, Continuous-Integration-Config, Composer
thcipriani committed rMSCAaacb6a542d6d: Log message to explain canary behavior (authored by thcipriani).
Log message to explain canary behavior
Tue, Jan 2, 7:29 PM
thcipriani closed D939: Log message to explain canary behavior.
Tue, Jan 2, 7:29 PM · Release-Engineering-Team
thcipriani closed T173146: Scap MediaWiki canaries should prompt to continue as Resolved by committing rMSCAaacb6a542d6d: Log message to explain canary behavior.
Tue, Jan 2, 7:29 PM · Scap
thcipriani added a comment to T183952: Investigate deployment that caused high error-rate and was prevented from going past canaries by Scap.

Scap does seem to have failed along with that error spike: http://tools.wmflabs.org/sal/log/AWC3SqMzwg13V6286YVJ

Tue, Jan 2, 7:08 PM · Release-Engineering-Team (Kanban), Wikimedia-Incident, Scap
thcipriani added a comment to T183952: Investigate deployment that caused high error-rate and was prevented from going past canaries by Scap.

Logstash graph based on the query looks like it should have caught it: https://logstash.wikimedia.org/goto/800d886d5e05b8e1f9d11454717cf183

Tue, Jan 2, 7:04 PM · Release-Engineering-Team (Kanban), Wikimedia-Incident, Scap
thcipriani requested review of D939: Log message to explain canary behavior.
Tue, Jan 2, 5:18 PM · Release-Engineering-Team
thcipriani added a revision to T173146: Scap MediaWiki canaries should prompt to continue: D939: Log message to explain canary behavior.
Tue, Jan 2, 5:16 PM · Scap
thcipriani abandoned D923: Debian package README.
In D923#18655, @hashar wrote:
  • checkout release
  • attempt to merge master
  • build a snapshot of the result
  • deploy or whine LOUDLY that the package is broken

    So this way we continuously build the deb package and can adjust the release branch in case a breaking change to master did not come with a counterpart in release.
Tue, Jan 2, 4:35 PM · Release-Engineering-Team

Dec 20 2017

zeljkofilipin awarded D924: Scap say: moar cowsay compat a Y So Serious token.
Dec 20 2017, 10:32 AM · Release-Engineering-Team
mmodell awarded D924: Scap say: moar cowsay compat a Goat token.
Dec 20 2017, 4:10 AM · Release-Engineering-Team

Dec 19 2017

thcipriani committed rMSCAca5322465709: Scap say: moar cowsay compat (authored by thcipriani).
Scap say: moar cowsay compat
Dec 19 2017, 10:43 PM
thcipriani closed D924: Scap say: moar cowsay compat.
Dec 19 2017, 10:43 PM · Release-Engineering-Team
thcipriani committed rMSCA19aaa5e2566f: Add debian/README to release branch (authored by thcipriani).
Add debian/README to release branch
Dec 19 2017, 10:34 PM
thcipriani added a comment to D923: Debian package README.
In D923#18450, @mmodell wrote:
In D923#18438, @demon wrote:

Won't this create a merge conflict each time?

If we merge and fix the conflict once by overwriting the file with the other version, then no it shouldn't conflict after that.

Dec 19 2017, 10:31 PM · Release-Engineering-Team
thcipriani requested review of D925: Add debian/README to release branch.
Dec 19 2017, 6:33 PM · Release-Engineering-Team
thcipriani added a comment to D924: Scap say: moar cowsay compat.

Aww, phab formatting ruined my commit message...

Dec 19 2017, 6:03 PM · Release-Engineering-Team
thcipriani requested review of D924: Scap say: moar cowsay compat.
Dec 19 2017, 6:02 PM · Release-Engineering-Team
thcipriani requested review of D923: Debian package README.
Dec 19 2017, 6:02 PM · Release-Engineering-Team