thcipriani (Tyler Cipriani)
Rappa Ternt RelEng

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Feb 9 2015, 10:04 PM (111 w, 22 m)
Availability
Available
IRC Nick
thcipriani
LDAP User
Unknown
MediaWiki User
TCipriani (WMF)

Recent Activity

Today

thcipriani created D605: WIP: Use git to sync proxies.
Mon, Mar 27, 3:57 PM · Release-Engineering-Team
thcipriani created D604: Default .gitignore contra subdirectory .gitignore.
Mon, Mar 27, 3:28 PM · Release-Engineering-Team

Thu, Mar 23

thcipriani added a comment to T161036: Warning: ZeroAPI: Unable to parse json of page Zero:.

Are we sure this co-incides with the deployment? Or has someone changed a page onwiki at the same time?

Thu, Mar 23, 7:04 PM · Patch-For-Review, Wikimedia-log-errors, Zero
thcipriani removed a parent task for T161036: Warning: ZeroAPI: Unable to parse json of page Zero:: T160549: MW-1.29.0-wmf.17 deployment blockers.
Thu, Mar 23, 7:02 PM · Patch-For-Review, Wikimedia-log-errors, Zero
thcipriani removed a subtask for T160549: MW-1.29.0-wmf.17 deployment blockers: T161036: Warning: ZeroAPI: Unable to parse json of page Zero:.
Thu, Mar 23, 7:02 PM · Release, Release-Engineering-Team (Deployment-Blockers)

Wed, Mar 22

thcipriani created T161159: Cannot access the database: Can't connect to MySQL server on '10.192.48.41' (111) (10.192.48.41).
Wed, Mar 22, 9:18 PM · DBA
thcipriani added a comment to T160549: MW-1.29.0-wmf.17 deployment blockers.

Looks like most blockers got movement if not 100% resolution. I just sync'd out wmf.17 to group0. Monitoring now. Thanks all for the quick actions on all blockers :)

Wed, Mar 22, 8:47 PM · Release, Release-Engineering-Team (Deployment-Blockers)
thcipriani committed rKEYHOLDERf2da21b28a30: Add CONTRIBUTING help (authored by thcipriani).
Add CONTRIBUTING help
Wed, Mar 22, 7:29 PM
thcipriani closed D602: Add CONTRIBUTING help by committing rKEYHOLDERf2da21b28a30: Add CONTRIBUTING help.
Wed, Mar 22, 7:29 PM · Release-Engineering-Team
thcipriani updated the diff for D602: Add CONTRIBUTING help.

And fix the spelling of "licensed" :(

Wed, Mar 22, 7:27 PM · Release-Engineering-Team
thcipriani updated the diff for D602: Add CONTRIBUTING help.

Fix spelling of "phabricator" :(

Wed, Mar 22, 6:58 PM · Release-Engineering-Team
thcipriani created D602: Add CONTRIBUTING help.
Wed, Mar 22, 6:55 PM · Release-Engineering-Team

Tue, Mar 21

thcipriani added a comment to T160549: MW-1.29.0-wmf.17 deployment blockers.

Rollout to group0 resulted in a spike of a number of new Warnings:

Tue, Mar 21, 9:06 PM · Release, Release-Engineering-Team (Deployment-Blockers)
thcipriani added a subtask for T160549: MW-1.29.0-wmf.17 deployment blockers: T161040: Unknown column 'workflow_last_update_timestamp' in 'order clause'.
Tue, Mar 21, 9:03 PM · Release, Release-Engineering-Team (Deployment-Blockers)
thcipriani added a parent task for T161040: Unknown column 'workflow_last_update_timestamp' in 'order clause': T160549: MW-1.29.0-wmf.17 deployment blockers.
Tue, Mar 21, 9:03 PM · Collaboration-Team-Triage, Wikimedia-log-errors, Flow
thcipriani created T161040: Unknown column 'workflow_last_update_timestamp' in 'order clause'.
Tue, Mar 21, 9:03 PM · Collaboration-Team-Triage, Wikimedia-log-errors, Flow
thcipriani created T161039: Warning: Revision::decompressRevisionText: gzinflate() failed .
Tue, Mar 21, 8:48 PM · Wikimedia-log-errors
thcipriani created T161037: Warning: data error in /srv/mediawiki/php-1.29.0-wmf.17/includes/Revision.php on line 1351.
Tue, Mar 21, 8:44 PM · MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)), MW-1.29-release-notes, Performance-Team, MediaWiki-Cache, MediaWiki-Page-editing, Patch-For-Review, Wikimedia-log-errors
thcipriani created T161036: Warning: ZeroAPI: Unable to parse json of page Zero:.
Tue, Mar 21, 8:41 PM · Patch-For-Review, Wikimedia-log-errors, Zero

Mon, Mar 20

thcipriani closed T121585: sync-wikiversions not syncing wikiversions.json with mira as "Resolved".

Haven't noticed the problem since rMSCA9fbd6f8f486a: sync-wikiversions: Add missing sync steps merged.

Mon, Mar 20, 3:43 PM · User-bd808, Scap, Deployment-Systems
thcipriani moved T111062: Scap should abort early when Keyholder is not armed from Needs triage to Debt on the Scap board.
Mon, Mar 20, 3:35 PM · Scap, Release-Engineering-Team
thcipriani triaged T159991: If aborting a scap due to test canary error rate, output some errors for reference as "Normal" priority.

It does seem like showing some errors would be helpful; although possibly a little tricky to get at that information.

Mon, Mar 20, 3:33 PM · Scap

Fri, Mar 17

thcipriani accepted D591: Always compute new git info for all syncs, not just full scap.

lgtm, let's see what overhead it adds.

Fri, Mar 17, 9:46 PM · Release-Engineering-Team

Thu, Mar 16

thcipriani created T160669: Remove zend tests from wmf branches.
Thu, Mar 16, 5:35 PM · Continuous-Integration-Infrastructure (Little Steps Sprint)
thcipriani created T160668: Create "High Priority" gate-and-submit pipeline.
Thu, Mar 16, 5:32 PM · Continuous-Integration-Infrastructure (Little Steps Sprint)
thcipriani created T160667: Create "High Priority" test pipeline.
Thu, Mar 16, 5:27 PM · Continuous-Integration-Infrastructure (Little Steps Sprint)
thcipriani added a project to T155483: Speed up oojs/ui Jenkins jobs: Continuous-Integration-Infrastructure (Little Steps Sprint).
Thu, Mar 16, 5:25 PM · Patch-For-Review, Continuous-Integration-Infrastructure (Little Steps Sprint), OOjs-UI, Continuous-Integration-Config
thcipriani created Continuous-Integration-Infrastructure (Little Steps Sprint).
Thu, Mar 16, 5:23 PM

Sat, Mar 11

thcipriani closed T159865: Install Kubernetes Continuous Deployment Pipeline plugin on ci-staging-jenkins as "Resolved".

This plugin is installed and has all the jobs that are created in the Pearson base docker image (but not in a docker image). The seed job is looking at a scap repo that includes an environment.bitesize, application.bitesize, and a build.bitesize added. The plugin created a seed job along with the other jobs you see here:

Sat, Mar 11, 11:58 PM · Release Pipeline
thcipriani closed T159865: Install Kubernetes Continuous Deployment Pipeline plugin on ci-staging-jenkins, a subtask of T159864: Build testing k8s setup in ci-staging, as "Resolved".
Sat, Mar 11, 11:58 PM · Release Pipeline

Fri, Mar 10

thcipriani closed T146656: remove hard-coded upstart commands as "Resolved".

We use /usr/sbin/service now, is that correct?

Fri, Mar 10, 10:26 PM · Easy, Patch-For-Review, Scap
thcipriani accepted D595: New scap sayings....

Strong work.

Fri, Mar 10, 6:48 PM · Release-Engineering-Team

Wed, Mar 8

thcipriani updated subscribers of D588: Default failure_limit needs to be None to fallback to 1.

So contrary to the principal of least astonishment, setting the default failure_limit to None sets it to 1 due to https://phabricator.wikimedia.org/source/scap/browse/master/scap/targets.py;2c4894c1a1ec4cfdad3c5981325267ec434f7852$273

Wed, Mar 8, 10:32 PM · Release-Engineering-Team
thcipriani added a comment to D590: Allow reloading service instead of restarting service.

Will need to add service_reload to deploy.Deploy.DEPLOY_CONF as well so that it makes its way into the .git/DEPLOY_HEAD file and out to the targets. Apart from that, looks great!

Wed, Mar 8, 10:03 PM · Release-Engineering-Team

Tue, Mar 7

mmodell awarded T157392: Warning: Cannot modify header information - headers already sent in /srv/mediawiki/php-1.29.0-wmf.10/includes/WebResponse.php on line 45 a Like token.
Tue, Mar 7, 10:13 PM · MW-1.29-release (WMF-deploy-2017-02-28_(1.29.0-wmf.14)), MW-1.29-release-notes, Patch-For-Review, MediaWiki-General-or-Unknown, Wikimedia-log-errors
thcipriani changed the status of T159332: Beta cluster scap job ( beta-scap-eqiad ) fails due to puppet erasing /etc/ssh/ssh_known_hosts from "Open" to "Stalled".

So I put a dumb fix in place and copied a good version of the known_hosts file into ~/.ssh/known_hosts for the jenkins-deploy user—beta-scap-eqiad will continue to work, but the global known_hosts file will continue to flap.

Tue, Mar 7, 6:44 PM · Release-Engineering-Team, Beta-Cluster-Infrastructure
thcipriani added a comment to T156687: Delay repooling trending service after a restart.

@mobrovac what is the ETA on this? Just curious for planing

Tue, Mar 7, 6:35 PM · Patch-For-Review, Scap, User-mobrovac, Services (later)
thcipriani moved T159867: Add a delay configuration option to checks from Needs triage to Services improvements on the Scap board.
Tue, Mar 7, 6:31 PM · Scap
thcipriani created T159867: Add a delay configuration option to checks.
Tue, Mar 7, 6:31 PM · Scap
thcipriani created T159865: Install Kubernetes Continuous Deployment Pipeline plugin on ci-staging-jenkins.
Tue, Mar 7, 6:24 PM · Release Pipeline
thcipriani created T159864: Build testing k8s setup in ci-staging.
Tue, Mar 7, 6:21 PM · Release Pipeline
thcipriani created T159863: Evaluate deployment-pipeline-jenkins plugin.
Tue, Mar 7, 6:18 PM · Release Pipeline
thcipriani triaged T159387: Check 'depool' exceeded 30.0s timeout as "Normal" priority.

Should we maybe increase the 30s timeout, or allow for a higher failure limit?

Tue, Mar 7, 2:04 PM · Patch-For-Review, Parsoid, Scap

Mon, Mar 6

thcipriani closed T157694: scap should handle changes to .gitmodules as "Resolved".

@Halfak thank you for the fix!

Mon, Mar 6, 4:33 PM · Scap, Release-Engineering-Team

Fri, Mar 3

thcipriani added a comment to T159332: Beta cluster scap job ( beta-scap-eqiad ) fails due to puppet erasing /etc/ssh/ssh_known_hosts.

Good query to note

Fri, Mar 3, 10:00 PM · Release-Engineering-Team, Beta-Cluster-Infrastructure
thcipriani added a comment to T159332: Beta cluster scap job ( beta-scap-eqiad ) fails due to puppet erasing /etc/ssh/ssh_known_hosts.

I think this means these resrouces exist. I found them in the psql db that backs Puppet DB. Documenting here:

Fri, Mar 3, 9:46 PM · Release-Engineering-Team, Beta-Cluster-Infrastructure
thcipriani added a comment to T159332: Beta cluster scap job ( beta-scap-eqiad ) fails due to puppet erasing /etc/ssh/ssh_known_hosts.

And I guess puppet has two difference resources in race condition for /etc/ssh/ssh_known_hosts. The one above tentatively causing an empty file.

Fri, Mar 3, 8:17 PM · Release-Engineering-Team, Beta-Cluster-Infrastructure
thcipriani triaged T159332: Beta cluster scap job ( beta-scap-eqiad ) fails due to puppet erasing /etc/ssh/ssh_known_hosts as "Normal" priority.

It seems this is as a result of a patch on beta puppetmaster related to T153163: Set up and use exported resources for Tool Labs's shared knowledge. Specifically I8cfba01da13191384a607ac75917c55a28ecfda1

Fri, Mar 3, 4:52 PM · Release-Engineering-Team, Beta-Cluster-Infrastructure
thcipriani added a comment to T159332: Beta cluster scap job ( beta-scap-eqiad ) fails due to puppet erasing /etc/ssh/ssh_known_hosts.

So this is happening continually, but the root cause is somewhat confounding.

Fri, Mar 3, 4:48 PM · Release-Engineering-Team, Beta-Cluster-Infrastructure

Wed, Mar 1

thcipriani added a comment to D582: Add semantic_version python library to dependencies.

I think this may cause problems for 14.04 machines.

Wed, Mar 1, 4:36 PM · Release-Engineering-Team
thcipriani closed T159286: Create jenkins master instance to fiddle with pipeline/k8s plugins as "Resolved".

@hashar thanks for the tips! Got an instance running: http://ci-staging-jenkins.wmflabs.org/ci/

Wed, Mar 1, 1:24 AM · Release Pipeline

Tue, Feb 28

thcipriani added a comment to T159286: Create jenkins master instance to fiddle with pipeline/k8s plugins.

So I build ci-staging-jenkins-master01 in the ci-staging project. When I attempted to apply role::ci::master I hit a snag as role::ci::master includes role::ci::website which includes role::ci::slave which isn't meant to run in labs, seemingly.

Tue, Feb 28, 10:44 PM · Release Pipeline
thcipriani triaged T159286: Create jenkins master instance to fiddle with pipeline/k8s plugins as "Normal" priority.
Tue, Feb 28, 10:40 PM · Release Pipeline
thcipriani moved T159286: Create jenkins master instance to fiddle with pipeline/k8s plugins from Backlog to Build PoC on the Release Pipeline board.
Tue, Feb 28, 10:39 PM · Release Pipeline
thcipriani created T159286: Create jenkins master instance to fiddle with pipeline/k8s plugins.
Tue, Feb 28, 10:39 PM · Release Pipeline

Mon, Feb 27

thcipriani reopened T127762: Update Debian Package for Scap3 as "Open".

Hiya @fgiunchedi I just tagged a new Scap version (3.5.3-1) this morning rMSCA7a2395a559ee: Bump debian version 3.5.3-1 — could I get you to rebuild the scap package?

Mon, Feb 27, 7:04 PM · Patch-For-Review, Deployment-Systems, Scap
thcipriani committed rMSCA7a2395a559ee: Bump debian version 3.5.3-1 (authored by thcipriani).
Bump debian version 3.5.3-1
Mon, Feb 27, 7:00 PM
thcipriani committed rMSCA3657e0c26a33: Bump version to 3.5.3 (authored by thcipriani).
Bump version to 3.5.3
Mon, Feb 27, 7:00 PM
thcipriani committed rMSCA9efa1c09f2c9: Merge branch 'master' into release (authored by thcipriani).
Merge branch 'master' into release
Mon, Feb 27, 7:00 PM
thcipriani committed rMSCAf8c8062a09d3: Explicit sync-common step in mediawiki deploy (authored by thcipriani).
Explicit sync-common step in mediawiki deploy
Mon, Feb 27, 4:35 PM
thcipriani closed T156851: scap wikiversions compile happening too late in scap sync as "Resolved" by committing rMSCAf8c8062a09d3: Explicit sync-common step in mediawiki deploy.
Mon, Feb 27, 4:35 PM · Deployment-Systems, Scap, scap2
thcipriani closed D572: Explicit sync-common step in mediawiki deploy by committing rMSCAf8c8062a09d3: Explicit sync-common step in mediawiki deploy.
Mon, Feb 27, 4:35 PM · Release-Engineering-Team

Feb 24 2017

thcipriani updated the diff for D575: Improve TypeError message for LogstashFormatter.

Include original TypeError message in new message
Don't use regex in TypeError test

Feb 24 2017, 7:54 PM · Release-Engineering-Team
thcipriani added a comment to D575: Improve TypeError message for LogstashFormatter.
In D575#11351, @hashar wrote:

Once second thought. Assuming str.format() fails because of %<foo>, we can have that fixed by escaping percentage sign in fields['msg'] with:

fields['msg'].replace('%', '%%')

And keep the exception catching proposed in this diff. That would ease debugging if there is some other failure reason.

Feb 24 2017, 6:21 PM · Release-Engineering-Team
thcipriani updated the diff for D575: Improve TypeError message for LogstashFormatter.

Add TypeError message test

Feb 24 2017, 6:14 PM · Release-Engineering-Team
thcipriani created T158928: Notice: Undefined index: \d+ in /srv/mediawiki/php-1.29.0-wmf.13/extensions/Collection/Collection.body.php on line 75(5|6).
Feb 24 2017, 12:12 AM · Wikimedia-log-errors, Collection

Feb 23 2017

thcipriani closed T158818: Notice: Undefined index: europeana2802016 in /srv/mediawiki/php-1.29.0-wmf.13/extensions/ContentTranslation/ContentTranslation.hooks.php on line 213 as "Resolved".

Looks like this one went away when rECTX5aca93407514: Really disable europeana2802016 campaign went out for T158297: Disable europeana2802016 campaign today at 14:30 UTC

Feb 23 2017, 9:37 PM · Wikimedia-log-errors, ContentTranslation
thcipriani created D575: Improve TypeError message for LogstashFormatter.
Feb 23 2017, 9:22 PM · Release-Engineering-Team
thcipriani added a comment to T149872: Considering adding a --no-touch flag to scap that stops automatic touch of InitialiseSettings.php.

I honestly haven't gone back to read all of this in depth and remember what we discussed exactly, but I do remember that the use of --beta-only-change was a conscious decision based on the usages we wanted to encourage and discourage and that we specifically blocked usage with arbitrary php files.

Feb 23 2017, 7:32 PM · Scap, Deployment-Systems

Feb 22 2017

thcipriani edited the description of T158818: Notice: Undefined index: europeana2802016 in /srv/mediawiki/php-1.29.0-wmf.13/extensions/ContentTranslation/ContentTranslation.hooks.php on line 213.
Feb 22 2017, 10:04 PM · Wikimedia-log-errors, ContentTranslation
thcipriani moved T158818: Notice: Undefined index: europeana2802016 in /srv/mediawiki/php-1.29.0-wmf.13/extensions/ContentTranslation/ContentTranslation.hooks.php on line 213 from Backlog to Production Impact on the Wikimedia-log-errors board.
Feb 22 2017, 10:02 PM · Wikimedia-log-errors, ContentTranslation
thcipriani created T158818: Notice: Undefined index: europeana2802016 in /srv/mediawiki/php-1.29.0-wmf.13/extensions/ContentTranslation/ContentTranslation.hooks.php on line 213.
Feb 22 2017, 10:02 PM · Wikimedia-log-errors, ContentTranslation

Feb 21 2017

thcipriani created D572: Explicit sync-common step in mediawiki deploy.
Feb 21 2017, 9:40 PM · Release-Engineering-Team
thcipriani added a revision to T156851: scap wikiversions compile happening too late in scap sync: D572: Explicit sync-common step in mediawiki deploy.
Feb 21 2017, 9:40 PM · Deployment-Systems, Scap, scap2

Feb 16 2017

thcipriani created T158367: Session "{session}": Unverified user provided and no metadata to auth it.
Feb 16 2017, 11:48 PM · MW-1.29-release (WMF-deploy-2017-02-21_(1.29.0-wmf.13)), MW-1.29-release-notes, Patch-For-Review, Wikimedia-log-errors
thcipriani created T158365: Session "{session}": Metadata merge failed: {exception}.
Feb 16 2017, 11:41 PM · Wikimedia-log-errors
thcipriani created T158364: Throttler data not found for {user}.
Feb 16 2017, 11:36 PM · MW-1.29-release (WMF-deploy-2017-02-21_(1.29.0-wmf.13)), MW-1.29-release-notes, Patch-For-Review, Wikimedia-log-errors
thcipriani created P4942 (An Untitled Masterwork).
Feb 16 2017, 10:55 PM
thcipriani closed T158217: Fatal error: Call to undefined method __PHP_Incomplete_Class::asOfTime() in /srv/mediawiki/php-1.29.0-wmf.11/includes/libs/rdbms/ChronologyProtector.php on line 327 as "Resolved".

Seems to have fixed the error in group0. Thanks for the quick work.

Feb 16 2017, 12:14 AM · MW-1.29-release-notes, MW-1.29-release (WMF-deploy-2017-02-21_(1.29.0-wmf.13)), MW-1.29-release (WMF-deploy-2017-02-14_(1.29.0-wmf.12)), Patch-For-Review, MediaWiki-Database, Performance-Team, Wikimedia-log-errors
thcipriani closed T158217: Fatal error: Call to undefined method __PHP_Incomplete_Class::asOfTime() in /srv/mediawiki/php-1.29.0-wmf.11/includes/libs/rdbms/ChronologyProtector.php on line 327, a subtask of T155527: MW-1.29.0-wmf.12 deployment blockers, as "Resolved".
Feb 16 2017, 12:14 AM · Patch-For-Review, Release, Release-Engineering-Team (Deployment-Blockers)

Feb 15 2017

thcipriani added a comment to T158217: Fatal error: Call to undefined method __PHP_Incomplete_Class::asOfTime() in /srv/mediawiki/php-1.29.0-wmf.11/includes/libs/rdbms/ChronologyProtector.php on line 327.

@aaron in the task description I was referring to https://gerrit.wikimedia.org/r/#/c/337730/1

Feb 15 2017, 8:30 PM · MW-1.29-release-notes, MW-1.29-release (WMF-deploy-2017-02-21_(1.29.0-wmf.13)), MW-1.29-release (WMF-deploy-2017-02-14_(1.29.0-wmf.12)), Patch-For-Review, MediaWiki-Database, Performance-Team, Wikimedia-log-errors
thcipriani triaged T158217: Fatal error: Call to undefined method __PHP_Incomplete_Class::asOfTime() in /srv/mediawiki/php-1.29.0-wmf.11/includes/libs/rdbms/ChronologyProtector.php on line 327 as "Unbreak Now!" priority.

Raising to UBN, added as a 1.29.0-wmf.12 train blocker.

Feb 15 2017, 6:06 PM · MW-1.29-release-notes, MW-1.29-release (WMF-deploy-2017-02-21_(1.29.0-wmf.13)), MW-1.29-release (WMF-deploy-2017-02-14_(1.29.0-wmf.12)), Patch-For-Review, MediaWiki-Database, Performance-Team, Wikimedia-log-errors
thcipriani added a subtask for T155527: MW-1.29.0-wmf.12 deployment blockers: T158217: Fatal error: Call to undefined method __PHP_Incomplete_Class::asOfTime() in /srv/mediawiki/php-1.29.0-wmf.11/includes/libs/rdbms/ChronologyProtector.php on line 327.
Feb 15 2017, 6:05 PM · Patch-For-Review, Release, Release-Engineering-Team (Deployment-Blockers)
thcipriani added a parent task for T158217: Fatal error: Call to undefined method __PHP_Incomplete_Class::asOfTime() in /srv/mediawiki/php-1.29.0-wmf.11/includes/libs/rdbms/ChronologyProtector.php on line 327: T155527: MW-1.29.0-wmf.12 deployment blockers.
Feb 15 2017, 6:05 PM · MW-1.29-release-notes, MW-1.29-release (WMF-deploy-2017-02-21_(1.29.0-wmf.13)), MW-1.29-release (WMF-deploy-2017-02-14_(1.29.0-wmf.12)), Patch-For-Review, MediaWiki-Database, Performance-Team, Wikimedia-log-errors
thcipriani created T158217: Fatal error: Call to undefined method __PHP_Incomplete_Class::asOfTime() in /srv/mediawiki/php-1.29.0-wmf.11/includes/libs/rdbms/ChronologyProtector.php on line 327.
Feb 15 2017, 6:05 PM · MW-1.29-release-notes, MW-1.29-release (WMF-deploy-2017-02-21_(1.29.0-wmf.13)), MW-1.29-release (WMF-deploy-2017-02-14_(1.29.0-wmf.12)), Patch-For-Review, MediaWiki-Database, Performance-Team, Wikimedia-log-errors
thcipriani added a comment to T155527: MW-1.29.0-wmf.12 deployment blockers.

Last week I called out some of the logspam that lowered my confidence about deployments to great effect! All of the tasks I pointed to saw some kind of movement which was awesome.

Feb 15 2017, 4:56 PM · Patch-For-Review, Release, Release-Engineering-Team (Deployment-Blockers)
thcipriani triaged T157392: Warning: Cannot modify header information - headers already sent in /srv/mediawiki/php-1.29.0-wmf.10/includes/WebResponse.php on line 45 as "Normal" priority.
Feb 15 2017, 4:56 PM · MW-1.29-release (WMF-deploy-2017-02-28_(1.29.0-wmf.14)), MW-1.29-release-notes, Patch-For-Review, MediaWiki-General-or-Unknown, Wikimedia-log-errors
thcipriani added a comment to T125735: Warning: timed out after 0.2 seconds when connecting to rdb1001.eqiad.wmnet [110]: Connection timed out.

As you mentioned before, maybe 200ms of timeout for a Jobrunner is too tight? I understand a limit like that for an appserver but not for a batch job scheduler :)

Feb 15 2017, 4:54 PM · User-Elukey, Operations, Wikimedia-log-errors
thcipriani triaged T157646: wfShellExec errors end up in HHVM log as "Normal" priority.

@hashar believes (and I have no reason to doubt) that this is the ultimate cause of all of the Syntax Error/Warning type messages in the logs, e.g.:

Feb 15 2017, 4:30 PM · Wikimedia-log-errors, MediaWiki-Debug-Logger
thcipriani closed T157030: cannot delete non-empty directory: php-1.29.0-wmf.3 messages on 'scap sync' on mwdebug1002 as "Resolved".

What's happening here is that we removed the wmf-1.29.0-wmf.{3,4} directory on tin; however rsync --delete does not have the permission to remove files on a remote machine that are owned by a different user (in this case l10nupdate).

Feb 15 2017, 12:06 AM · Scap, Release-Engineering-Team, Deployment-Systems, Operations, User-Addshore

Feb 14 2017

thcipriani raised the priority of T158127: Catchable fatal error: Object of class __PHP_Incomplete_Class could not be converted to string in ChronologyProtector.php from "High" to "Unbreak Now!".

Not sure if this should be unbreak so setting it as high priority.

Feb 14 2017, 10:15 PM · MW-1.29-release (WMF-deploy-2017-02-07_(1.29.0-wmf.11)), MW-1.29-release (WMF-deploy-2017-02-14_(1.29.0-wmf.12)), MW-1.29-release (WMF-deploy-2017-02-21_(1.29.0-wmf.13)), MW-1.29-release-notes, Performance-Team, MediaWiki-Database, Wikimedia-log-errors
thcipriani added a subtask for T155527: MW-1.29.0-wmf.12 deployment blockers: T158127: Catchable fatal error: Object of class __PHP_Incomplete_Class could not be converted to string in ChronologyProtector.php.
Feb 14 2017, 10:14 PM · Patch-For-Review, Release, Release-Engineering-Team (Deployment-Blockers)
thcipriani added a parent task for T158127: Catchable fatal error: Object of class __PHP_Incomplete_Class could not be converted to string in ChronologyProtector.php: T155527: MW-1.29.0-wmf.12 deployment blockers.
Feb 14 2017, 10:14 PM · MW-1.29-release (WMF-deploy-2017-02-07_(1.29.0-wmf.11)), MW-1.29-release (WMF-deploy-2017-02-14_(1.29.0-wmf.12)), MW-1.29-release (WMF-deploy-2017-02-21_(1.29.0-wmf.13)), MW-1.29-release-notes, Performance-Team, MediaWiki-Database, Wikimedia-log-errors
thcipriani created T158127: Catchable fatal error: Object of class __PHP_Incomplete_Class could not be converted to string in ChronologyProtector.php.
Feb 14 2017, 10:07 PM · MW-1.29-release (WMF-deploy-2017-02-07_(1.29.0-wmf.11)), MW-1.29-release (WMF-deploy-2017-02-14_(1.29.0-wmf.12)), MW-1.29-release (WMF-deploy-2017-02-21_(1.29.0-wmf.13)), MW-1.29-release-notes, Performance-Team, MediaWiki-Database, Wikimedia-log-errors
thcipriani triaged T157469: Mathoid CI Container Image POC as "Normal" priority.

So now that service-runner has the generate subcommand you can run something like:

Feb 14 2017, 4:28 PM · User-mobrovac, Release Pipeline, Continuous-Integration-Infrastructure
thcipriani added a comment to D568: Update debian changelog for 3.5.2-1.

Minor nit inline.

Feb 14 2017, 2:50 PM · Scap, Release-Engineering-Team
thcipriani accepted D567: Add simple subcommand for logging to server admin log.

tested. Works fine, makes sense to me.

Feb 14 2017, 12:45 AM · Release-Engineering-Team

Feb 13 2017

thcipriani accepted D566: Exclude empty deploy groups.

Manually tested a few different ways and reviewed code — nice succinct solution.

Feb 13 2017, 11:46 PM · Release-Engineering-Team
thcipriani accepted D558: Fix `failure_rate` percentages.

LGTM!

Feb 13 2017, 11:20 PM · Release-Engineering-Team
thcipriani committed rMSCAa1ae82075ab3: Make checks serial by default (authored by thcipriani).
Make checks serial by default
Feb 13 2017, 10:27 PM