thcipriani (Tyler Cipriani)Administrator
Rappa Ternt RelEng

Projects (16)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Feb 9 2015, 10:04 PM (196 w, 6 d)
Roles
Administrator
Availability
Available
IRC Nick
thcipriani
LDAP User
Unknown
MediaWiki User
TCipriani (WMF) [ Global Accounts ]

Recent Activity

Fri, Nov 16

thcipriani committed rGBLBR236d8df84f9f: Fix entrypoint merge behavior (authored by thcipriani).
Fix entrypoint merge behavior
Fri, Nov 16, 11:44 PM
thcipriani committed rGBLBRa93bb3df77d2: Combine blubberoid.yaml and .pipeline/blubber.yaml (authored by thcipriani).
Combine blubberoid.yaml and .pipeline/blubber.yaml
Fri, Nov 16, 11:17 PM
thcipriani committed rGBLBR256a9d093425: Add: a Blubber file for a Blubberoid service Docker image (authored by LarsWirzenius).
Add: a Blubber file for a Blubberoid service Docker image
Fri, Nov 16, 11:07 PM
thcipriani committed rGBLBRac5f08e9b613: Combine blubberoid.yaml and .pipeline/blubber.yaml (authored by thcipriani).
Combine blubberoid.yaml and .pipeline/blubber.yaml
Fri, Nov 16, 11:07 PM
thcipriani awarded Blog Post: Bring in 'da noise, bring in defunct. It's a zombie party! a Pirate Logo token.
Fri, Nov 16, 10:43 PM · Continuous-Integration-Infrastructure, Release-Engineering-Team
thcipriani committed rGBLBRfd8309accee2: Bump version number (authored by thcipriani).
Bump version number
Fri, Nov 16, 9:16 PM
thcipriani updated subscribers of T207707: contint1001 store docker images on separate partition or disk.

@RobH would you know if it's possible to add physical storage to this machine? If not we'll have to work out a different solution.

Fri, Nov 16, 8:59 PM · Operations, Release-Engineering-Team, Continuous-Integration-Infrastructure
mmodell awarded T209456: Gerrit is down "502 Proxy Error" a Orange Medal token.
Fri, Nov 16, 5:49 PM · Operations, Gerrit
thcipriani closed T209456: Gerrit is down "502 Proxy Error" as Resolved.

Two days later -- after adding in some better monitoring and blocking bots from indexing git blame for giant files with a lot of history -- we seem to be in a stable place.

Fri, Nov 16, 4:42 PM · Operations, Gerrit

Thu, Nov 15

thcipriani awarded T209361: Phase out Nodepool from production a Yellow Medal token.
Thu, Nov 15, 8:08 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Continuous-Integration-Infrastructure (shipyard), Release-Engineering-Team (Kanban)
thcipriani triaged T207864: Allow configurable wait / watch time on canaries for mediawiki scap sync as Normal priority.

What's the use case for longer time on the canaries? Do you want to check something, or allow more time for traffic to hit those machines? If you want to check something maybe we could prompt deployer to move off of canaries and onto the rest of production rather than having a configurable time.

Thu, Nov 15, 7:51 PM · Scap

Wed, Nov 14

thcipriani closed T209526: Add javamelody to gerrit as Resolved.

Now available for gerrit admins at: https://gerrit.wikimedia.org/r/monitoring

Wed, Nov 14, 9:50 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Operations, Gerrit
thcipriani closed T209526: Add javamelody to gerrit, a subtask of T209456: Gerrit is down "502 Proxy Error", as Resolved.
Wed, Nov 14, 9:50 PM · Operations, Gerrit
thcipriani committed rGERRITDEPLOYab2fa18f4bf0: Add javamelody plugin and dependencies (authored by thcipriani).
Add javamelody plugin and dependencies
Wed, Nov 14, 8:55 PM
thcipriani committed rGERRITDEPLOY43215804344c: Add JavaMelody plugin (authored by thcipriani).
Add JavaMelody plugin
Wed, Nov 14, 8:08 PM
thcipriani claimed T209526: Add javamelody to gerrit.
Wed, Nov 14, 7:35 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Operations, Gerrit
thcipriani triaged T209526: Add javamelody to gerrit as High priority.
Wed, Nov 14, 7:34 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Operations, Gerrit
thcipriani added a comment to T184086: Add prometheus exporter to Gerrit.

Seems like the metrics reporter plugin hasn't received any updates for 8 months now: https://gerrit.googlesource.com/plugins/metrics-reporter-jmx/

Wed, Nov 14, 7:24 PM · Release-Engineering-Team (Watching / External), Gerrit, Operations
thcipriani added a comment to T209456: Gerrit is down "502 Proxy Error".

We had a lot of stack overflow errors on random pages such as gittiles, that is due to some pages being very long to prettify.

Wed, Nov 14, 4:06 PM · Operations, Gerrit

Fri, Nov 9

thcipriani triaged T209088: Design pipeline image versioning scheme as Normal priority.
Fri, Nov 9, 3:36 PM · Release-Engineering-Team, Operations, Release Pipeline

Thu, Nov 8

thcipriani triaged T209123: contint1001 build cleanup as High priority.
Thu, Nov 8, 11:02 PM · Patch-For-Review, Release-Engineering-Team, Continuous-Integration-Infrastructure
thcipriani created T209123: contint1001 build cleanup.
Thu, Nov 8, 11:01 PM · Patch-For-Review, Release-Engineering-Team, Continuous-Integration-Infrastructure
thcipriani edited P7781 clean.sh.
Thu, Nov 8, 10:55 PM
thcipriani created P7781 clean.sh.
Thu, Nov 8, 10:52 PM
thcipriani closed T209036: The title "Foo" does not refer to an existing page as Resolved.

Mentioned in SAL (#wikimedia-operations) [2018-11-08T19:35:40Z] <thcipriani@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:472411|Revert "Disable wmgUseTwoColConflict everywhere"]] T205942 T208840 T209012 T209036 (duration: 00m 54s)

Thu, Nov 8, 8:18 PM · MW-1.33-notes (1.33.0-wmf.3; 2018-11-06), Patch-For-Review, TCB-Team, Two-Column-Edit-Conflict-Merge
thcipriani closed T209036: The title "Foo" does not refer to an existing page, a subtask of T206657: 1.33.0-wmf.3 deployment blockers, as Resolved.
Thu, Nov 8, 8:18 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
thcipriani created T209088: Design pipeline image versioning scheme.
Thu, Nov 8, 7:00 PM · Release-Engineering-Team, Operations, Release Pipeline
thcipriani closed T207460: Update L10n-bot@phabricator ssh key as Resolved.

Done, let me know if anything is amiss.

Thu, Nov 8, 4:49 PM · Release-Engineering-Team (Kanban), translatewiki.net, Phabricator-Bot-Requests
thcipriani added a comment to T192006: wdqs-frontend docker image should (BLUBBER) rebuild automatically when a new patch is pushed to master.

So, while trying to move this to blubber I hit the first issue.

 $ docker run --rm -it wdqs-gui-test-$(date --iso)
 
> wikidata-query-gui@0.1.4 test /srv/service
> grunt test
 
>> Local Npm module "grunt-auto-install" not found. Is it installed?
>> Local Npm module "grunt-banana-checker" not found. Is it installed?
>> Local Npm module "grunt-contrib-clean" not found. Is it installed?
>> Local Npm module "grunt-contrib-concat" not found. Is it installed?
>> Local Npm module "grunt-contrib-copy" not found. Is it installed?
>> Local Npm module "grunt-contrib-cssmin" not found. Is it installed?
>> Local Npm module "grunt-contrib-htmlmin" not found. Is it installed?
>> Local Npm module "grunt-contrib-jshint" not found. Is it installed?
>> Local Npm module "grunt-contrib-less" not found. Is it installed?
>> Local Npm module "grunt-contrib-qunit" not found. Is it installed?
>> Local Npm module "grunt-contrib-uglify" not found. Is it installed?
>> Local Npm module "grunt-filerev" not found. Is it installed?
>> Local Npm module "grunt-jscs" not found. Is it installed?
>> Local Npm module "grunt-jsonlint" not found. Is it installed?
>> Local Npm module "grunt-merge-i18n" not found. Is it installed?
>> Local Npm module "grunt-shell" not found. Is it installed?
>> Local Npm module "grunt-stylelint" not found. Is it installed?
>> Local Npm module "grunt-usemin" not found. Is it installed?
 Warning: Task "jshint" not found. Use --force to continue.
 
 Aborted due to warnings.
 npm ERR! Test failed.  See above for more details.
 npm WARN Local package.json exists, but node_modules missing, did you mean to install?

It looks like this is due to the dockerfile generated by blubber doing the install of npm packages in one directory:

COPY --chown=65533:65533 ["package.json", "/opt/lib/"]
RUN cd "/opt/lib" && npm install
Thu, Nov 8, 4:39 PM · Patch-For-Review, Release Pipeline (Blubber), Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), User-Addshore, wikidata-tech-focus, Wikidata, Wikibase-Containers

Wed, Nov 7

thcipriani closed T208995: PHP Fatal Error: Call to private method LdapAuthenticationPlugin::bindAs() from context 'OpenStackNovaLdapConnection' as Resolved.

After re-deploy of wmf.3 to labswiki, I no longer see this error. Thanks for the quick action on this!

Wed, Nov 7, 10:16 PM · MW-1.33-notes (1.33.0-wmf.3; 2018-11-06), MediaWiki-extensions-OpenStackManager, Wikimedia-production-error
thcipriani closed T208995: PHP Fatal Error: Call to private method LdapAuthenticationPlugin::bindAs() from context 'OpenStackNovaLdapConnection', a subtask of T206657: 1.33.0-wmf.3 deployment blockers, as Resolved.
Wed, Nov 7, 10:16 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
thcipriani added a subtask for T206657: 1.33.0-wmf.3 deployment blockers: T208995: PHP Fatal Error: Call to private method LdapAuthenticationPlugin::bindAs() from context 'OpenStackNovaLdapConnection'.
Wed, Nov 7, 8:33 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
thcipriani added a parent task for T208995: PHP Fatal Error: Call to private method LdapAuthenticationPlugin::bindAs() from context 'OpenStackNovaLdapConnection': T206657: 1.33.0-wmf.3 deployment blockers.
Wed, Nov 7, 8:33 PM · MW-1.33-notes (1.33.0-wmf.3; 2018-11-06), MediaWiki-extensions-OpenStackManager, Wikimedia-production-error
thcipriani triaged T208995: PHP Fatal Error: Call to private method LdapAuthenticationPlugin::bindAs() from context 'OpenStackNovaLdapConnection' as Unbreak Now! priority.
Wed, Nov 7, 8:30 PM · MW-1.33-notes (1.33.0-wmf.3; 2018-11-06), MediaWiki-extensions-OpenStackManager, Wikimedia-production-error
thcipriani added a comment to T208870: TCP connections between analytics and deployment-prep.

Re-opening to see if we can find a temporary solution (as discussed with @thcipriani)

Wed, Nov 7, 6:28 PM · Analytics, User-Elukey, Beta-Cluster-Infrastructure
thcipriani created P7771 (An Untitled Masterwork).
Wed, Nov 7, 6:06 PM
thcipriani committed rMSCA0f2f1cc6765f: Bumping Debian version to 3.8.9-1 (authored by thcipriani).
Bumping Debian version to 3.8.9-1
Wed, Nov 7, 5:55 PM
thcipriani committed rMSCA08d9d92724a2: Bump version to 3.8.9 (authored by thcipriani).
Bump version to 3.8.9
Wed, Nov 7, 5:55 PM
thcipriani committed rMSCAe2c454d930df: Merge branch 'master' into release (authored by thcipriani).
Merge branch 'master' into release
Wed, Nov 7, 5:55 PM
thcipriani committed rMSCA60b2cc96ca83: Fix ascii art regression (authored by thcipriani).
Fix ascii art regression
Wed, Nov 7, 5:52 PM
thcipriani closed D1126: Fix ascii art regression.
Wed, Nov 7, 5:52 PM · Release-Engineering-Team
thcipriani requested review of D1126: Fix ascii art regression.
Wed, Nov 7, 5:48 PM · Release-Engineering-Team
thcipriani committed rMSCA01d9d88acfc3: Bump number of shame (authored by thcipriani).
Bump number of shame
Wed, Nov 7, 5:24 PM
thcipriani added a comment to D1117: Only check logstash for canaries in the active datacenter.

is this ready to merge? I'm cutting 3.8.8 without this patch unless someone shouts at me soon.

Wed, Nov 7, 5:04 PM · Release-Engineering-Team

Tue, Nov 6

thcipriani committed rGERRITDEPLOYd2763c6a35a8: Gerrit 2.15.6 release (authored by thcipriani).
Gerrit 2.15.6 release
Tue, Nov 6, 4:44 PM

Mon, Nov 5

thcipriani created T208813: Scap deployers should have the ability to depool and restart HHVM.
Mon, Nov 5, 11:03 PM · Operations, Release-Engineering-Team, Scap
thcipriani closed T208549: HHVM CPU usage when deploying MediaWiki as Resolved.

@akosiaris helped to roll wmf.2 out today. It required us to use cumin to depool each appserver and restart hhvm as part of deployment:

Mon, Nov 5, 10:55 PM · Wikimedia-production-error, Release-Engineering-Team, Operations
thcipriani closed T208549: HHVM CPU usage when deploying MediaWiki, a subtask of T206656: 1.33.0-wmf.2 deployment blockers, as Resolved.
Mon, Nov 5, 10:55 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
thcipriani closed T206656: 1.33.0-wmf.2 deployment blockers as Resolved.
Mon, Nov 5, 10:51 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
thcipriani committed rGERRITDEPLOY4b9c84520027: Merge tag 'v2.15.6' into wmf/stable-2.15 (authored by thcipriani).
Merge tag 'v2.15.6' into wmf/stable-2.15
Mon, Nov 5, 5:03 PM
thcipriani added a comment to T207694: Adopt JSON as blubber's internal configuration format.

As I understand it that is for blubber internals, can you confirm we would still have the repository configuration held with yaml (.pipeline/blubber.yaml)? YAML is, to me, friendlier (comments, optionally no brackets, curly braces etc) and has a few useful tricks (aliases).

Mon, Nov 5, 4:40 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Release Pipeline (Blubber)

Fri, Nov 2

thcipriani added a comment to T208549: HHVM CPU usage when deploying MediaWiki.

A bunch of new messages are getting tidied and going through remex now ( https://gerrit.wikimedia.org/r/#/q/topic:deprecate-wgtidy ) -- although that *shouldn't* cause problems, here are a couple of wild theories:

  1. MessageCache is missing somehow ( 4b1db1190bb8f2a115c6a81a5ee487b7d18cd303 ) and tying up CPU (but I'm pretty sure the misses would show in logs). Rolling back would cause misses on the updated messages so "not solve the issue".
Fri, Nov 2, 6:10 PM · Wikimedia-production-error, Release-Engineering-Team, Operations
thcipriani added a comment to T208549: HHVM CPU usage when deploying MediaWiki.

I don't see this as a blocker to try to redeploy the train on monday.

Fri, Nov 2, 5:13 PM · Wikimedia-production-error, Release-Engineering-Team, Operations
thcipriani committed rGBLBR50f8269b3baa: [WIP] Update library to convert YAML to JSON (authored by thcipriani).
[WIP] Update library to convert YAML to JSON
Fri, Nov 2, 4:25 PM

Thu, Nov 1

thcipriani renamed T208549: HHVM CPU usage when deploying MediaWiki from HHVM CPU usage with 1.33.0-wmf.2 to HHVM CPU usage when deploying MediaWiki.
Thu, Nov 1, 9:37 PM · Wikimedia-production-error, Release-Engineering-Team, Operations
thcipriani added a subtask for T206656: 1.33.0-wmf.2 deployment blockers: T208549: HHVM CPU usage when deploying MediaWiki.
Thu, Nov 1, 9:36 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
thcipriani added a parent task for T208549: HHVM CPU usage when deploying MediaWiki: T206656: 1.33.0-wmf.2 deployment blockers.
Thu, Nov 1, 9:36 PM · Wikimedia-production-error, Release-Engineering-Team, Operations
thcipriani created T208549: HHVM CPU usage when deploying MediaWiki.
Thu, Nov 1, 9:35 PM · Wikimedia-production-error, Release-Engineering-Team, Operations
thcipriani added a comment to T208469: Spike in PHP warnings from User.php; triggered from MobileFrontend's new use of ContribsPager with a User object instead of a string.

Removed as a deployment blocker, but leaving open for further review per IRC.

Thu, Nov 1, 7:16 PM · MW-1.33-notes (1.33.0-wmf.2; 2018-10-30), Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), MobileFrontend, Wikimedia-production-error
thcipriani removed a parent task for T208469: Spike in PHP warnings from User.php; triggered from MobileFrontend's new use of ContribsPager with a User object instead of a string: T206656: 1.33.0-wmf.2 deployment blockers.
Thu, Nov 1, 7:15 PM · MW-1.33-notes (1.33.0-wmf.2; 2018-10-30), Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), MobileFrontend, Wikimedia-production-error
thcipriani removed a subtask for T206656: 1.33.0-wmf.2 deployment blockers: T208469: Spike in PHP warnings from User.php; triggered from MobileFrontend's new use of ContribsPager with a User object instead of a string.
Thu, Nov 1, 7:15 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
thcipriani added a comment to T208469: Spike in PHP warnings from User.php; triggered from MobileFrontend's new use of ContribsPager with a User object instead of a string.

Hrm, I'm no longer seeing these errors in logstash, and I see that a couple patches are merged and at least 1 is backported to 1.33.0-wmf.2, does anything remain to be resolved? Still blocking further train rollout?

Thu, Nov 1, 7:08 PM · MW-1.33-notes (1.33.0-wmf.2; 2018-10-30), Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), MobileFrontend, Wikimedia-production-error
thcipriani triaged T208529: Install docker on releases-jenkins as Normal priority.
Thu, Nov 1, 6:05 PM · Security, Release-Engineering-Team, MediaWiki-Releasing
thcipriani triaged T208528: Create fab task to deploy jenkins job to either ci-jenkins or releases-jenkins as Normal priority.
Thu, Nov 1, 6:02 PM · Security, Release-Engineering-Team, MediaWiki-Releasing
thcipriani triaged T208527: Create jenkins job to build MediaWiki tarball releases as Normal priority.
Thu, Nov 1, 5:58 PM · Patch-For-Review, Security, Release-Engineering-Team, MediaWiki-Releasing
thcipriani closed T207346: Decide where to store jobs for releases-jenkins as Resolved.

Sounds like the general consensus is to start with integration/config and work from there, so I'll got ahead and close out this task.

Thu, Nov 1, 5:55 PM · Continuous-Integration-Config, Release-Engineering-Team

Wed, Oct 31

thcipriani added a comment to T208469: Spike in PHP warnings from User.php; triggered from MobileFrontend's new use of ContribsPager with a User object instead of a string.
Wed, Oct 31, 9:18 PM · MW-1.33-notes (1.33.0-wmf.2; 2018-10-30), Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), MobileFrontend, Wikimedia-production-error
thcipriani updated subscribers of T208469: Spike in PHP warnings from User.php; triggered from MobileFrontend's new use of ContribsPager with a User object instead of a string.

I don't think that it is partial blocks related. I noticed this at a low volume when I rolled forward to 1.33.0-wmf.2 and filed T208468.

Wed, Oct 31, 9:17 PM · MW-1.33-notes (1.33.0-wmf.2; 2018-10-30), Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), MobileFrontend, Wikimedia-production-error
thcipriani merged task T208468: Illegal offset type/Array key should be either a string or an integer in /srv/mediawiki/php-1.33.0-wmf.2/includes/user/User.php into T208469: Spike in PHP warnings from User.php; triggered from MobileFrontend's new use of ContribsPager with a User object instead of a string.
Wed, Oct 31, 9:14 PM · Wikimedia-production-error
thcipriani merged T208468: Illegal offset type/Array key should be either a string or an integer in /srv/mediawiki/php-1.33.0-wmf.2/includes/user/User.php into T208469: Spike in PHP warnings from User.php; triggered from MobileFrontend's new use of ContribsPager with a User object instead of a string.
Wed, Oct 31, 9:14 PM · MW-1.33-notes (1.33.0-wmf.2; 2018-10-30), Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), MobileFrontend, Wikimedia-production-error
thcipriani created T208468: Illegal offset type/Array key should be either a string or an integer in /srv/mediawiki/php-1.33.0-wmf.2/includes/user/User.php.
Wed, Oct 31, 9:04 PM · Wikimedia-production-error
thcipriani lowered the priority of T208462: Error Unknown column ipb_sitewide in field list on query from Unbreak Now! to High.

<volans|off> ok all confirmed, the only one without that field in the core DBs is only db2050 only for ruwikiquote

Wed, Oct 31, 8:24 PM · DBA, Anti-Harassment, Operations
thcipriani removed a subtask for T206656: 1.33.0-wmf.2 deployment blockers: T208462: Error Unknown column ipb_sitewide in field list on query.
Wed, Oct 31, 8:23 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
thcipriani removed a parent task for T208462: Error Unknown column ipb_sitewide in field list on query: T206656: 1.33.0-wmf.2 deployment blockers.
Wed, Oct 31, 8:23 PM · DBA, Anti-Harassment, Operations
thcipriani triaged T208462: Error Unknown column ipb_sitewide in field list on query as Unbreak Now! priority.

Changing priority after adding as train blocker

Wed, Oct 31, 7:45 PM · DBA, Anti-Harassment, Operations
thcipriani added a subtask for T206656: 1.33.0-wmf.2 deployment blockers: T208462: Error Unknown column ipb_sitewide in field list on query.
Wed, Oct 31, 7:44 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
thcipriani added a parent task for T208462: Error Unknown column ipb_sitewide in field list on query: T206656: 1.33.0-wmf.2 deployment blockers.
Wed, Oct 31, 7:44 PM · DBA, Anti-Harassment, Operations
thcipriani added a project to T208462: Error Unknown column ipb_sitewide in field list on query: Anti-Harassment (AHT Sprint 32).

Adding @dbarratt @Catrope @Mooeypoo based on a quick look at https://gerrit.wikimedia.org/r/q/ipb_sitewide

Wed, Oct 31, 7:43 PM · DBA, Anti-Harassment, Operations
thcipriani created T208462: Error Unknown column ipb_sitewide in field list on query.
Wed, Oct 31, 7:41 PM · DBA, Anti-Harassment, Operations

Tue, Oct 30

thcipriani added a comment to T208196: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated.

Should we have a follow up task to validate the json files against the MD5 sums?

Tue, Oct 30, 7:19 PM · Deployments, Release-Engineering-Team (Kanban), MediaWiki-extensions-LocalisationUpdate
thcipriani added a comment to T178663: Switch CI Docker Storage Driver to its own partition and to use devicemapper.

Added T207707 as a subtask since it is about getting additional storage space without specifying any movement of the storage driver.

Tue, Oct 30, 4:17 PM · Release-Engineering-Team (Kanban), Operations, Release Pipeline, Continuous-Integration-Infrastructure (shipyard)
thcipriani added a subtask for T178663: Switch CI Docker Storage Driver to its own partition and to use devicemapper: T207707: contint1001 store docker images on separate partition or disk.
Tue, Oct 30, 4:11 PM · Release-Engineering-Team (Kanban), Operations, Release Pipeline, Continuous-Integration-Infrastructure (shipyard)
thcipriani added a parent task for T207707: contint1001 store docker images on separate partition or disk: T178663: Switch CI Docker Storage Driver to its own partition and to use devicemapper.
Tue, Oct 30, 4:11 PM · Operations, Release-Engineering-Team, Continuous-Integration-Infrastructure

Mon, Oct 29

hashar awarded T208262: Ensure there are no hard-coded IPs in use for beta a Love token.
Mon, Oct 29, 9:10 PM · Patch-For-Review, cloud-services-team, Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure
thcipriani added a comment to T208262: Ensure there are no hard-coded IPs in use for beta.

A git grep for 10.68 yields a good amount of stuff related to beta-cluster.

Mon, Oct 29, 8:02 PM · Patch-For-Review, cloud-services-team, Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure
thcipriani created T208262: Ensure there are no hard-coded IPs in use for beta.
Mon, Oct 29, 7:58 PM · Patch-For-Review, cloud-services-team, Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure
thcipriani closed T208196: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated as Resolved.

So it looks like

Mon, Oct 29, 7:11 PM · Deployments, Release-Engineering-Team (Kanban), MediaWiki-extensions-LocalisationUpdate
thcipriani added a comment to P7734 (An Untitled Masterwork).
>>> os.utime('blazegraph-service-0.3.1-SNAPSHOT.war', None)                                                                                           
>>> stat = os.lstat('blazegraph-service-0.3.1-SNAPSHOT.war')                                                                                          
>>> os.utime('blazegraph-service-0.3.1-SNAPSHOT.war', (stat.st_atime, stat.st_mtime + 1))                                                             
Traceback (most recent call last):                                                                                                                    
  File "<stdin>", line 1, in <module>
OSError: [Errno 1] Operation not permitted: 'blazegraph-service-0.3.1-SNAPSHOT.war'
Mon, Oct 29, 5:35 PM
thcipriani created P7734 (An Untitled Masterwork).
Mon, Oct 29, 5:33 PM
thcipriani added a comment to T208196: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated.

After D983: Remove `scap l10n-update` we no longer have the code to just update the localisation cache on the master, but a full sync should get everything back in order.

I preformed a full sync in the hour after the issue, but still got a log message, I guess we can try again though!

Mon, Oct 29, 4:00 PM · Deployments, Release-Engineering-Team (Kanban), MediaWiki-extensions-LocalisationUpdate
thcipriani added a comment to T208196: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated.
08:35 gilles@deploy1001: sync aborted: T208088 Enable performance QuickSurvey shuffling (duration: 06m 30s)
Mon, Oct 29, 3:55 PM · Deployments, Release-Engineering-Team (Kanban), MediaWiki-extensions-LocalisationUpdate

Fri, Oct 26

thcipriani created P7723 (An Untitled Masterwork).
Fri, Oct 26, 6:41 PM
thcipriani updated subscribers of T208063: AssertionError: /src/.git/config has tabs.

Found https://gerrit.wikimedia.org/r/#/c/integration/config/+/469346/ adding @Legoktm

Fri, Oct 26, 3:59 PM · Patch-For-Review, Continuous-Integration-Config
thcipriani created T208063: AssertionError: /src/.git/config has tabs.
Fri, Oct 26, 3:56 PM · Patch-For-Review, Continuous-Integration-Config

Thu, Oct 25

thcipriani closed D1121: Fix "invalid escape sequence '\' (W605)".
Thu, Oct 25, 7:32 PM · Release-Engineering-Team
thcipriani committed rMSCAd7b8bfa2fa5e: Fix "invalid escape sequence '\' (W605)" (authored by thcipriani).
Fix "invalid escape sequence '\' (W605)"
Thu, Oct 25, 7:32 PM
thcipriani requested review of D1121: Fix "invalid escape sequence '\' (W605)".
Thu, Oct 25, 5:30 PM · Release-Engineering-Team

Wed, Oct 24

thcipriani committed rMSCA1b4b1f84401c: Remove '/' from include in sync-wikiversions (authored by thcipriani).
Remove '/' from include in sync-wikiversions
Wed, Oct 24, 3:12 PM
thcipriani closed D1120: Remove '/' from include in sync-wikiversions.
Wed, Oct 24, 3:12 PM · Release-Engineering-Team

Tue, Oct 23

thcipriani requested review of D1120: Remove '/' from include in sync-wikiversions.
Tue, Oct 23, 10:01 PM · Release-Engineering-Team