Page MenuHomePhabricator

brennen (Brennen Bearnes)
UserAdministrator

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Feb 3 2019, 8:29 PM (129 w, 2 d)
Roles
Administrator
Availability
Available
IRC Nick
brennen
LDAP User
Brennen Bearnes
MediaWiki User
BBearnes (WMF) [ Global Accounts ]

Recent Activity

Yesterday

brennen closed T281121: logspam-watch broken as Resolved.
Tue, Jul 27, 11:37 PM · Release-Engineering-Team (Doing), User-brennen
brennen moved T281121: logspam-watch broken from Needs/Waiting Review to Done / Defunct on the User-brennen board.
Tue, Jul 27, 11:36 PM · Release-Engineering-Team (Doing), User-brennen
brennen added a comment to T274462: Logging for GitLab.

Thanks for all of the assistance!

Tue, Jul 27, 11:30 PM · observability, SRE Observability (FY2021/2022-Q1), serviceops, Wikimedia-Logstash, User-brennen, GitLab (Initialization)
brennen moved T286958: Document long-term requirements for GitLab job runners from INBOX to Seen on the Release-Engineering-Team board.
Tue, Jul 27, 5:28 PM · Release-Engineering-Team (Seen), User-brennen, GitLab
brennen updated the task description for T286958: Document long-term requirements for GitLab job runners.
Tue, Jul 27, 5:17 PM · Release-Engineering-Team (Seen), User-brennen, GitLab
brennen moved T287504: Puppetize GitLab runner configuration from Backlog to Initialization on the GitLab board.
Tue, Jul 27, 5:13 PM · Patch-For-Review, GitLab (Initialization), User-brennen, Release-Engineering-Team (Doing)
brennen moved T287504: Puppetize GitLab runner configuration from Backlog to Watching on the User-brennen board.
Tue, Jul 27, 5:13 PM · Patch-For-Review, GitLab (Initialization), User-brennen, Release-Engineering-Team (Doing)
brennen added a project to T287504: Puppetize GitLab runner configuration: User-brennen.
Tue, Jul 27, 5:12 PM · Patch-For-Review, GitLab (Initialization), User-brennen, Release-Engineering-Team (Doing)
brennen added a subtask for T287279: Experiment with gitlab-runners project and provision a pool of shared runners for general use: T287504: Puppetize GitLab runner configuration.
Tue, Jul 27, 5:12 PM · User-brennen, Release-Engineering-Team (Doing), GitLab (Initialization)
brennen added a parent task for T287504: Puppetize GitLab runner configuration: T287279: Experiment with gitlab-runners project and provision a pool of shared runners for general use.
Tue, Jul 27, 5:12 PM · Patch-For-Review, GitLab (Initialization), User-brennen, Release-Engineering-Team (Doing)
brennen moved T287279: Experiment with gitlab-runners project and provision a pool of shared runners for general use from Backlog to In Progress on the GitLab (Initialization) board.
Tue, Jul 27, 4:19 PM · User-brennen, Release-Engineering-Team (Doing), GitLab (Initialization)

Mon, Jul 26

brennen updated subscribers of T287279: Experiment with gitlab-runners project and provision a pool of shared runners for general use.

Our rough plan at the moment is to puppetize installation of gitlab-runner on nodes, and leave a manual configuration step for registering each as a Docker executor.

Mon, Jul 26, 8:47 PM · User-brennen, Release-Engineering-Team (Doing), GitLab (Initialization)

Fri, Jul 23

brennen created T287279: Experiment with gitlab-runners project and provision a pool of shared runners for general use.
Fri, Jul 23, 7:25 PM · User-brennen, Release-Engineering-Team (Doing), GitLab (Initialization)

Thu, Jul 22

dduvall awarded T287211: Figure out the future of (or replacements for) PipelineLib in a GitLab world a Pirate Logo token.
Thu, Jul 22, 8:55 PM · Release-Engineering-Team (Doing), Continuous-Integration-Infrastructure, User-brennen, Release Pipeline, GitLab
brennen changed the subtype of T287117: Establish a routine GitLab deployment / update window from "Bug Report" to "Task".
Thu, Jul 22, 8:46 PM · Release-Engineering-Team (Next), GitLab (Initialization)
brennen created T287211: Figure out the future of (or replacements for) PipelineLib in a GitLab world.
Thu, Jul 22, 8:43 PM · Release-Engineering-Team (Doing), Continuous-Integration-Infrastructure, User-brennen, Release Pipeline, GitLab

Wed, Jul 21

brennen renamed T287122: resolve gerrit.config disprepancy between managed config and gerrit init from resolve gerrit.config dispredancy between managed config and gerrit init to resolve gerrit.config disprepancy between managed config and gerrit init.
Wed, Jul 21, 9:10 PM · Release-Engineering-Team (Doing), Patch-For-Review, Gerrit
brennen created T287117: Establish a routine GitLab deployment / update window.
Wed, Jul 21, 8:22 PM · Release-Engineering-Team (Next), GitLab (Initialization)
brennen created T287099: Gerrit download dialog is missing branch info after upgrade to 3.2.11.
Wed, Jul 21, 4:29 PM · User-brennen, Release-Engineering-Team, Gerrit

Tue, Jul 20

brennen removed a project from T264231: Investigate whether issues, operations, wikis, etc. can be disabled globally on GitLab: GitLab-Test.

See Draft: Add tools for configuring all projects. Needs a bit of work, but presumably we can run configure-projects as a periodic job.

Tue, Jul 20, 9:43 PM · GitLab (Initialization), Release-Engineering-Team (Doing), User-brennen
brennen created User-jeena.
Tue, Jul 20, 7:32 PM
brennen moved T274954: gitlab-test: undefined method `user_cas_omniauth_authorize_path' with omniauth_auto_sign_in_with_provider from Backlog to Completed on the GitLab board.
Tue, Jul 20, 6:46 PM · Release-Engineering-Team (Doing), User-brennen, GitLab-Test, GitLab
brennen closed T274954: gitlab-test: undefined method `user_cas_omniauth_authorize_path' with omniauth_auto_sign_in_with_provider as Resolved.

Since we're currently using a value of cas3, I'm assuming this was just an error on my part.

Tue, Jul 20, 6:46 PM · Release-Engineering-Team (Doing), User-brennen, GitLab-Test, GitLab
brennen closed T274954: gitlab-test: undefined method `user_cas_omniauth_authorize_path' with omniauth_auto_sign_in_with_provider, a subtask of T274461: Define auth strategy for GitLab, as Resolved.
Tue, Jul 20, 6:45 PM · Release-Engineering-Team (Doing), SRE, GitLab (Initialization), User-brennen
brennen claimed T265617: Experiment with GitLab-Phabricator integration.
Tue, Jul 20, 6:43 PM · Release-Engineering-Team (Doing), Phabricator, GitLab, User-brennen, GitLab-Test
brennen moved T265617: Experiment with GitLab-Phabricator integration from Next to In Progress on the User-brennen board.
Tue, Jul 20, 6:42 PM · Release-Engineering-Team (Doing), Phabricator, GitLab, User-brennen, GitLab-Test
brennen placed T265617: Experiment with GitLab-Phabricator integration up for grabs.
Tue, Jul 20, 6:42 PM · Release-Engineering-Team (Doing), Phabricator, GitLab, User-brennen, GitLab-Test
brennen moved T274462: Logging for GitLab from Watching to Needs/Waiting Review on the User-brennen board.
Tue, Jul 20, 6:41 PM · observability, SRE Observability (FY2021/2022-Q1), serviceops, Wikimedia-Logstash, User-brennen, GitLab (Initialization)

Mon, Jul 19

brennen closed T285907: gitlab-settings: Figure out how to handle options that take an array of strings, a subtask of T284336: Create a GitLab settings script / repo, as Resolved.
Mon, Jul 19, 11:12 PM · Release-Engineering-Team (Doing), User-brennen, GitLab (Initialization)
brennen closed T285907: gitlab-settings: Figure out how to handle options that take an array of strings as Resolved.
Mon, Jul 19, 11:12 PM · Release-Engineering-Team (Next), User-brennen, GitLab
brennen added a subtask for T284336: Create a GitLab settings script / repo: T285907: gitlab-settings: Figure out how to handle options that take an array of strings.
Mon, Jul 19, 11:12 PM · Release-Engineering-Team (Doing), User-brennen, GitLab (Initialization)
brennen added a parent task for T285907: gitlab-settings: Figure out how to handle options that take an array of strings: T284336: Create a GitLab settings script / repo.
Mon, Jul 19, 11:12 PM · Release-Engineering-Team (Next), User-brennen, GitLab
brennen closed T284336: Create a GitLab settings script / repo as Resolved.

Multiple-value settings: https://gitlab.wikimedia.org/releng/gitlab-settings/-/merge_requests/1

Mon, Jul 19, 11:11 PM · Release-Engineering-Team (Doing), User-brennen, GitLab (Initialization)
brennen created T286958: Document long-term requirements for GitLab job runners.
Mon, Jul 19, 9:16 PM · Release-Engineering-Team (Seen), User-brennen, GitLab

Fri, Jul 16

brennen added a comment to T274462: Logging for GitLab.

Working on a patch; it's a bit fiddly because the omnibus config wants to just jam the value of nginx[log_format] inside a single-quoted string, and I'm not sure if it allows for setting the escape, but there's probably some workaround.

Fri, Jul 16, 6:59 PM · observability, SRE Observability (FY2021/2022-Q1), serviceops, Wikimedia-Logstash, User-brennen, GitLab (Initialization)
brennen added a comment to T274303: dev-images +2 rights and Docker registry credentials for FR-Tech.

Ah cool gotcha... on that point, yes, if it'd be possible then to publish the images from these merged changes, that'd be fantastic:

Fri, Jul 16, 6:14 PM · Release-Engineering-Team (Radar), Gerrit-Privilege-Requests, User-brennen, Developer Productivity, dev-images, FR-Docker, Fundraising-Backlog
brennen added a comment to T274462: Logging for GitLab.

Ok, it looks like we just need to set nginx['log_format'] to that value in /etc/gitlab.rb.

Fri, Jul 16, 6:09 PM · observability, SRE Observability (FY2021/2022-Q1), serviceops, Wikimedia-Logstash, User-brennen, GitLab (Initialization)
brennen added a comment to T284195: Add a Swift image to Wikimedia's docker registry for testing.

(Apologies - above log message is an error on my part, was attempting to reuse that log line for a different change.)

Fri, Jul 16, 5:45 PM · Release-Engineering-Team, dev-images, User-brennen, Product-Data-Infrastructure, Metrics-Platform, Continuous-Integration-Infrastructure
brennen added a comment to T274462: Logging for GitLab.

Is it possible to configure the nginx component to emit ECS-compatible access logs natively?

Fri, Jul 16, 5:42 PM · observability, SRE Observability (FY2021/2022-Q1), serviceops, Wikimedia-Logstash, User-brennen, GitLab (Initialization)

Thu, Jul 15

brennen added a comment to T274303: dev-images +2 rights and Docker registry credentials for FR-Tech.

My understanding is there's still a manual step of actually publishing these image (unless there was a recent change). I might ask @jeena or @brennen if there have been/are planned any changes in this space.

Thu, Jul 15, 4:53 PM · Release-Engineering-Team (Radar), Gerrit-Privilege-Requests, User-brennen, Developer Productivity, dev-images, FR-Docker, Fundraising-Backlog

Wed, Jul 14

brennen moved T284195: Add a Swift image to Wikimedia's docker registry for testing from Backlog to Watching on the User-brennen board.
Wed, Jul 14, 11:05 PM · Release-Engineering-Team, dev-images, User-brennen, Product-Data-Infrastructure, Metrics-Platform, Continuous-Integration-Infrastructure
brennen added projects to T284195: Add a Swift image to Wikimedia's docker registry for testing: User-brennen, dev-images, Release-Engineering-Team.

Published successfully I think - let me know if it's not working.

Wed, Jul 14, 11:05 PM · Release-Engineering-Team, dev-images, User-brennen, Product-Data-Infrastructure, Metrics-Platform, Continuous-Integration-Infrastructure
brennen removed a subtask for T281153: 1.37.0-wmf.12 deployment blockers: T286521: Deadlock found when trying to get lock (UserOptionsManager::saveOptionsQuery).
Wed, Jul 14, 10:35 PM · User-brennen, Release-Engineering-Team (Doing), Release, Train Deployments
brennen removed a subtask for T281155: 1.37.0-wmf.14 deployment blockers: T286521: Deadlock found when trying to get lock (UserOptionsManager::saveOptionsQuery).
Wed, Jul 14, 10:35 PM · Release-Engineering-Team (Doing), Release, Train Deployments
brennen added a subtask for T281156: 1.37.0-wmf.15 deployment blockers: T286521: Deadlock found when trying to get lock (UserOptionsManager::saveOptionsQuery).
Wed, Jul 14, 10:35 PM · Patch-For-Review, Release-Engineering-Team (Doing), Release, Train Deployments
brennen edited parent tasks for T286521: Deadlock found when trying to get lock (UserOptionsManager::saveOptionsQuery), added: T281156: 1.37.0-wmf.15 deployment blockers; removed: T281155: 1.37.0-wmf.14 deployment blockers, T281153: 1.37.0-wmf.12 deployment blockers.
Wed, Jul 14, 10:35 PM · MW-1.37-notes (1.37.0-wmf.14; 2021-07-12), Wikimedia-production-error, Patch-For-Review, Platform Team Workboards (MW Expedition), MediaWiki-User-management
brennen added a comment to T286521: Deadlock found when trying to get lock (UserOptionsManager::saveOptionsQuery).

it's been 9 of them in the last 3 hours, and they're happening on wmf.12 as well. Does this actually constitute a train blocker?

Wed, Jul 14, 10:35 PM · MW-1.37-notes (1.37.0-wmf.14; 2021-07-12), Wikimedia-production-error, Patch-For-Review, Platform Team Workboards (MW Expedition), MediaWiki-User-management
brennen added a comment to T281155: 1.37.0-wmf.14 deployment blockers.

Late (US) Wednesday afternoon status: After work on T286521, we've rolled forward to group1. Generally, things seem quiet, although T286521 continues to crop up.

Wed, Jul 14, 9:46 PM · Release-Engineering-Team (Doing), Release, Train Deployments
brennen added a comment to T286521: Deadlock found when trying to get lock (UserOptionsManager::saveOptionsQuery).

Still seeing instances of this since the most recent patch was deployed:

Wed, Jul 14, 9:38 PM · MW-1.37-notes (1.37.0-wmf.14; 2021-07-12), Wikimedia-production-error, Patch-For-Review, Platform Team Workboards (MW Expedition), MediaWiki-User-management
brennen moved T275072: LocalFileLockError: Could not acquire lock for "mwstore: …" (via ApiUpload.php) from Backlog to Logs/Train on the User-brennen board.
Wed, Jul 14, 9:32 PM · User-brennen, MediaWiki-Uploading, MediaWiki-File-management, Wikimedia-production-error, Commons
brennen added a project to T275072: LocalFileLockError: Could not acquire lock for "mwstore: …" (via ApiUpload.php): User-brennen.
Wed, Jul 14, 9:32 PM · User-brennen, MediaWiki-Uploading, MediaWiki-File-management, Wikimedia-production-error, Commons

Tue, Jul 13

brennen added a comment to T286521: Deadlock found when trying to get lock (UserOptionsManager::saveOptionsQuery).

Frequency seems down, but @dancy just pointed out ~3 of these in the last hour.

Tue, Jul 13, 8:00 PM · MW-1.37-notes (1.37.0-wmf.14; 2021-07-12), Wikimedia-production-error, Patch-For-Review, Platform Team Workboards (MW Expedition), MediaWiki-User-management
brennen added a comment to T281155: 1.37.0-wmf.14 deployment blockers.

Note: During sync, you're likely to see something like:

Tue, Jul 13, 5:26 PM · Release-Engineering-Team (Doing), Release, Train Deployments
brennen updated subscribers of T286521: Deadlock found when trying to get lock (UserOptionsManager::saveOptionsQuery).

Should this be backported to both wmf.12 and .14?

Tue, Jul 13, 3:36 PM · MW-1.37-notes (1.37.0-wmf.14; 2021-07-12), Wikimedia-production-error, Patch-For-Review, Platform Team Workboards (MW Expedition), MediaWiki-User-management

Mon, Jul 12

brennen added a comment to T286521: Deadlock found when trying to get lock (UserOptionsManager::saveOptionsQuery).

Noting that, given the rate of these, I probably observed a handful while doing log triage during 1.37.0-wmf.12 (T281153), and mentally filed as "typical low-level database error". I could use a better working model of which db errors represent a code regression (and might be worth documenting a bit for other deployers).

Mon, Jul 12, 10:17 PM · MW-1.37-notes (1.37.0-wmf.14; 2021-07-12), Wikimedia-production-error, Patch-For-Review, Platform Team Workboards (MW Expedition), MediaWiki-User-management
brennen moved T273802: Update/organize train deployment and related policy documentation from In Progress to Backlog on the User-brennen board.
Mon, Jul 12, 7:21 PM · Release-Engineering-Team (Doing), Documentation, User-brennen

Sun, Jul 4

brennen removed a subtask for T281153: 1.37.0-wmf.12 deployment blockers: T286140: AbuseLog no longer recording revids of saved edits.
Sun, Jul 4, 5:50 PM · User-brennen, Release-Engineering-Team (Doing), Release, Train Deployments
brennen removed a subtask for T281155: 1.37.0-wmf.14 deployment blockers: T286140: AbuseLog no longer recording revids of saved edits.
Sun, Jul 4, 5:50 PM · Release-Engineering-Team (Doing), Release, Train Deployments
brennen removed parent tasks for T286140: AbuseLog no longer recording revids of saved edits: T281155: 1.37.0-wmf.14 deployment blockers, T281153: 1.37.0-wmf.12 deployment blockers.
Sun, Jul 4, 5:49 PM · MW-1.37-notes (1.37.0-wmf.15; 2021-07-19), User-DannyS712, Regression, AbuseFilter
brennen lowered the priority of T286140: AbuseLog no longer recording revids of saved edits from Unbreak Now! to Needs Triage.

Backport deployed, confirmed that enwiki AbuseLog now shows diff links for new edits.

Sun, Jul 4, 5:49 PM · MW-1.37-notes (1.37.0-wmf.15; 2021-07-19), User-DannyS712, Regression, AbuseFilter
brennen added a comment to T281153: 1.37.0-wmf.12 deployment blockers.

Doing an emergency deploy shortly for T286140.

Sun, Jul 4, 5:20 PM · User-brennen, Release-Engineering-Team (Doing), Release, Train Deployments
brennen added a comment to T286140: AbuseLog no longer recording revids of saved edits.

I can deploy; going ahead as I think it will be clear from reproduction steps above if the fix worked.

Sun, Jul 4, 5:19 PM · MW-1.37-notes (1.37.0-wmf.15; 2021-07-19), User-DannyS712, Regression, AbuseFilter

Fri, Jul 2

brennen added a comment to T274462: Logging for GitLab.

Thanks for digging into the layout of this so thoroughly.

Fri, Jul 2, 11:28 PM · observability, SRE Observability (FY2021/2022-Q1), serviceops, Wikimedia-Logstash, User-brennen, GitLab (Initialization)
brennen closed T276148: SSH Access of Git data in GitLab as Resolved.

Port is open and in use.

Fri, Jul 2, 9:32 PM · Patch-For-Review, Release-Engineering-Team (Doing), SRE, User-brennen, GitLab (Initialization)
brennen closed T276148: SSH Access of Git data in GitLab, a subtask of T276144: open firewall ports on gitlab1001.wikimedia.org (was: Port map of how Gitlab is accessed), as Resolved.
Fri, Jul 2, 9:32 PM · SRE, Traffic, Patch-For-Review, User-brennen, GitLab (Initialization), Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3))
brennen moved T262618: Proof of concept for cross-repo dependencies from Backlog to Migration on the GitLab board.
Fri, Jul 2, 9:29 PM · Release-Engineering-Team (Seen), GitLab, User-brennen, GitLab-Test
brennen closed T264130: Experiment with GitLab 2-factor authentication as Resolved.

Does GitLab 2fa work in conjunction with LDAP?

Fri, Jul 2, 9:29 PM · GitLab, User-brennen, GitLab-Test
brennen moved T268283: Figure out submodule updating in GitLab from Backlog to Migration on the GitLab board.
Fri, Jul 2, 9:28 PM · Release-Engineering-Team (Seen), User-brennen, GitLab
brennen closed T281153: 1.37.0-wmf.12 deployment blockers as Resolved.

Many thanks to @Jdforrester-WMF et al. for handling of emergency deploys during the US night.

Fri, Jul 2, 4:15 PM · User-brennen, Release-Engineering-Team (Doing), Release, Train Deployments

Thu, Jul 1

brennen renamed T285913: Request creation of gitlab-runners VPS project from Request creation of gitlab-workers VPS project to Request creation of gitlab-runners VPS project.
Thu, Jul 1, 11:32 PM · Release-Engineering-Team (Radar), User-brennen, GitLab (Initialization), Cloud-VPS (Project-requests)
brennen added a comment to T285913: Request creation of gitlab-runners VPS project.

If we're bikeshedding the naming, I don't have strong feelings here, but "gitlab-runners" does align exactly with the name of the software we intend to run.

Thu, Jul 1, 11:01 PM · Release-Engineering-Team (Radar), User-brennen, GitLab (Initialization), Cloud-VPS (Project-requests)
brennen added a project to T285978: AbuseFilterCachingParser: PHP Warning: mb_strpos(): Offset not contained in string: Platform Engineering.
Thu, Jul 1, 8:20 PM · Platform Team Workboards (External Code Reviews), Patch-For-Review, User-brennen, AbuseFilter, Wikimedia-production-error
brennen created T285978: AbuseFilterCachingParser: PHP Warning: mb_strpos(): Offset not contained in string.
Thu, Jul 1, 8:19 PM · Platform Team Workboards (External Code Reviews), Patch-For-Review, User-brennen, AbuseFilter, Wikimedia-production-error
brennen lowered the priority of T285951: Some section links in search results are redlinks from Unbreak Now! to Needs Triage.
Thu, Jul 1, 7:57 PM · MW-1.37-notes (1.37.0-wmf.14; 2021-07-12), User-brennen, Regression, CirrusSearch, Discovery-Search
brennen removed a subtask for T281153: 1.37.0-wmf.12 deployment blockers: T285951: Some section links in search results are redlinks.
Thu, Jul 1, 7:57 PM · User-brennen, Release-Engineering-Team (Doing), Release, Train Deployments
brennen removed a parent task for T285951: Some section links in search results are redlinks: T281153: 1.37.0-wmf.12 deployment blockers.
Thu, Jul 1, 7:57 PM · MW-1.37-notes (1.37.0-wmf.14; 2021-07-12), User-brennen, Regression, CirrusSearch, Discovery-Search
brennen added a comment to T285951: Some section links in search results are redlinks.

Confirmed fixed on mwdebug2001; synching.

Thu, Jul 1, 7:32 PM · MW-1.37-notes (1.37.0-wmf.14; 2021-07-12), User-brennen, Regression, CirrusSearch, Discovery-Search
brennen added a comment to T285966: Minerva views missing content styles on wmf.11 wikis that were briefly on wmf.12.

Noticing some broken styles in links at https://commons.m.wikimedia.org/wiki/Main_Page

Thu, Jul 1, 7:28 PM · Performance-Team, MinervaNeue
brennen added a comment to T281153: 1.37.0-wmf.12 deployment blockers.

Rolling back to group0 for T285951, per IRC discussion.

Thu, Jul 1, 4:28 PM · User-brennen, Release-Engineering-Team (Doing), Release, Train Deployments
brennen moved T285951: Some section links in search results are redlinks from Backlog to Logs/Train on the User-brennen board.
Thu, Jul 1, 3:06 PM · MW-1.37-notes (1.37.0-wmf.14; 2021-07-12), User-brennen, Regression, CirrusSearch, Discovery-Search
brennen triaged T285951: Some section links in search results are redlinks as Unbreak Now! priority.
Thu, Jul 1, 3:06 PM · MW-1.37-notes (1.37.0-wmf.14; 2021-07-12), User-brennen, Regression, CirrusSearch, Discovery-Search
brennen added a parent task for T285951: Some section links in search results are redlinks: T281153: 1.37.0-wmf.12 deployment blockers.
Thu, Jul 1, 3:05 PM · MW-1.37-notes (1.37.0-wmf.14; 2021-07-12), User-brennen, Regression, CirrusSearch, Discovery-Search
brennen added a subtask for T281153: 1.37.0-wmf.12 deployment blockers: T285951: Some section links in search results are redlinks.
Thu, Jul 1, 3:05 PM · User-brennen, Release-Engineering-Team (Doing), Release, Train Deployments
brennen added a comment to T281153: 1.37.0-wmf.12 deployment blockers.

It looks like T285951: Some section links in search results are redlinks is a regression in wmf.12. I don’t know if it’s significant enough to block the train.

Thu, Jul 1, 3:05 PM · User-brennen, Release-Engineering-Team (Doing), Release, Train Deployments
brennen moved T285913: Request creation of gitlab-runners VPS project from Limbo to Watching/External on the Release-Engineering-Team (Radar) board.
Thu, Jul 1, 12:42 AM · Release-Engineering-Team (Radar), User-brennen, GitLab (Initialization), Cloud-VPS (Project-requests)
brennen moved T285913: Request creation of gitlab-runners VPS project from INBOX to Radar on the Release-Engineering-Team board.
Thu, Jul 1, 12:42 AM · Release-Engineering-Team (Radar), User-brennen, GitLab (Initialization), Cloud-VPS (Project-requests)
brennen moved T285913: Request creation of gitlab-runners VPS project from Backlog to In Progress on the GitLab (Initialization) board.
Thu, Jul 1, 12:41 AM · Release-Engineering-Team (Radar), User-brennen, GitLab (Initialization), Cloud-VPS (Project-requests)
brennen moved T285913: Request creation of gitlab-runners VPS project from Backlog to Needs/Waiting Review on the User-brennen board.
Thu, Jul 1, 12:41 AM · Release-Engineering-Team (Radar), User-brennen, GitLab (Initialization), Cloud-VPS (Project-requests)
brennen created T285913: Request creation of gitlab-runners VPS project.
Thu, Jul 1, 12:41 AM · Release-Engineering-Team (Radar), User-brennen, GitLab (Initialization), Cloud-VPS (Project-requests)

Wed, Jun 30

brennen added a comment to T281153: 1.37.0-wmf.12 deployment blockers.

End-of-US-day status: Things are looking pretty stable on group1, expect to proceed as usual to all wikis tomorrow.

Wed, Jun 30, 11:00 PM · User-brennen, Release-Engineering-Team (Doing), Release, Train Deployments
brennen moved T285907: gitlab-settings: Figure out how to handle options that take an array of strings from INBOX to Next on the Release-Engineering-Team board.
Wed, Jun 30, 10:49 PM · Release-Engineering-Team (Next), User-brennen, GitLab
brennen created T285907: gitlab-settings: Figure out how to handle options that take an array of strings.
Wed, Jun 30, 10:26 PM · Release-Engineering-Team (Next), User-brennen, GitLab
brennen moved T284336: Create a GitLab settings script / repo from In Progress to Done / Defunct on the User-brennen board.
Wed, Jun 30, 10:25 PM · Release-Engineering-Team (Doing), User-brennen, GitLab (Initialization)
brennen moved T284336: Create a GitLab settings script / repo from In Progress to Done on the GitLab (Initialization) board.

This is now at https://gitlab.wikimedia.org/releng/gitlab-settings/ - GPL3 license in repo.

Wed, Jun 30, 10:25 PM · Release-Engineering-Team (Doing), User-brennen, GitLab (Initialization)
brennen moved T248359: Clarify MediaWiki Docker installation instructions regarding re-installation and access after installation from Next to Backlog on the User-brennen board.
Wed, Jun 30, 3:08 PM · User-brennen, Documentation, MediaWiki-Docker
brennen moved T273100: dev-images: upgrade images for MediaWiki-Docker from wikimedia-stretch to wikimedia-buster from Next to In Progress on the User-brennen board.
Wed, Jun 30, 3:08 PM · Patch-For-Review, Release-Engineering-Team (Doing), dev-images, MediaWiki-Docker, User-brennen
brennen moved T255197: scap's logstash_checker.py is blissfully unaware of any logstash indexing latency from Backlog to Watching on the User-brennen board.
Wed, Jun 30, 3:07 PM · Release-Engineering-Team (Seen), User-brennen, Sustainability (Incident Followup), Scap, SRE
brennen moved T279013: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" from Backlog to Watching on the User-brennen board.
Wed, Jun 30, 3:07 PM · serviceops, User-brennen, Phabricator
brennen moved T282173: Possible uptick in "DBTransactionSizeError: Transaction spent [n] second(s) in writes, exceeding the limit of 3" from Backlog to Logs/Train on the User-brennen board.
Wed, Jun 30, 3:07 PM · Platform Team Workboards (Clinic Duty Team), User-brennen, Wikimedia-production-error
brennen moved T279322: Design m8s deployment workflows and tooling from Backlog to Watching on the User-brennen board.
Wed, Jun 30, 3:06 PM · User-brennen, Patch-For-Review, Release-Engineering-Team (Doing), MW-on-K8s, Release Pipeline