[23:00:04] <jouncebot>	 addshore, hashar, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for Evening SWAT (Max 6 patches) deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181001T2300).
[23:00:05] <jouncebot>	 davidwbarratt: A patch you scheduled for Evening SWAT (Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker.
[23:00:17] <davidwbarratt>	 I'm here!
[23:02:02] <davidwbarratt>	 who is SWATing?
[23:06:18] <davidwbarratt>	 hello?
[23:08:29] <dmaza>	 hello :p
[23:10:44] <davidwbarratt>	 ping addshore, hashar, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof:
[23:11:00] <MaxSem>	 I'm in a meeting, sorry
[23:11:03] <twentyafterfour>	 I can swat
[23:11:06] <davidwbarratt>	 no problem
[23:11:09] <wikibugs>	 10Operations, 10ops-ulsfo, 10netops: Interface errors on cr4-ulsfo:et-0/0/1 - https://phabricator.wikimedia.org/T205937 (10ayounsi) p:05Triage>03Normal
[23:11:11] <davidwbarratt>	 twentyafterfour thanks!
[23:11:27] <davidwbarratt>	 twentyafterfour it's a SQL patch to create a table
[23:12:04] <twentyafterfour>	 have we gotten any dba review of the change? 
[23:12:17] <davidwbarratt>	 yes, it's already merged into master
[23:12:51] <davidwbarratt>	 here's the task https://phabricator.wikimedia.org/T197144 and the DBA review(s) https://phabricator.wikimedia.org/T193449
[23:12:59] <davidwbarratt>	 but it is not a "schema change"
[23:13:17] <davidwbarratt>	 https://wikitech.wikimedia.org/wiki/Schema_changes#What_is_not_a_schema_change
[23:13:30] <twentyafterfour>	 ah I see
[23:13:43] <twentyafterfour>	 ok just creating a table should be fine, indeed 
[23:14:35] <RoanKattouw>	 Looks to me like you should run foreachwiki sql.php maintenance/postgres/archives/patch-ipblocks_restrictions-table.sql  , right?
[23:14:44] <RoanKattouw>	 Uh, except without the /postgres/ bit
[23:14:51] <wikibugs>	 (03CR) 10Cwhite: [C: 032] "NOOP on existing: https://puppet-compiler.wmflabs.org/compiler1001/12705/" [puppet] - 10https://gerrit.wikimedia.org/r/463868 (https://phabricator.wikimedia.org/T202782) (owner: 10Cwhite)
[23:14:52] <davidwbarratt>	 yep, just this: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/440871/21/maintenance/archives/patch-ipblocks_restrictions-table.sql
[23:15:36] <twentyafterfour>	 ok do we need to cherry pick anything to the branch or was this already merged prior to branch cut?
[23:15:43] * twentyafterfour looks at date of merge
[23:15:54] <RoanKattouw>	 "Included in" says wmf.22 and wmf.23
[23:16:00] <davidwbarratt>	 it's already merged, it should be deployed already
[23:16:05] <twentyafterfour>	 yeah cool 
[23:16:13] <twentyafterfour>	 so just the foreachwiki should do it 
[23:16:34] <Reedy>	 gonna have to be run from a codfw mtx host though, not deploy1001
[23:16:51] <RoanKattouw>	 What is the codfw maintenance host? mwmaint2001?
[23:16:55] <Reedy>	 yeah
[23:17:06] <Reedy>	 codfw is the active, but deployment still is in eqiad
[23:17:06] <RoanKattouw>	 Also, for completeness, let me correct my mistake:  foreachwiki sql.php maintenance/archives/patch-ipblocks_restrictions-table.sql
[23:17:11] <twentyafterfour>	 not sure if I have access to that? 
[23:17:14] * twentyafterfour checks
[23:17:18] <Reedy>	 you will :)
[23:17:20] <RoanKattouw>	 OK cool. I've never been on mwmaint2001, only on mwmaint1001
[23:17:32] <Reedy>	 mwmaint2001.codfw.wmnet obvs
[23:18:12] <davidwbarratt>	 I assume that creating the tables will persist after the datacenter switch-backover ?
[23:18:21] <twentyafterfour>	 !log creating ipblocks_restrictions table (command run on mwmaint2001: foreachwiki sql.php maintenance/archives/patch-ipblocks_restrictions-table.sql)
[23:18:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[23:18:31] <twentyafterfour>	 davidwbarratt: it should
[23:18:42] <davidwbarratt>	 good
[23:18:48] <Reedy>	 we're still replicating across DC
[23:18:53] <davidwbarratt>	 ah
[23:18:54] <Reedy>	 just the masters are currently in codfw
[23:19:06] <Reedy>	 https://dbtree.wikimedia.org/ is a nice illustration of it
[23:19:35] <James_F>	 RoanKattouw: Welcome, newbie. ;-)
[23:19:50] <twentyafterfour>	 ok the queries are running
[23:20:00] <davidwbarratt>	 yay!
[23:20:09] <twentyafterfour>	 so far all say " Query OK, 0 row(s) affected"
[23:20:30] <Reedy>	 That's normal IIRC
[23:20:44] <twentyafterfour>	 is there an easy way to confirm that the table is created?
[23:21:01] <Reedy>	 sql aawiki
[23:21:07] <Reedy>	 explain ipblocks_restrictions;
[23:21:16] <Reedy>	 I've just confirmed it's there on the eqiad hosts
[23:21:40] <twentyafterfour>	 Cool, Thanks Reedy!
[23:21:46] <James_F>	 Table exists on codfw aawiki too.
[23:21:54] <twentyafterfour>	 still running, there are a lot of wikis ;) 
[23:21:56] <Reedy>	 James_F: I'd be amazed if they didn't :P
[23:22:03] <Reedy>	 That would be some witchcraft
[23:22:36] <twentyafterfour>	 replag == witchcraft
[23:22:48] <James_F>	 Reedy: I've seen some stuff you wouldn't believe. RepLag, C-beams glittering off the belt of Orion, etc.
[23:22:54] <James_F>	 Bah, twentyafterfour got there first.
[23:23:02] <twentyafterfour>	 hah 
[23:23:02] <Reedy>	 Are you planning on deploying code that uses that table soon after?
[23:23:30] <davidwbarratt>	 we can't until the schema change is complete (i.e. the patch that changes a column)
[23:24:00] <davidwbarratt>	 https://phabricator.wikimedia.org/T204006
[23:24:34] <Reedy>	 fair
[23:27:50] <icinga-wm>	 PROBLEM - puppet last run on wdqs1010 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[enforce-users-groups-cleanup]
[23:30:01] <davidwbarratt>	 twentyafterfour still running?
[23:30:21] <Reedy>	 probably...  over 900 wikis takes a while :)
[23:30:39] <twentyafterfour>	 davidwbarratt: yeah almost done I think 
[23:30:57] <twentyafterfour>	 at zawiki
[23:31:06] <twentyafterfour>	 and now it's done
[23:31:16] <mutante>	 herron: is the wdqs error about the new group?
[23:31:24] <twentyafterfour>	 !log finished creating database tables
[23:31:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[23:32:48] <wikibugs>	 (03Abandoned) 10MacFan4000: Remove MW 1.29 from ExtDist as it is now no longer supported [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440745 (owner: 10MacFan4000)
[23:33:25] <mutante>	 herron: ignore it. it is definitely not. i see an unrelated issue
[23:34:05] <herron>	 mutante ok, I didn’t merge that yet fwiw
[23:34:26] <mutante>	 herron: yea, i see it. it just seemed like it because "groups-cleanup" 
[23:34:37] <davidwbarratt>	 YAY!
[23:34:46] <davidwbarratt>	 thanks twentyafterfour !
[23:34:51] <mutante>	 but the issue that somebody manually added a user or installed softare
[23:35:06] <twentyafterfour>	 davidwbarratt: You're welcome. Thanks to Reedy and RoanKattouw for helping out.