I have renamed all the tables on enwiki on db1089:
root@db1089.eqiad.wmnet[enwiki]> show tables like 'T174%'; +-----------------------------+ | Tables_in_enwiki (T174%) | +-----------------------------+ | T174802_ep_articles | | T174802_ep_cas | | T174802_ep_courses | | T174802_ep_events | | T174802_ep_instructors | | T174802_ep_oas | | T174802_ep_orgs | | T174802_ep_revisions | | T174802_ep_students | | T174802_ep_users_per_course | +-----------------------------+ 10 rows in set (0.00 sec)
@paravoid gave us some food for thought:
stuck at "loading ramdisk" is sometimes an indication of misconfigured serial redirection after boot basically when Linux and the BIOS are fighting over control of the serial port
Same thing just happened with db1106 (PowerEdge R630 - same chassis as db2085)
@MoritzMuehlenhoff can you help us with the approach you mentioned at T214840#4918369 ?
@Cmjohnson I can take care of the installations once you've done the RAID and added DNS and pxeboot entries with the MACs :-)
In T210478#4942496, @Marostegui wrote:After having al the sections ready and compressed on all hosts, there is one thought I had, where to leave staging database, either dbstore1003 or dbstore1005.
This is the current situation:
dbstore1003: s1, s5, s7, staging
root@dbstore1003:/srv# df -hT /srv Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/tank-data xfs 4.4T 2.7T 1.8T 61% /srv root@dbstore1003:/srv# du -sh * 926G sqldata.s1 695G sqldata.s5 914G sqldata.s7 144G sqldata.stagingdbstore1005: s6, s8, x1
root@dbstore1005:/srv# df -hT /srv Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/tank-data xfs 4.4T 1.7T 2.7T 39% /srv root@dbstore1005:/srv# du -sh * 479G sqldata.s6 1.1T sqldata.s8 172G sqldata.x1I think it would make sense to move staging to dbstore1005
s8 eqiad progress
The clicktracking no longer exist on either s1 master or dbstore1002:
root@db1067.eqiad.wmnet[enwiki]> show tables like '%click%'; Empty set (0.00 sec)
After having al the sections ready and compressed on all hosts, there is one thought I had, where to leave staging database, either dbstore1003 or dbstore1005.
Now that we know that the biggest and and most painful table (as it was Aria and it was huge - around 180GB) was killed T215450: Sqoop staging.mep_word_persistence to HDFS and drop the table from dbstore1002
I have tested a migration process for the staging database, to see how much we'd need to have it on read only.
I restarted icinga and they are recovering and downtimes are working again
All the passive checks went awol just now.
I tested a downtime to db1100 and it didn't work (either using icinga-downtime or the icinga web ui)
While tailing the logs during the downtimes test I got:
[1549868178] External command error: Malformed command [1549868178] External command error: Malformed command [1549868178] External command error: Malformed command [1549868178] External command error: Malformed command [1549868178] External command error: Malformed command [1549868178] External command error: Malformed command [1549868178] External command error: Malformed command [1549868178] External command error: Malformed command [1549868178] External command error: Malformed command [1549868178] External command error: Malformed command [1549868178] External command error: Malformed command [1549868178] External command error: Malformed command [1549868178] External command error: Malformed command
I am fine going to unsigned for rc_id . Once tables.sql is merged with this change, just add Schema-change-in-production as a tag (https://wikitech.wikimedia.org/wiki/Schema_changes#Workflow_of_a_schema_change) and we can take care of it :)
In T63111#4941569, @Krinkle wrote:Re-opening so as to let the DBAs triage this. One question I wasn't able to answer quickly is: What units and signed-ness do all our current integer fields use in core's default schema, and in WMF production?
Thanks for clarifying that @Volans!
Then probably failing over to another host is a good idea so we can debug icinga1001 without having service interruptions
Sorry, I wasn't available during the weekend.
I am normally available from Monday to Friday from 7:00 UTC to 16:00 UTC
In T214760#4941652, @Volans wrote:I propose to failover to icinga2001 until we find out what's wrong with this one and we fix it.
Thoughts?
Another crash just happened:
InnoDB: We intentionally crash the server, because it appears to be hung. 2019-02-10 09:54:08 7fa2cfffe700 InnoDB: Assertion failure in thread 140337251084032 in file srv0srv.cc line 2200 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 190210 9:54:08 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.
I have disabled notifications for lag checks (only for lag ones) for dbstore1002, as they are very noisy. They will show up on icinga, but not on IRC.
In T215107#4940347, @1997kB wrote:@Marostegui Well, they're the one being renamed, so I think they just responded without realising that it have to do by steward or renamer.
In T215107#4940262, @Wilfredor wrote:It could be done on any moment, preferably a Friday. Thanks
And if crashed again with the same error:
/admin1/system1/logs1/log1-> show record13
This host is under warranty until April 14, 2019 so we might want to try to debug this before it expires in case we need some replacement CPU or mainboard.
/admin1/system1/logs1/log1-> show record27
This can be closed as there are no more Aria tables on the staging database:
root@dbstore1002.eqiad.wmnet[(none)]> select TABLE_NAME from information_schema.tables where ENGINE='Aria' and TABLE_SCHEMA='staging'; Empty set (0.00 sec)
root@DBSTORE[staging]> drop table mep_word_persistence; Query OK, 0 rows affected (5.53 sec)
root@DBSTORE[staging]> drop table mep_word_persistence; Query OK, 0 rows affected (5.53 sec)
Thanks @Cmjohnson for replacing disk #6!
17:31 <+icinga-wm> RECOVERY - MegaRAID on db1073 is OK: OK: optimal, 1 logical, 2 physical, WriteBack policy
In T215450#4935181, @Halfak wrote:Yes. Thanks for your patience.
I finished converting mep_word_persistence to InnoDB on dbstore1003:
(1 day 9 hours 52 min 45.49 sec)
Thanks @Halfak, can we drop the table from dbstore1002 then?
Done - thank you
Thanks @elukey - ok to delete the .sql file I created from dbstore1003?
@elukey the .sql file is at: dbstore1003:/srv/tmp.staging/staging.sql
Can you grab it and store it somewhere else, so we have two copies, the HIVE one and this?
@elukey @jcrespo Any objection to put dbstore1002 as IDEMPOTENT?
This host crashes every single day, the data is already drifts a lot from production and this host will be depooled in a matter of weeks, it doesn't make any sense that we keep spending time to fix. Specially on x1.
If we feel x1 is absolutely necessary, we should re-import it again (although I am not 100% sure dbstore1002 wouldn't crash during the re-import)
I am also mysqldumping that table, just in case.
s7 eqiad progress
And back again: RECOVERY - EDAC syslog messages on db1068 is OK: (C)4 ge (W)2 ge 1
As Jaime said: T213664#4924636 this won't be fully gone until this is fully decommissioned.
Great - so the crash happened before. Thanks a lot for helping out here :)
Let's wait for @Halfak to verify he's got everything he needs.
@JAllemandou at what time did you start the job?
From what I can see it crashed at around 18:32, so it crashed before then (if we use your comment at T215450#4932773 as a reference for time).
dbstore1002 crashed, possibly due to T215450: Sqoop staging.mep_word_persistence to HDFS and drop the table from dbstore1002
@JAllemandou dbstore1002 crashed, let's start the same thing but with less jobs I would suggest.
@Cmjohnson you can proceed with the one on slot 6.
The one on slot #1 finished correctly
In T215445#4932828, @Anomie wrote:To get it all in one place, here's that proposal again. It relies on the fact that https://dev.mysql.com/doc/refman/5.5/en/replication-features-differing-tables.html says the slave can have extra columns as long as they come after all the normal columns and have a default value (and that we're not using the "different data types" thing also described at that page.
- Add an extra column to the actor and comment tables on the WMCS copies of the tables (on sanitarium?), something like
ALTER TABLE actor ADD COLUMN wmcs_is_visible TINYINT NOT NULL DEFAULT 0; ALTER TABLE comment ADD COLUMN wmcs_is_visible TINYINT NOT NULL DEFAULT 0;We'll also want to create copies of all the indexes on actor and comment with wmcs_is_visible prefixed.
- Create a bunch of functions and triggers. Note $LOG_TYPE_LIST$ is supposed to be replaced with the list of types from allowed_logtypes in maintain-views.yaml.
1 delimiter // 2 CREATE FUNCTION wmcsCommentShow(id BIGINT) RETURNS INT NOT DETERMINISTIC READS SQL DATA 3 BEGIN 4 RETURN COALESCE( 5 ( SELECT 1 FROM image WHERE img_description_id = id LIMIT 1 ) OR 6 ( SELECT 1 FROM filearchive WHERE fa_deleted_reason_id = id LIMIT 1 ) OR 7 ( SELECT 1 FROM filearchive WHERE fa_description_id = id AND fa_deleted&2 = 0 LIMIT 1 ) OR 8 ( SELECT 1 FROM ipblocks WHERE ipb_reason_id = id and ipb_deleted = 0 LIMIT 1 ) OR 9 ( SELECT 1 FROM oldimage WHERE oi_description_id = id AND oi_deleted&2 = 0 LIMIT 1 ) OR 10 ( SELECT 1 FROM protected_titles WHERE pt_reason_id = id LIMIT 1 ) OR 11 ( SELECT 1 FROM recentchanges WHERE rc_comment_id = id AND rc_deleted&2 = 0 LIMIT 1 ) OR 12 ( SELECT 1 FROM revision JOIN revision_comment_temp ON(revcomment_rev = rev_id) WHERE revcomment_comment_id = id AND rev_deleted&2 = 0 LIMIT 1 ) OR 13 ( SELECT 1 FROM logging WHERE log_comment_id = id AND log_deleted&2 = 0 AND log_type IN ($LOG_TYPE_LIST$) LIMIT 1 ) 14 , 0); 15 END // 16 delimiter ; 17 18 delimiter // 19 CREATE PROCEDURE wmcsCommentOnInsert(id BIGINT, del TINYINT) NOT DETERMINISTIC MODIFIES SQL DATA 20 BEGIN 21 IF NOT del THEN 22 UPDATE comment SET wmcs_is_visible = 1 WHERE comment_id = id; 23 END IF; 24 END // 25 delimiter ; 26 27 delimiter // 28 CREATE PROCEDURE wmcsCommentOnDelete(id BIGINT, del TINYINT) NOT DETERMINISTIC MODIFIES SQL DATA 29 BEGIN 30 IF NOT del AND NOT wmcsCommentShow(id) THEN 31 UPDATE comment SET wmcs_is_visible = 0 WHERE comment_id = id; 32 END IF; 33 END // 34 delimiter ; 35 36 delimiter // 37 CREATE PROCEDURE wmcsCommentOnUpdate(oldId BIGINT, oldDel TINYINT, newId BIGINT, newDel TINYINT) NOT DETERMINISTIC MODIFIES SQL DATA 38 BEGIN 39 IF oldId != newId OR (NOT oldDel) != (NOT newDel) THEN 40 CALL wmcsCommentOnDelete(oldId, oldDel); 41 CALL wmcsCommentOnInsert(newId, newDel); 42 END IF; 43 END // 44 delimiter ; 45 46 CREATE TRIGGER image_wmcsCommentOnInsert AFTER INSERT ON image 47 FOR EACH ROW CALL wmcsCommentOnInsert( NEW.img_description_id, 0 ); 48 CREATE TRIGGER image_wmcsCommentOnUpdate AFTER UPDATE ON image 49 FOR EACH ROW CALL wmcsCommentOnUpdate( OLD.img_description_id, 0, NEW.img_description_id, 0 ); 50 CREATE TRIGGER image_wmcsCommentOnDelete AFTER DELETE ON image 51 FOR EACH ROW CALL wmcsCommentOnDelete( OLD.img_description_id, 0 ); 52 53 CREATE TRIGGER filearchive_wmcsCommentOnInsert_reason AFTER INSERT ON filearchive 54 FOR EACH ROW CALL wmcsCommentOnInsert( NEW.fa_deleted_reason_id, 0 ); 55 CREATE TRIGGER filearchive_wmcsCommentOnUpdate_reason AFTER UPDATE ON filearchive 56 FOR EACH ROW CALL wmcsCommentOnUpdate( OLD.fa_deleted_reason_id, 0, NEW.fa_deleted_reason_id, 0 ); 57 CREATE TRIGGER filearchive_wmcsCommentOnDelete_reason AFTER DELETE ON filearchive 58 FOR EACH ROW CALL wmcsCommentOnDelete( OLD.fa_deleted_reason_id, 0 ); 59 CREATE TRIGGER filearchive_wmcsCommentOnInsert_description AFTER INSERT ON filearchive 60 FOR EACH ROW CALL wmcsCommentOnInsert( NEW.fa_description_id, NEW.fa_deleted&2 ); 61 CREATE TRIGGER filearchive_wmcsCommentOnUpdate_description AFTER UPDATE ON filearchive 62 FOR EACH ROW CALL wmcsCommentOnUpdate( OLD.fa_description_id, OLD.fa_deleted&2, NEW.fa_description_id, NEW.fa_deleted&2 ); 63 CREATE TRIGGER filearchive_wmcsCommentOnDelete_description AFTER DELETE ON filearchive 64 FOR EACH ROW CALL wmcsCommentOnDelete( OLD.fa_description_id, OLD.fa_deleted&2 ); 65 66 CREATE TRIGGER ipblocks_wmcsCommentOnInsert AFTER INSERT ON ipblocks 67 FOR EACH ROW CALL wmcsCommentOnInsert( NEW.ipb_reason_id, NEW.ipb_deleted ); 68 CREATE TRIGGER ipblocks_wmcsCommentOnUpdate AFTER UPDATE ON ipblocks 69 FOR EACH ROW CALL wmcsCommentOnUpdate( OLD.ipb_reason_id, OLD.ipb_deleted, NEW.ipb_reason_id, NEW.ipb_deleted ); 70 CREATE TRIGGER ipblocks_wmcsCommentOnDelete AFTER DELETE ON ipblocks 71 FOR EACH ROW CALL wmcsCommentOnDelete( OLD.ipb_reason_id, OLD.ipb_deleted ); 72 73 CREATE TRIGGER oldimage_wmcsCommentOnInsert AFTER INSERT ON oldimage 74 FOR EACH ROW CALL wmcsCommentOnInsert( NEW.oi_description_id, NEW.oi_deleted&2 ); 75 CREATE TRIGGER oldimage_wmcsCommentOnUpdate AFTER UPDATE ON oldimage 76 FOR EACH ROW CALL wmcsCommentOnUpdate( OLD.oi_description_id, OLD.oi_deleted&2, NEW.oi_description_id, NEW.oi_deleted&2 ); 77 CREATE TRIGGER oldimage_wmcsCommentOnDelete AFTER DELETE ON oldimage 78 FOR EACH ROW CALL wmcsCommentOnDelete( OLD.oi_description_id, OLD.oi_deleted&2 ); 79 80 CREATE TRIGGER protected_titles_wmcsCommentOnInsert AFTER INSERT ON protected_titles 81 FOR EACH ROW CALL wmcsCommentOnInsert( NEW.pt_reason_id, 0 ); 82 CREATE TRIGGER protected_titles_wmcsCommentOnUpdate AFTER UPDATE ON protected_titles 83 FOR EACH ROW CALL wmcsCommentOnUpdate( OLD.pt_reason_id, 0, NEW.pt_reason_id, 0 ); 84 CREATE TRIGGER protected_titles_wmcsCommentOnDelete AFTER DELETE ON protected_titles 85 FOR EACH ROW CALL wmcsCommentOnDelete( OLD.pt_reason_id, 0 ); 86 87 CREATE TRIGGER recentchanges_wmcsCommentOnInsert AFTER INSERT ON recentchanges 88 FOR EACH ROW CALL wmcsCommentOnInsert( NEW.rc_comment_id, NEW.rc_deleted&2 ); 89 CREATE TRIGGER recentchanges_wmcsCommentOnUpdate AFTER UPDATE ON recentchanges 90 FOR EACH ROW CALL wmcsCommentOnUpdate( OLD.rc_comment_id, OLD.rc_deleted&2, NEW.rc_comment_id, NEW.rc_deleted&2 ); 91 CREATE TRIGGER recentchanges_wmcsCommentOnDelete AFTER DELETE ON recentchanges 92 FOR EACH ROW CALL wmcsCommentOnDelete( OLD.rc_comment_id, OLD.rc_deleted&2 ); 93 94 CREATE TRIGGER revision_comment_temp_wmcsCommentOnInsert AFTER INSERT ON revision_comment_temp 95 FOR EACH ROW CALL wmcsCommentOnInsert( NEW.revcomment_comment_id, (SELECT rev_deleted&2 FROM revision WHERE rev_id=NEW.revcomment_rev) ); 96 CREATE TRIGGER revision_comment_temp_wmcsCommentOnUpdate AFTER UPDATE ON revision_comment_temp 97 FOR EACH ROW CALL wmcsCommentOnUpdate( OLD.revcomment_comment_id, (SELECT rev_deleted&2 FROM revision WHERE rev_id=OLD.revcomment_rev), NEW.revcomment_comment_id, (SELECT rev_deleted&2 FROM revision WHERE rev_id=NEW.revcomment_rev) ); 98 CREATE TRIGGER revision_comment_temp_wmcsCommentOnDelete AFTER DELETE ON revision_comment_temp 99 FOR EACH ROW CALL wmcsCommentOnDelete( OLD.revcomment_comment_id, (SELECT rev_deleted&2 FROM revision WHERE rev_id=OLD.revcomment_rev) ); 100 CREATE TRIGGER revision_wmcsCommentOnInsert AFTER INSERT ON revision 101 FOR EACH ROW CALL wmcsCommentOnInsert( (SELECT revcomment_comment_id FROM revision_comment_temp WHERE revcomment_rev=NEW.rev_id), NEW.rev_deleted&2 ); 102 CREATE TRIGGER revision_wmcsCommentOnUpdate AFTER UPDATE ON revision 103 FOR EACH ROW CALL wmcsCommentOnUpdate( (SELECT revcomment_comment_id FROM revision_comment_temp WHERE revcomment_rev=OLD.rev_id), OLD.rev_deleted&2, (SELECT revcomment_comment_id FROM revision_comment_temp WHERE revcomment_rev=NEW.rev_id), NEW.rev_deleted&2 ); 104 CREATE TRIGGER revision_wmcsCommentOnDelete AFTER DELETE ON revision 105 FOR EACH ROW CALL wmcsCommentOnDelete( (SELECT revcomment_comment_id FROM revision_comment_temp WHERE revcomment_rev=OLD.rev_id), OLD.rev_deleted&2 ); 106 107 CREATE TRIGGER logging_wmcsCommentOnInsert AFTER INSERT ON logging 108 FOR EACH ROW CALL wmcsCommentOnInsert( NEW.log_comment_id, NOT ( NEW.log_deleted&2 = 0 AND NEW.log_type IN ($LOG_TYPE_LIST$) ) ); 109 CREATE TRIGGER logging_wmcsCommentOnUpdate AFTER UPDATE ON logging 110 FOR EACH ROW CALL wmcsCommentOnUpdate( OLD.log_comment_id, NOT ( OLD.log_deleted&2 = 0 AND OLD.log_type IN ($LOG_TYPE_LIST$) ), NEW.log_comment_id, NOT ( NEW.log_deleted&2 = 0 AND NEW.log_type IN ($LOG_TYPE_LIST$) ) ); 111 CREATE TRIGGER logging_wmcsCommentOnDelete AFTER DELETE ON logging 112 FOR EACH ROW CALL wmcsCommentOnDelete( OLD.log_comment_id, NOT ( OLD.log_deleted&2 = 0 AND OLD.log_type IN ($LOG_TYPE_LIST$) ) );
1 delimiter // 2 CREATE FUNCTION wmcsActorShow(id BIGINT) RETURNS INT NOT DETERMINISTIC READS SQL DATA 3 BEGIN 4 RETURN COALESCE( 5 ( SELECT 1 FROM actor JOIN user ON(actor_user=user_id) WHERE actor_id = id LIMIT 1 ) OR 6 ( SELECT 1 FROM archive WHERE ar_actor = id AND ar_deleted&4 = 0 LIMIT 1 ) OR 7 ( SELECT 1 FROM ipblocks WHERE ipb_by_actor = id and ipb_deleted = 0 LIMIT 1 ) OR 8 ( SELECT 1 FROM image WHERE img_actor = id LIMIT 1 ) OR 9 ( SELECT 1 FROM oldimage WHERE oi_actor = id AND oi_deleted&4 = 0 LIMIT 1 ) OR 10 ( SELECT 1 FROM filearchive WHERE fa_actor = id AND fa_deleted&4 = 0 LIMIT 1 ) OR 11 ( SELECT 1 FROM recentchanges WHERE rc_actor = id AND rc_deleted&4 = 0 LIMIT 1 ) OR 12 ( SELECT 1 FROM logging WHERE log_actor = id AND log_deleted&4 = 0 AND log_type IN ($LOG_TYPE_LIST$) LIMIT 1 ) OR 13 ( SELECT 1 FROM revision JOIN revision_actor_temp ON(revactor_rev = rev_id) WHERE revactor_actor = id AND rev_deleted&4 = 0 LIMIT 1 ) 14 , 0); 15 END // 16 delimiter ; 17 18 delimiter // 19 CREATE PROCEDURE wmcsActorOnInsert(id BIGINT, del TINYINT) NOT DETERMINISTIC MODIFIES SQL DATA 20 BEGIN 21 IF NOT del THEN 22 UPDATE actor SET wmcs_is_visible = 1 WHERE actor_id = id; 23 END IF; 24 END // 25 delimiter ; 26 27 delimiter // 28 CREATE PROCEDURE wmcsActorOnDelete(id BIGINT, del TINYINT) NOT DETERMINISTIC MODIFIES SQL DATA 29 BEGIN 30 IF NOT del AND NOT wmcsActorShow(id) THEN 31 UPDATE actor SET wmcs_is_visible = 0 WHERE actor_id = id; 32 END IF; 33 END // 34 delimiter ; 35 36 delimiter // 37 CREATE PROCEDURE wmcsActorOnUpdate(oldId BIGINT, oldDel TINYINT, newId BIGINT, newDel TINYINT) NOT DETERMINISTIC MODIFIES SQL DATA 38 BEGIN 39 IF oldId != newId OR (NOT oldDel) != (NOT newDel) THEN 40 CALL wmcsActorOnDelete(oldId, oldDel); 41 CALL wmcsActorOnInsert(newId, newDel); 42 END IF; 43 END // 44 delimiter ; 45 46 CREATE TRIGGER archive_wmcsActorOnInsert AFTER INSERT ON archive 47 FOR EACH ROW CALL wmcsActorOnInsert( NEW.ar_actor, NEW.ar_deleted&4 ); 48 CREATE TRIGGER archive_wmcsActorOnUpdate AFTER UPDATE ON archive 49 FOR EACH ROW CALL wmcsActorOnUpdate( OLD.ar_actor, OLD.ar_deleted&4, NEW.ar_actor, NEW.ar_deleted&4 ); 50 CREATE TRIGGER archive_wmcsActorOnDelete AFTER DELETE ON archive 51 FOR EACH ROW CALL wmcsActorOnDelete( OLD.ar_actor, OLD.ar_deleted&4 ); 52 53 CREATE TRIGGER ipblocks_wmcsActorOnInsert AFTER INSERT ON ipblocks 54 FOR EACH ROW CALL wmcsActorOnInsert( NEW.ipb_by_actor, NEW.ipb_deleted ); 55 CREATE TRIGGER ipblocks_wmcsActorOnUpdate AFTER UPDATE ON ipblocks 56 FOR EACH ROW CALL wmcsActorOnUpdate( OLD.ipb_by_actor, OLD.ipb_deleted, NEW.ipb_by_actor, NEW.ipb_deleted ); 57 CREATE TRIGGER ipblocks_wmcsActorOnDelete AFTER DELETE ON ipblocks 58 FOR EACH ROW CALL wmcsActorOnDelete( OLD.ipb_by_actor, OLD.ipb_deleted ); 59 60 CREATE TRIGGER image_wmcsActorOnInsert AFTER INSERT ON image 61 FOR EACH ROW CALL wmcsActorOnInsert( NEW.img_actor, 0 ); 62 CREATE TRIGGER image_wmcsActorOnUpdate AFTER UPDATE ON image 63 FOR EACH ROW CALL wmcsActorOnUpdate( OLD.img_actor, 0, NEW.img_actor, 0 ); 64 CREATE TRIGGER image_wmcsActorOnDelete AFTER DELETE ON image 65 FOR EACH ROW CALL wmcsActorOnDelete( OLD.img_actor, 0 ); 66 67 CREATE TRIGGER oldimage_wmcsActorOnInsert AFTER INSERT ON oldimage 68 FOR EACH ROW CALL wmcsActorOnInsert( NEW.oi_actor, NEW.oi_deleted&4 ); 69 CREATE TRIGGER oldimage_wmcsActorOnUpdate AFTER UPDATE ON oldimage 70 FOR EACH ROW CALL wmcsActorOnUpdate( OLD.oi_actor, OLD.oi_deleted&4, NEW.oi_actor, NEW.oi_deleted&4 ); 71 CREATE TRIGGER oldimage_wmcsActorOnDelete AFTER DELETE ON oldimage 72 FOR EACH ROW CALL wmcsActorOnDelete( OLD.oi_actor, OLD.oi_deleted&4 ); 73 74 CREATE TRIGGER filearchive_wmcsActorOnInsert AFTER INSERT ON filearchive 75 FOR EACH ROW CALL wmcsActorOnInsert( NEW.fa_actor, NEW.fa_deleted&4 ); 76 CREATE TRIGGER filearchive_wmcsActorOnUpdate AFTER UPDATE ON filearchive 77 FOR EACH ROW CALL wmcsActorOnUpdate( OLD.fa_actor, OLD.fa_deleted&4, NEW.fa_actor, NEW.fa_deleted&4 ); 78 CREATE TRIGGER filearchive_wmcsActorOnDelete AFTER DELETE ON filearchive 79 FOR EACH ROW CALL wmcsActorOnDelete( OLD.fa_actor, OLD.fa_deleted&4 ); 80 81 CREATE TRIGGER recentchanges_wmcsActorOnInsert AFTER INSERT ON recentchanges 82 FOR EACH ROW CALL wmcsActorOnInsert( NEW.rc_actor, NEW.rc_deleted&4 ); 83 CREATE TRIGGER recentchanges_wmcsActorOnUpdate AFTER UPDATE ON recentchanges 84 FOR EACH ROW CALL wmcsActorOnUpdate( OLD.rc_actor, OLD.rc_deleted&4, NEW.rc_actor, NEW.rc_deleted&4 ); 85 CREATE TRIGGER recentchanges_wmcsActorOnDelete AFTER DELETE ON recentchanges 86 FOR EACH ROW CALL wmcsActorOnDelete( OLD.rc_actor, OLD.rc_deleted&4 ); 87 88 CREATE TRIGGER logging_wmcsActorOnInsert AFTER INSERT ON logging 89 FOR EACH ROW CALL wmcsActorOnInsert( NEW.log_actor, NOT ( NEW.log_deleted&4 = 0 AND NEW.log_type IN ($LOG_TYPE_LIST$) ) ); 90 CREATE TRIGGER logging_wmcsActorOnUpdate AFTER UPDATE ON logging 91 FOR EACH ROW CALL wmcsActorOnUpdate( OLD.log_actor, NOT ( OLD.log_deleted&4 = 0 AND OLD.log_type IN ($LOG_TYPE_LIST$) ), NEW.log_actor, NOT ( NEW.log_deleted&4 = 0 AND NEW.log_type IN ($LOG_TYPE_LIST$) ) ); 92 CREATE TRIGGER logging_wmcsActorOnDelete AFTER DELETE ON logging 93 FOR EACH ROW CALL wmcsActorOnDelete( OLD.log_actor, NOT ( OLD.log_deleted&4 = 0 AND OLD.log_type IN ($LOG_TYPE_LIST$) ) ); 94 95 CREATE TRIGGER revision_actor_temp_wmcsActorOnInsert AFTER INSERT ON revision_actor_temp 96 FOR EACH ROW CALL wmcsActorOnInsert( NEW.revactor_actor, (SELECT rev_deleted&4 FROM revision WHERE rev_id=NEW.revactor_rev) ); 97 CREATE TRIGGER revision_actor_temp_wmcsActorOnUpdate AFTER UPDATE ON revision_actor_temp 98 FOR EACH ROW CALL wmcsActorOnUpdate( OLD.revactor_actor, (SELECT rev_deleted&4 FROM revision WHERE rev_id=OLD.revactor_rev), NEW.revactor_actor, (SELECT rev_deleted&4 FROM revision WHERE rev_id=NEW.revactor_rev) ); 99 CREATE TRIGGER revision_actor_temp_wmcsActorOnDelete AFTER DELETE ON revision_actor_temp 100 FOR EACH ROW CALL wmcsActorOnDelete( OLD.revactor_actor, (SELECT rev_deleted&4 FROM revision WHERE rev_id=OLD.revactor_rev) ); 101 CREATE TRIGGER revision_wmcsActorOnInsert AFTER INSERT ON revision 102 FOR EACH ROW CALL wmcsActorOnInsert( (SELECT revactor_actor FROM revision_actor_temp WHERE revactor_rev=NEW.rev_id), NEW.rev_deleted&4 ); 103 CREATE TRIGGER revision_wmcsActorOnUpdate AFTER UPDATE ON revision 104 FOR EACH ROW CALL wmcsActorOnUpdate( (SELECT revactor_actor FROM revision_actor_temp WHERE revactor_rev=OLD.rev_id), OLD.rev_deleted&4, (SELECT revactor_actor FROM revision_actor_temp WHERE revactor_rev=NEW.rev_id), NEW.rev_deleted&4 ); 105 CREATE TRIGGER revision_wmcsActorOnDelete AFTER DELETE ON revision 106 FOR EACH ROW CALL wmcsActorOnDelete( (SELECT revactor_actor FROM revision_actor_temp WHERE revactor_rev=OLD.rev_id), OLD.rev_deleted&4 );
- Populate wmcs_is_visible for existing rows. That might be as simple as UPDATE comment SET wmcs_is_visible = wmcsCommentShow( comment_id ) and the same for actor, or we could unroll it with a bunch of queries like UPDATE comment JOIN recentchanges ON(rc_comment_id = commend_id) SET wmcs_is_visible = 1 WHERE wmcs_is_visible = 0 AND rc_deleted&2 = 0, and/or we could run it through in batches.
- Update the views, instead of all the subqueries they can be just WHERE wmcs_is_visible.
Open questions:
- Does the replication actually work like the docs say it does?
In T210478#4931616, @elukey wrote:analytics-in4 diff:
+ term mysql-dbstore { + from { + destination-address { + /* dbstore1003 */ + 10.64.0.137/32; + /* dbstore1004 */ + 10.64.16.26/32; + /* dbstore1005 */ + 10.64.32.30/32; + } + protocol tcp; + destination-port [ 3311 3315 3317 3312 3313 3314 3316 3318 3320 3340 ]; + } + then accept; + }
Then also make sure to whitelist dbstore1003:3340 as that is where the staging database will leave.
Grants revoked:
root@db1073.eqiad.wmnet[(none)]> show grants for 'ssastry'@'10.64.16.151'; ERROR 1141 (42000): There is no such grant defined for user 'ssastry' on host '10.64.16.151' root@db1073.eqiad.wmnet[(none)]> show grants for 'testreduce'@'10.64.16.151'; ERROR 1141 (42000): There is no such grant defined for user 'testreduce' on host '10.64.16.151'
@elukey after merging: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/487000/ I have done the following:
s4 eqiad progress
@elukey from what I can see, the research user is used to access wikis, not only for staging. So I guess we need to create it on all the instances and not only where staging lives.
Example: T200801#4570548
In T215231#4926632, @jcrespo wrote:current labsdb hosts
Those will be on multi-instance soon(TM).
In T215231#4926627, @jcrespo wrote:I don't think we should setup new hosts using multi-source.
An attempt to run mydumper for T210478 on dbstore1002 made it crash.
I have suggested to use labsdb1012 as a hostname, as this host has the same hardware as the other labsdb1009-1011 and will be setup the same way of those hosts.
When do you want to do this?