Page MenuHomePhabricator
Feed Advanced Search

Yesterday

Marostegui committed rOSTDb98d81453bc4: tendril-host-add.sh: Add comment about MySQL 5.7+ (authored by Marostegui).
tendril-host-add.sh: Add comment about MySQL 5.7+
Fri, Jan 17, 7:32 PM
Marostegui closed T243081: Make Percona 8 report the basics on tendril, a subtask of T193224: Evaluate and decide the future of relational datastore at WMF after the upgrade of MariaDB 10.1 is finished, as Resolved.
Fri, Jan 17, 3:50 PM · MediaWiki-General, Operations, DBA
Marostegui closed T243081: Make Percona 8 report the basics on tendril as Resolved.
Fri, Jan 17, 3:50 PM · Operations, DBA
Marostegui created T243081: Make Percona 8 report the basics on tendril.
Fri, Jan 17, 3:50 PM · Operations, DBA
Marostegui created T243073: Add primary key to pagetriage_page_tags.
Fri, Jan 17, 2:08 PM · Growth Design, MediaWiki-General, Schema-change
Marostegui claimed T243052: Productionize es1020-es1025, es2020-es2025.
Fri, Jan 17, 1:48 PM · DBA
Marostegui moved T243055: Publish SQL dumps of CodeReview tables from Triage to Backlog on the DBA board.
Fri, Jan 17, 1:05 PM · DBA, Dumps-Generation, MediaWiki-extensions-CodeReview
Marostegui updated subscribers of T243055: Publish SQL dumps of CodeReview tables.

From what is worth, one of those tables had a change yesterday:

root@db1075:/srv/sqldata/mediawikiwiki# ls -lh code_*.ibd
-rw-rw---- 1 mysql mysql 128K Nov 10  2017 code_authors.ibd
-rw-rw---- 1 mysql mysql 704K Nov 18  2015 code_bugs.ibd
-rw-rw---- 1 mysql mysql  24M Aug 14  2018 code_comment.ibd
-rw-rw---- 1 mysql mysql 200M Nov 18  2015 code_paths.ibd
-rw-rw---- 1 mysql mysql  40M Aug  1  2017 code_prop_changes.ibd
-rw-rw---- 1 mysql mysql 5.0M Nov 18  2015 code_relations.ibd
-rw-rw---- 1 mysql mysql 112K Jan 16 05:02 code_repo.ibd
-rw-rw---- 1 mysql mysql 2.1G Nov 18  2015 code_rev.ibd
-rw-rw---- 1 mysql mysql 368K Aug  1  2017 code_signoffs.ibd
-rw-rw---- 1 mysql mysql 272K Nov 18  2015 code_tags.ibd
Fri, Jan 17, 10:29 AM · DBA, Dumps-Generation, MediaWiki-extensions-CodeReview
Marostegui added a comment to T241359: (Needed by 31st January) eqiad: rack/setup/install es102[0-5].eqiad.wmnet.

When adding the MAC addresses to the DHCP file, make sure to add the following line:

option pxelinux.pathprefix "http://apt.wikimedia.org/tftpboot/stretch-installer-bootif/";
Fri, Jan 17, 8:39 AM · DBA, Operations, ops-eqiad
Marostegui updated the task description for T239791: DB: perform rolling restart of mariadb deamons to pick up CA changes.
Fri, Jan 17, 7:52 AM · DBA, User-jbond, Puppet, Operations
Marostegui updated the task description for T243052: Productionize es1020-es1025, es2020-es2025.
Fri, Jan 17, 6:52 AM · DBA
Marostegui updated the task description for T232446: Compress new Wikibase tables.
Fri, Jan 17, 6:31 AM · DBA
Marostegui moved T243052: Productionize es1020-es1025, es2020-es2025 from Triage to In progress on the DBA board.
Fri, Jan 17, 6:23 AM · DBA
Marostegui updated the task description for T186188: Failover DB masters in row D.
Fri, Jan 17, 6:16 AM · DBA
Marostegui added a parent task for T241336: (Needed By 31st January) codfw: rack/setup/install es202[0-5].codfw.wmnet: T243052: Productionize es1020-es1025, es2020-es2025.
Fri, Jan 17, 6:15 AM · Operations, DBA, ops-codfw
Marostegui added a parent task for T241359: (Needed by 31st January) eqiad: rack/setup/install es102[0-5].eqiad.wmnet: T243052: Productionize es1020-es1025, es2020-es2025.
Fri, Jan 17, 6:15 AM · DBA, Operations, ops-eqiad
Marostegui added subtasks for T243052: Productionize es1020-es1025, es2020-es2025: T241359: (Needed by 31st January) eqiad: rack/setup/install es102[0-5].eqiad.wmnet, T241336: (Needed By 31st January) codfw: rack/setup/install es202[0-5].codfw.wmnet.
Fri, Jan 17, 6:15 AM · DBA
Marostegui added a parent task for T243052: Productionize es1020-es1025, es2020-es2025: T226704: Setup es4 and es5 replica sets for new read-write external store service.
Fri, Jan 17, 6:14 AM · DBA
Marostegui added a subtask for T226704: Setup es4 and es5 replica sets for new read-write external store service: T243052: Productionize es1020-es1025, es2020-es2025.
Fri, Jan 17, 6:13 AM · Epic, DBA
Marostegui triaged T243052: Productionize es1020-es1025, es2020-es2025 as Medium priority.
Fri, Jan 17, 6:13 AM · DBA
Marostegui created T243052: Productionize es1020-es1025, es2020-es2025.
Fri, Jan 17, 6:13 AM · DBA
Marostegui closed T241336: (Needed By 31st January) codfw: rack/setup/install es202[0-5].codfw.wmnet as Resolved.
Fri, Jan 17, 6:10 AM · Operations, DBA, ops-codfw
Marostegui added a comment to T241336: (Needed By 31st January) codfw: rack/setup/install es202[0-5].codfw.wmnet.

Thank you Papaul!

Fri, Jan 17, 6:08 AM · Operations, DBA, ops-codfw

Thu, Jan 16

Marostegui updated the task description for T239791: DB: perform rolling restart of mariadb deamons to pick up CA changes.
Thu, Jan 16, 1:40 PM · DBA, User-jbond, Puppet, Operations
Marostegui added a comment to T241359: (Needed by 31st January) eqiad: rack/setup/install es102[0-5].eqiad.wmnet.

You guys think this will be ready by 31st Jan?
Thanks.

Thu, Jan 16, 1:29 PM · DBA, Operations, ops-eqiad
Marostegui added a comment to T241336: (Needed By 31st January) codfw: rack/setup/install es202[0-5].codfw.wmnet.

@Papaul you can proceed with the installation of es2021, es2022, es2023, es2024 and es2025
es2020 is installed already

Thu, Jan 16, 12:39 PM · Operations, DBA, ops-codfw
Marostegui created P10173 (An Untitled Masterwork).
Thu, Jan 16, 12:27 PM
Marostegui updated the task description for T239791: DB: perform rolling restart of mariadb deamons to pick up CA changes.
Thu, Jan 16, 7:25 AM · DBA, User-jbond, Puppet, Operations
Marostegui created P10167 (An Untitled Masterwork).
Thu, Jan 16, 6:59 AM

Wed, Jan 15

Marostegui added a comment to T227717: Drop DB tables for now-deleted zerowiki from production.

That's way easier then. We should still get a full logical dump before attempting to do so.

Wed, Jan 15, 6:40 PM · Release-Engineering-Team-TODO, DBA, Product-Infrastructure-Team-Backlog
Marostegui added a comment to T242481: d-i fails to install on servers with BRCM 2P 1G BT + 2P 10G SFP NDC.

I like option 3 indeed.
There is one more option which is a bit more painful, which is to live hack pxelinux and set the MAC address of the host we are going to install and do it manually. We have 6 hosts in eqiad and 12 in codfw.

Wed, Jan 15, 12:01 PM · Patch-For-Review, Operations, DBA, ops-codfw
Marostegui added a comment to P10149 (An Untitled Masterwork).
root@db1107:~# mysql -e "show master status\G"
*************************** 1. row ***************************
            File: db1107-bin.000003
        Position: 110399237
    Binlog_Do_DB:
Binlog_Ignore_DB:
Wed, Jan 15, 9:58 AM
Marostegui added a comment to P10149 (An Untitled Masterwork).
root@db1107:~# mysql -e "show master status\G"
*************************** 1. row ***************************
            File: db1107-bin.000003
        Position: 343
    Binlog_Do_DB:
Binlog_Ignore_DB:
Wed, Jan 15, 9:54 AM
Marostegui closed T242749: Drop `echo_notification_user_hash_timestamp` index where it exists as Resolved.

All done!

Wed, Jan 15, 9:39 AM · Schema-change, DBA
Marostegui updated the task description for T242749: Drop `echo_notification_user_hash_timestamp` index where it exists.
Wed, Jan 15, 9:39 AM · Schema-change, DBA
Marostegui added a comment to T242749: Drop `echo_notification_user_hash_timestamp` index where it exists.

x1 eqiad progress

Wed, Jan 15, 9:28 AM · Schema-change, DBA
Marostegui updated the task description for T242749: Drop `echo_notification_user_hash_timestamp` index where it exists.
Wed, Jan 15, 9:25 AM · Schema-change, DBA
Marostegui claimed T242749: Drop `echo_notification_user_hash_timestamp` index where it exists.
Wed, Jan 15, 9:09 AM · Schema-change, DBA
Marostegui updated the task description for T239791: DB: perform rolling restart of mariadb deamons to pick up CA changes.
Wed, Jan 15, 6:28 AM · DBA, User-jbond, Puppet, Operations
Marostegui updated the task description for T239791: DB: perform rolling restart of mariadb deamons to pick up CA changes.
Wed, Jan 15, 6:19 AM · DBA, User-jbond, Puppet, Operations
Marostegui moved T242749: Drop `echo_notification_user_hash_timestamp` index where it exists from Triage to Next on the DBA board.
Wed, Jan 15, 6:16 AM · Schema-change, DBA
Marostegui updated the task description for T241336: (Needed By 31st January) codfw: rack/setup/install es202[0-5].codfw.wmnet.
Wed, Jan 15, 6:14 AM · Operations, DBA, ops-codfw
Marostegui added a comment to T242817: m5 ran out of connections after openstack upgrade to "Pike".

In the course of this, @JHedden restarted several services, which reduced current connection usage to sane levels, and I set the max_connections on the m5 master to 600 to give more breathing room for troubleshooting (note to @Marostegui and @jcrespo that I did that and don't intend to keep it that way).
It took around 10 hours for this to fill connections previously, so people who are not in convenient timezones should be able to get some sleep until we can get this resolved.

Wed, Jan 15, 6:09 AM · Cloud-VPS, cloud-services-team (Kanban)
Marostegui created P10149 (An Untitled Masterwork).
Wed, Jan 15, 6:06 AM
Marostegui updated the task description for T239453: Remove partitions from revision table.
Wed, Jan 15, 5:58 AM · DBA

Tue, Jan 14

Marostegui updated the task description for T241336: (Needed By 31st January) codfw: rack/setup/install es202[0-5].codfw.wmnet.
Tue, Jan 14, 4:54 PM · Operations, DBA, ops-codfw
Marostegui added a comment to T242481: d-i fails to install on servers with BRCM 2P 1G BT + 2P 10G SFP NDC.

pxelinux has a generic option to pass the MAC in place when receiving the boot image as BOOTIF: https://wiki.syslinux.org/wiki/index.php?title=SYSLINUX#SYSAPPEND_bitmask
Let's try a reimage os es2020 with

ipappend 2

added to /srv/tftpboot/stretch-installer/pxelinux.cfg/ttyS1-115200 (between the lines starting with "kernel" and "append initrd" to confirm.

Tue, Jan 14, 4:34 PM · Patch-For-Review, Operations, DBA, ops-codfw
Marostegui added a comment to T242481: d-i fails to install on servers with BRCM 2P 1G BT + 2P 10G SFP NDC.

Forgot to thank @MoritzMuehlenhoff for all the help and time with the troubleshooting

Tue, Jan 14, 3:17 PM · Patch-For-Review, Operations, DBA, ops-codfw
Marostegui added a comment to T242481: d-i fails to install on servers with BRCM 2P 1G BT + 2P 10G SFP NDC.

So adding the 01- did the trick and es2020 is installing:

append initrd=debian-installer/amd64/initrd.gz vga=normal auto-install/enable=true preseed/url=http://apt.wikimedia.org/autoinstall/preseed.cfg DEBCONF_DEBUG=5 netcfg/choose_interface=auto netcfg/get_hostname=unassigned netcfg/get_domain=unassigned netcfg/dhcp_timeout=60 BOOTIF=01-b0-26-28-f5-28-02 --- console=ttyS1,115200n8
Tue, Jan 14, 3:17 PM · Patch-For-Review, Operations, DBA, ops-codfw
Marostegui closed T241506: Degraded RAID on db1100 as Resolved.

All good - thank you!

Number of Virtual Disks: 1
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 3.635 TB
Sector Size         : 512
Is VD emulated      : Yes
Mirror Data         : 3.635 TB
State               : Optimal
Strip Size          : 256 KB
Number Of Drives    : 10
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
Default Power Savings Policy: Controller Defined
Current Power Savings Policy: None
Can spin up in 1 minute: No
LD has drives that support T10 power conditions: No
LD's IO profile supports MAX power savings with cached writes: No
Bad Blocks Exist: No
Is VD Cached: No
Number of Spans: 1
Span: 0 - Number of PDs: 10
Tue, Jan 14, 3:10 PM · DBA, ops-eqiad, Operations
Marostegui created P10145 (An Untitled Masterwork).
Tue, Jan 14, 3:03 PM
Marostegui added a comment to T242481: d-i fails to install on servers with BRCM 2P 1G BT + 2P 10G SFP NDC.

So, I digged into this a little: Interface auto setup not working if one the devices has no link seems to be a known bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=700292
If we don't have a clean way to disable the 10G card until it's actually linked/used, we can try passing the MAC address of the 1G interface via BOOTIF=aa-bb-cc-dd-ee-ff in the pxelinux.cfg (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=615600)

Tue, Jan 14, 2:42 PM · Patch-For-Review, Operations, DBA, ops-codfw
Marostegui added a comment to T241506: Degraded RAID on db1100.

Thanks - it is now rebuilding. I will close the task once it is finished

PD: 0 Information
Enclosure Device ID: 32
Slot Number: 0
Drive's position: DiskGroup: 0, Span: 0, Arm: 0
Enclosure position: 1
Device Id: 0
WWN: 55cd2e4150a93376
Sequence Number: 11
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA
Tue, Jan 14, 2:22 PM · DBA, ops-eqiad, Operations
Marostegui updated the task description for T239791: DB: perform rolling restart of mariadb deamons to pick up CA changes.
Tue, Jan 14, 2:00 PM · DBA, User-jbond, Puppet, Operations
Marostegui added a comment to T242481: d-i fails to install on servers with BRCM 2P 1G BT + 2P 10G SFP NDC.

@Papaul can you double check (maybe even with the vendor) if there is a way to disable the 10G port for now?

Tue, Jan 14, 1:41 PM · Patch-For-Review, Operations, DBA, ops-codfw
Marostegui added a comment to T242437: Unable to view some articles due to fatal LogicException: Cannot roll back missing named ref "" (from ReferenceStack.php).

Not sure which one you fixed, but it definitely made the one I reported stop :-)
https://logstash.wikimedia.org/goto/f89dbf583c75b464f941a73e51f35fc5

Tue, Jan 14, 10:44 AM · MW-1.35-notes (1.35.0-wmf.15; 2020-01-14), Wikimedia-Incident, Book-Referencing, TCB-Team, WMDE-QWERTY-Team, WMDE-QWERTY-Sprint-2020-01-08, Wikimedia-production-error, Cite
Marostegui triaged T241359: (Needed by 31st January) eqiad: rack/setup/install es102[0-5].eqiad.wmnet as Medium priority.
Tue, Jan 14, 6:51 AM · DBA, Operations, ops-eqiad
Marostegui triaged T242481: d-i fails to install on servers with BRCM 2P 1G BT + 2P 10G SFP NDC as High priority.
Tue, Jan 14, 6:51 AM · Patch-For-Review, Operations, DBA, ops-codfw
Marostegui moved T242702: Test MariaDB 10.4 in production from Triage to Next on the DBA board.
Tue, Jan 14, 6:49 AM · DBA
Marostegui created T242702: Test MariaDB 10.4 in production.
Tue, Jan 14, 6:49 AM · DBA
Marostegui closed T242688: Extend flow_ext_ref.ref_src_wiki as Resolved.

All done!

Tue, Jan 14, 6:27 AM · Blocked-on-schema-change, DBA
Marostegui updated the task description for T242688: Extend flow_ext_ref.ref_src_wiki.
Tue, Jan 14, 6:26 AM · Blocked-on-schema-change, DBA
Marostegui updated the task description for T242688: Extend flow_ext_ref.ref_src_wiki.
Tue, Jan 14, 6:23 AM · Blocked-on-schema-change, DBA
Marostegui updated the task description for T242688: Extend flow_ext_ref.ref_src_wiki.
Tue, Jan 14, 6:22 AM · Blocked-on-schema-change, DBA
Marostegui added a comment to T242688: Extend flow_ext_ref.ref_src_wiki.
officewiki
*************************** 1. row ***************************
       Table: flow_ext_ref
Create Table: CREATE TABLE `flow_ext_ref` (
  `ref_src_object_id` binary(11) NOT NULL,
  `ref_src_object_type` varbinary(32) NOT NULL,
  `ref_src_workflow_id` binary(11) NOT NULL,
  `ref_src_namespace` int(11) NOT NULL,
  `ref_src_title` varbinary(255) NOT NULL,
  `ref_target` blob NOT NULL,
  `ref_type` varbinary(16) NOT NULL,
  `ref_src_wiki` varbinary(64) NOT NULL,
  `ref_id` binary(11) NOT NULL,
  PRIMARY KEY (`ref_id`),
  KEY `flow_ext_ref_idx_v2` (`ref_src_wiki`,`ref_src_namespace`,`ref_src_title`,`ref_type`,`ref_target`(255),`ref_src_object_type`,`ref_src_object_id`),
  KEY `flow_ext_ref_revision_v2` (`ref_src_wiki`,`ref_src_namespace`,`ref_src_title`,`ref_src_object_type`,`ref_src_object_id`,`ref_type`,`ref_target`(255))
) ENGINE=InnoDB DEFAULT CHARSET=binary
techconductwiki
*************************** 1. row ***************************
       Table: flow_ext_ref
Create Table: CREATE TABLE `flow_ext_ref` (
  `ref_id` binary(11) NOT NULL,
  `ref_src_wiki` varbinary(64) NOT NULL,
  `ref_src_object_id` binary(11) NOT NULL,
  `ref_src_object_type` varbinary(32) NOT NULL,
  `ref_src_workflow_id` binary(11) NOT NULL,
  `ref_src_namespace` int(11) NOT NULL,
  `ref_src_title` varbinary(255) NOT NULL,
  `ref_target` blob NOT NULL,
  `ref_type` varbinary(16) NOT NULL,
  PRIMARY KEY (`ref_id`),
  KEY `flow_ext_ref_idx_v3` (`ref_src_wiki`,`ref_src_namespace`,`ref_src_title`,`ref_type`,`ref_target`(255),`ref_src_object_type`,`ref_src_object_id`),
  KEY `flow_ext_ref_revision_v2` (`ref_src_wiki`,`ref_src_namespace`,`ref_src_title`,`ref_src_object_type`,`ref_src_object_id`,`ref_type`,`ref_target`(255))
) ENGINE=InnoDB DEFAULT CHARSET=binary
Tue, Jan 14, 6:22 AM · Blocked-on-schema-change, DBA
Marostegui added a comment to T242437: Unable to view some articles due to fatal LogicException: Cannot roll back missing named ref "" (from ReferenceStack.php).

This is causing some more exceptions starting the 13th at 20:00 UTC
https://logstash.wikimedia.org/goto/f89dbf583c75b464f941a73e51f35fc5

Tue, Jan 14, 6:18 AM · MW-1.35-notes (1.35.0-wmf.15; 2020-01-14), Wikimedia-Incident, Book-Referencing, TCB-Team, WMDE-QWERTY-Team, WMDE-QWERTY-Sprint-2020-01-08, Wikimedia-production-error, Cite
Marostegui moved T242688: Extend flow_ext_ref.ref_src_wiki from Backlog to In progress on the Blocked-on-schema-change board.
Tue, Jan 14, 6:13 AM · Blocked-on-schema-change, DBA
Marostegui claimed T242688: Extend flow_ext_ref.ref_src_wiki.
Tue, Jan 14, 6:13 AM · Blocked-on-schema-change, DBA
Marostegui added a comment to T143763: Remove unused bundling DB fields.

Thanks @Reedy
That index is existing on all the wikis as that index wasn't included on the original files of this task and hence it wasn't dropped.
https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Echo/+/306846/10/db_patches/patch-drop-notification_bundle_base.sql
https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Echo/+/306849/9/db_patches/patch-drop-notification_bundle_display_hash.sql

Tue, Jan 14, 6:13 AM · MW-1.33-notes (1.33.0-wmf.22; 2019-03-19), Blocked-on-schema-change, Schema-change, MW-1.28-release (WMF-deploy-2016-08-30_(1.28.0-wmf.17)), Patch-For-Review, Notifications
Marostegui updated the task description for T239453: Remove partitions from revision table.
Tue, Jan 14, 5:58 AM · DBA

Mon, Jan 13

Marostegui added a comment to T241506: Degraded RAID on db1100.

@Jclark-ctr feel free to replace the disk once you get to the DC, disk #0 is the one.

Mon, Jan 13, 5:41 PM · DBA, ops-eqiad, Operations
Marostegui added a comment to T242481: d-i fails to install on servers with BRCM 2P 1G BT + 2P 10G SFP NDC.

For what is worth, exactly the same behaviour is happening on es2020.
And what was done at T242481#5797643 for es2024 also works and the 1G NIC is able to ping its gateway.

Mon, Jan 13, 5:34 PM · Patch-For-Review, Operations, DBA, ops-codfw
Marostegui added a comment to T242481: d-i fails to install on servers with BRCM 2P 1G BT + 2P 10G SFP NDC.

Thanks @MoritzMuehlenhoff for your tests.

Mon, Jan 13, 5:01 PM · Patch-For-Review, Operations, DBA, ops-codfw
Marostegui added a comment to T242481: d-i fails to install on servers with BRCM 2P 1G BT + 2P 10G SFP NDC.

Some more notes.
The only way to ping the gateway seems to be using the 1G ports, and making sure the 10G aren't there.
The 1G ifaces are controlled by tg3 module, so I have removed both, 1G and 10G drivers and then only loaded 1G one:

~ # modprobe -r bnxt_en
~ # modprobe -r tg3
~ # modprobe tg3
~ # ip addr add 10.192.0.158/22 dev eno3
~ # ip link set eno3 up
~ # ping 10.192.0.1
PING 10.192.0.1 (10.192.0.1): 56 data bytes
64 bytes from 10.192.0.1: seq=0 ttl=64 time=1011.309 ms
64 bytes from 10.192.0.1: seq=1 ttl=64 time=11.212 ms
64 bytes from 10.192.0.1: seq=2 ttl=64 time=3.554 ms
Mon, Jan 13, 2:22 PM · Patch-For-Review, Operations, DBA, ops-codfw
Marostegui updated subscribers of T242481: d-i fails to install on servers with BRCM 2P 1G BT + 2P 10G SFP NDC.

Last test done with @ayounsi:
I have removed the drivers tg3 and bnxt_en from the OS:

/bin # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
/bin # ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
Mon, Jan 13, 2:02 PM · Patch-For-Review, Operations, DBA, ops-codfw
Marostegui added a comment to P10135 (An Untitled Masterwork).
/bin # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
Mon, Jan 13, 1:59 PM
Marostegui added a comment to P10137 (An Untitled Masterwork).
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:2c:04 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:2c:05 brd ff:ff:ff:ff:ff:ff
Mon, Jan 13, 1:58 PM
Marostegui added a comment to P10137 (An Untitled Masterwork).
/bin # ip link set eno1 down
/bin # ip link set eno2d1 down
/bin # ip link set eno3 down
/bin # ip link set eno4 down
/bin # ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:2c:06 brd ff:ff:ff:ff:ff:ff
3: eno2d1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:2c:07 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:2c:04 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:2c:05 brd ff:ff:ff:ff:ff:ff
Mon, Jan 13, 1:56 PM
Marostegui added a comment to P10137 (An Untitled Masterwork).
/bin # ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:2c:06 brd ff:ff:ff:ff:ff:ff
3: eno2d1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:2c:07 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:2c:04 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:2c:05 brd ff:ff:ff:ff:ff:ff
Mon, Jan 13, 1:55 PM
Marostegui created P10137 (An Untitled Masterwork).
Mon, Jan 13, 1:45 PM
Marostegui updated the task description for T239238: Switchover s8 primary database master db1109 -> db1104 - Date TBD.
Mon, Jan 13, 11:19 AM · DBA
Marostegui added a comment to T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language.

We might be doing a DC switchover in Q3, so we can wait till eqiad is passive to address the final alter on the current primary master.

Mon, Jan 13, 11:13 AM · DBA, Blocked-on-schema-change, Wikidata
Marostegui added a comment to T242481: d-i fails to install on servers with BRCM 2P 1G BT + 2P 10G SFP NDC.

The hack didn't work. It keeps choosing a different NIC than eno3, which unfortunately looks like the 10G one.
I tried a different approach which was uninstalling the kernel module BEFORE running the network check, and then I generated the interfaces file manually, so it would only pick eno3, but the installer now looks for eno1 which doesn't exist, as I uninstalled the kernel module bnxt_en.

Mon, Jan 13, 10:51 AM · Patch-For-Review, Operations, DBA, ops-codfw
Daimona awarded T234052: Add abuse_filter_log.afl_filter_id and afl_global columns a Like token.
Mon, Jan 13, 10:37 AM · AbuseFilter, DBA, Blocked-on-schema-change
Marostegui added a comment to T242481: d-i fails to install on servers with BRCM 2P 1G BT + 2P 10G SFP NDC.

Thanks @MoritzMuehlenhoff - I have merged https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/563975/ to make sure we install (or test) es2024 with stretch, as ideally we'll install stretch on those hosts.
Buster comes with 10.4 and we are not ready for that yet. There is no 10.1 for buster.

Mon, Jan 13, 10:18 AM · Patch-For-Review, Operations, DBA, ops-codfw
Marostegui closed T234052: Add abuse_filter_log.afl_filter_id and afl_global columns, a subtask of T220791: afl_filter should be split in afl_filter_id and afl_global, as Resolved.
Mon, Jan 13, 9:39 AM · MW-1.34-notes (1.34.0-wmf.25; 2019-10-01), Core Platform Team Workboards (Clinic Duty Team), Schema-change, Patch-For-Review, AbuseFilter
Marostegui closed T234052: Add abuse_filter_log.afl_filter_id and afl_global columns as Resolved.

All done

Mon, Jan 13, 9:39 AM · AbuseFilter, DBA, Blocked-on-schema-change
Marostegui updated the task description for T234052: Add abuse_filter_log.afl_filter_id and afl_global columns.
Mon, Jan 13, 9:38 AM · AbuseFilter, DBA, Blocked-on-schema-change
Marostegui added a comment to T242481: d-i fails to install on servers with BRCM 2P 1G BT + 2P 10G SFP NDC.

I have tried to look thru the BIOS to find a way to disable the 10G capability but I have found nothing.

Mon, Jan 13, 9:37 AM · Patch-For-Review, Operations, DBA, ops-codfw
Marostegui added a comment to P10135 (An Untitled Masterwork).
Integrated NIC 1 Port 1: Broadcom Adv. Dual 10Gb Ethernet -
B0:26:28:F5:2C:06
Main Configuration Page
Mon, Jan 13, 8:25 AM
Marostegui created P10135 (An Untitled Masterwork).
Mon, Jan 13, 8:24 AM
Marostegui added a comment to T242481: d-i fails to install on servers with BRCM 2P 1G BT + 2P 10G SFP NDC.

@Marostegui I will focus more on troubleshooting this on the NIC level on Monday. Since the 1GB and 10GB interfaces are on the same card, I will use the 10GB interface over the 1GB interface and see if if that helps. Have a great week end.

Mon, Jan 13, 8:10 AM · Patch-For-Review, Operations, DBA, ops-codfw
Marostegui updated the task description for T239791: DB: perform rolling restart of mariadb deamons to pick up CA changes.
Mon, Jan 13, 6:51 AM · DBA, User-jbond, Puppet, Operations
Marostegui updated the task description for T234052: Add abuse_filter_log.afl_filter_id and afl_global columns.
Mon, Jan 13, 6:33 AM · AbuseFilter, DBA, Blocked-on-schema-change
Marostegui triaged T242571: Automatically populate tendril/zarcillo with the list of databases in the infrastructure as Medium priority.
Mon, Jan 13, 6:05 AM · DBA
Marostegui created T242571: Automatically populate tendril/zarcillo with the list of databases in the infrastructure.
Mon, Jan 13, 6:05 AM · DBA
Marostegui closed T241387: Extend flow_wiki_ref.ref_src_wiki as Resolved.

All done

Mon, Jan 13, 5:56 AM · User-RhinosF1, Blocked-on-schema-change, DBA
Marostegui updated the task description for T241387: Extend flow_wiki_ref.ref_src_wiki.
Mon, Jan 13, 5:56 AM · User-RhinosF1, Blocked-on-schema-change, DBA
Marostegui updated the task description for T239453: Remove partitions from revision table.
Mon, Jan 13, 5:54 AM · DBA
Marostegui updated the task description for T232446: Compress new Wikibase tables.
Mon, Jan 13, 5:52 AM · DBA