Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts
Open, Needs TriagePublic

Description

The following shards needs to be imported and sanitized into db1095 (sanitarium2) and labsdb1009,1010 and 1011

  • s2
  • s4
  • s5
  • s6
  • s7
There are a very large number of changes, so older changes are hidden. Show Older Changes

data has been sanitized in all the hosts (sanitarium2, labsdb1009,10,11).
Triggers have been enabled on commonswiki on sanitarium2

I have run the check_private_data script on all the hosts and nothing is being reported there.

Pending:

create the views - @chasemp / @yuvipanda is that something you will take care of?
add the grants for commonswiki

BTW, we said that if we took care manually of the grants, maybe that could be automatized so we do not bother you much? (it is ok to say no/not yet, I just do not want to bother you unnecessarily).

Sure, I'll run it today. No worries on the ping for views, once we have everything there and it's down to changes to actual existing views I think I'll make the views run a notify in Puppet on demand. I'm still feeling more comfortable running it actively for now though. Still looking for surprises as it were :)

maintain-views --all-databases --replace-all --debug

on all three, small note labsdb1010 had that same issue which needed a flush privs or it would push back with:

pymysql.err.OperationalError: (1142, "CREATE VIEW command denied to user 'maintainviews'@'localhost' for table 'abuse_filter_action'"

We are not exactly sure why this pops up randomly at the moment.

maintain-views --all-databases --replace-all --debug

on all three, small note labsdb1010 had that same issue which needed a flush privs or it would push back with:

pymysql.err.OperationalError: (1142, "CREATE VIEW command denied to user 'maintainviews'@'localhost' for table 'abuse_filter_action'"

We are not exactly sure why this pops up randomly at the moment.

So for the record, the above issue has been fixed by @jcrespo doing a flush privileges.
As per @chasemp the views are looking good.
And I have just added the labsdbuser grants to access commonswiki view on labsdb1009,10 and 11

thanks guys!

Mentioned in SAL (#wikimedia-operations) [2017-02-09T07:41:38Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1064 - T153743 (duration: 00m 40s)

Change 336760 had a related patch set uploaded (by Marostegui):
db-eqiad.php: Add comment for db1064

https://gerrit.wikimedia.org/r/336760

Change 336760 merged by jenkins-bot:
db-eqiad.php: Add comment for db1064

https://gerrit.wikimedia.org/r/336760

Mentioned in SAL (#wikimedia-operations) [2017-02-09T07:47:16Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Added comment for db1064 being master of db1095 - T153743 (duration: 00m 40s)

Mentioned in SAL (#wikimedia-operations) [2017-02-09T08:17:18Z] <marostegui> Compressing commonswiki tables on db1095 - T153743

Mentioned in SAL (#wikimedia-operations) [2017-02-09T16:15:35Z] <marostegui> Compressing commonswiki on labsdb1009 - T153743

Mentioned in SAL (#wikimedia-operations) [2017-02-13T07:03:21Z] <marostegui> Compressing commonswiki tables on labsdb1010 and labsdb1011 - T153743

Marostegui added a comment.EditedFeb 13 2017, 7:07 AM

commonswiki has been compressed on db1095 and labsdb1009. It has gone from around 1.3T to 495G. There is still 1.6T available on db1095, so I believe we can try to import dewiki and wikidatawiki there.
Checking the master there db1049 shows that dewiki is around 130G and wikidatawiki is around 260G which sanitized and compressed will be a lot less.

db1071 a random slaves shows bigger numbers though (which are probably the ones we need to keep in mind for db1095)

dewiki -> 382G
wikidatawiki -> 858G

Those will be reduced a lot anyways by the sanitization and the compression

On all three new servers :)

2017-02-14 17:03:54,237 DEBUG SQL:
                    CREATE OR REPLACE
                    DEFINER=viewmaster
                    VIEW `gnwikibooks_p`.`abuse_filter_action`
                    AS SELECT * FROM `gnwikibooks`.`abuse_filter_action`;

maintain-views --all-databases --replace-all --debug
pymysql.err.OperationalError: (1142, "CREATE VIEW command denied to user 'maintainviews'@'localhost' for table 'abuse_filter_action'")

gnwikibooks?

Try now on labsdb1009.

Oh I thought that was solved a few posts before with a FLUSH PRIVILEGES

gnwikibooks?

It seems order of creation for views is not deterministic. that was on labsdb1011.

labsdb1009

2017-02-14 17:03:26,853 DEBUG SQL:
                   CREATE OR REPLACE
                   DEFINER=viewmaster
                   VIEW `ugwiki_p`.`abuse_filter_action`
                   AS SELECT * FROM `ugwiki`.`abuse_filter_action`;

labsdb1010

2017-02-14 17:03:42,993 DEBUG SQL:
                    CREATE OR REPLACE
                    DEFINER=viewmaster
                    VIEW `wowiktionary_p`.`abuse_filter_action`
                    AS SELECT * FROM `wowiktionary`.`abuse_filter_action`;

If I get specific it's the same issue though:

root@labsdb1010:~# maintain-views --databases enwiki --replace-all --debug
2017-02-14 17:10:42,937 DEBUG Removing 0 dbs as sensitive
2017-02-14 17:10:42,939 INFO Full views for enwiki:
2017-02-14 17:10:42,942 INFO [abuse_filter_action]
2017-02-14 17:10:42,942 DEBUG SQL:
                    CREATE OR REPLACE
                    DEFINER=viewmaster
                    VIEW `enwiki_p`.`abuse_filter_action`
                    AS SELECT * FROM `enwiki`.`abuse_filter_action`;

pymysql.err.OperationalError: (1142, "CREATE VIEW command denied to user 'maintainviews'@'localhost' for table 'abuse_filter_action'")

Try now on labsdb1009.

nope

root@labsdb1009:~# maintain-views --databases enwiki --replace-all

Traceback (most recent call last):
  File "/usr/local/sbin/maintain-views", line 442, in <module>
    ops.execute(fullviews, customviews)
  File "/usr/local/sbin/maintain-views", line 272, in execute
    self.do_fullview(view)
  File "/usr/local/sbin/maintain-views", line 125, in do_fullview
    """.format(self.definer, self.db_p, view, self.db))
  File "/usr/local/sbin/maintain-views", line 53, in write_execute
    self.cursor.execute(query)
  File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 167, in execute
    result = self._query(query)
  File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 323, in _query
    conn.query(q)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 836, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1020, in _read_query_result
    result.read()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1303, in read
    first_packet = self.connection._read_packet()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 982, in _read_packet
    packet.check_error()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 394, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/lib/python3/dist-packages/pymysql/err.py", line 120, in raise_mysql_exception
    _check_mysql_exception(errinfo)
  File "/usr/lib/python3/dist-packages/pymysql/err.py", line 112, in _check_mysql_exception
    raise errorclass(errno, errorvalue)
pymysql.err.OperationalError: (1142, "CREATE VIEW command denied to user 'maintainviews'@'localhost' for table 'abuse_filter_action'")

Then this is a different issue than the FLUSH STATUS one. Checking.

gnwikibooks?

It seems order of creation for views is not deterministic. that was on labsdb1011.

labsdb1009

2017-02-14 17:03:26,853 DEBUG SQL:
                    CREATE OR REPLACE
                    DEFINER=viewmaster
                    VIEW `ugwiki_p`.`abuse_filter_action`
                    AS SELECT * FROM `ugwiki`.`abuse_filter_action`;

labsdb1010

2017-02-14 17:03:42,993 DEBUG SQL:
                     CREATE OR REPLACE
                     DEFINER=viewmaster
                     VIEW `wowiktionary_p`.`abuse_filter_action`
                     AS SELECT * FROM `wowiktionary`.`abuse_filter_action`;

If I get specific it's the same issue though:

root@labsdb1010:~# maintain-views --databases enwiki --replace-all --debug
 2017-02-14 17:10:42,937 DEBUG Removing 0 dbs as sensitive
 2017-02-14 17:10:42,939 INFO Full views for enwiki:
 2017-02-14 17:10:42,942 INFO [abuse_filter_action]
 2017-02-14 17:10:42,942 DEBUG SQL:
                     CREATE OR REPLACE
                     DEFINER=viewmaster
                     VIEW `enwiki_p`.`abuse_filter_action`
                     AS SELECT * FROM `enwiki`.`abuse_filter_action`;

pymysql.err.OperationalError: (1142, "CREATE VIEW command denied to user 'maintainviews'@'localhost' for table 'abuse_filter_action'")

But the view already exists, right? At least on labsdb1009

Maybe it lacks the grants for replace (drop). I have dropped [ugwiki_p]> drop view abuse_filter_action;. Can you run the same query only for ugwiki ?

Yes, it does exist but to update the views it's easier to say "update all of them to match canonical source replacing any that already exist". Mainly because a certain custom view for a certain table can change and even if the view exists it is now stale when compared to our yaml definition. That has been the historical scheme anyway. I added logic to specify a certain wikidb for ease and sanity (and even a particular table of a particular wikidb) but it's still common if there are a wide variety of new wikidb's or odds and ends to update to do it en mass and replace existing.

It is failing to replace an existing view, which may be interesting?

Could be but the grants are ALL as per this (labsdb1009) so I would assume the replace should go through no?:

GRANT ALL PRIVILEGES ON `%wik%\_p`.* TO 'maintainviews'@'localhost'
chasemp added a comment.EditedFeb 14 2017, 5:28 PM

Maybe it lacks the grants for replace (drop). I have dropped [ugwiki_p]> drop view abuse_filter_action;. Can you run the same query only for ugwiki ?

root@labsdb1009:~# maintain-views --databases ugwiki --table abuse_filter_action
pymysql.err.OperationalError: (1142, "CREATE VIEW command denied to user 'maintainviews'@'localhost' for table 'abuse_filter_action'")

edit:

2017-02-14 17:28:56,409 DEBUG SQL:
                    CREATE OR REPLACE
                    DEFINER=viewmaster
                    VIEW `ugwiki_p`.`abuse_filter_action`
                    AS SELECT * FROM `ugwiki`.`abuse_filter_action`;

Silly question, are you connecting through localhost specifically?
I am thinking about the issue we had in September where it was doing something weird with localhost/127.0.0.1 and the client was getting a bit crazy. I cannot remember all the details...

Silly question, are you connecting through localhost specifically?
I am thinking about the issue we had in September where it was doing something weird with localhost/127.0.0.1 and the client was getting a bit crazy. I cannot remember all the details...

Yup, not silly at all. It hasn't changed in awhile but it's using:

dbh = pymysql.connect(
    user=config["mysql_user"],
    passwd=config["mysql_password"],
    unix_socket="/tmp/mysql.sock",
    charset="utf8"
)

where user/pass are defined in /etc/maintain-views.yaml

I've added a workaround that makes no sense but that works for now ,we need to revisit it in the future. Maybe there is a bug on the latest version?

I've added a workaround that makes no sense but that works for now ,we need to revisit it in the future. Maybe there is a bug on the latest version?

Just for the record and future references, I believe this is what was added:

GRANT SELECT, DROP, CREATE VIEW ON `%wik%`.* TO 'maintainviews'@'localhost'

Which as Jaime says, makes not much sense that it needs those privileges (DROP and CREATE) on the original databases...

@Marostegui : Quick update on that one as well - How many shards are still to be done?
Thanks !

@Marostegui : Quick update on that one as well - How many shards are still to be done?
Thanks !

Hi!

We are limited by disk space on the hosts, I believe we will soon import s5 (dewiki and wikidata). After that we need to reavaluate how much disk space we still available there.
Reminder it already has:

s1 (enwiki)
s3 (around 800 wikis)
s4 (commons)

Thanks for the headsup @Marostegui :)

Shall we go for s5 or s2 next? I am not sure we can do both (because of disk space issues).
s2 has more wikis, but s5 has wikidata :)

s5 for me

Agreed.
If that is the case, I believe db1070 can be a good option to be sanitarium's master.
It is the vslow slave now, and by checking the future hardware and how the shard will look like after the new hardware comes, db1070 will remain there being vslow slave. So we'd not need to change sanitarium's masters after all the hardware refresh.

Change 341007 had a related patch set uploaded (by marostegui):
[operations/puppet] site.pp: Enable ROW binlog for db1070

https://gerrit.wikimedia.org/r/341007

Change 341759 had a related patch set uploaded (by Marostegui):
[operations/mediawiki-config] db-eqiad.php: Depool db1070

https://gerrit.wikimedia.org/r/341759

Change 341007 merged by Marostegui:
[operations/puppet] site.pp: Enable ROW binlog for db1070

https://gerrit.wikimedia.org/r/341007

Change 341759 merged by jenkins-bot:
[operations/mediawiki-config] db-eqiad.php: Depool db1070

https://gerrit.wikimedia.org/r/341759

Mentioned in SAL (#wikimedia-operations) [2017-03-08T08:32:45Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1070 - T153743 (duration: 00m 41s)

Mentioned in SAL (#wikimedia-operations) [2017-03-08T08:36:15Z] <marostegui> Restart mysql on db1070 to change binlog to ROW - T153743

Mentioned in SAL (#wikimedia-operations) [2017-03-08T09:00:58Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1070 - T153743 (duration: 00m 41s)

Change 341797 had a related patch set uploaded (by Marostegui):
[operations/mediawiki-config] db-eqiad.php: Add a few comments

https://gerrit.wikimedia.org/r/341797

Change 341797 merged by jenkins-bot:
[operations/mediawiki-config] db-eqiad.php: Add a few comments

https://gerrit.wikimedia.org/r/341797

db1070 has been restarted with ROW based replication to serve as a master for db1095.

Mentioned in SAL (#wikimedia-operations) [2017-03-08T17:52:26Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: db1070 ROW based replication comments - T153743 (duration: 00m 41s)

In order to start getting ready to import s5 on sanitarium2 and labsdb1009,10,11 I am going to start:

  • Compressing InnoDB on db1070 (master for sanitarium2)
  • Backuping db1095 just in case

Change 342427 had a related patch set uploaded (by Marostegui):
[operations/mediawiki-config] db-eqiad.php: Depool db1070

https://gerrit.wikimedia.org/r/342427

Change 342427 merged by jenkins-bot:
[operations/mediawiki-config] db-eqiad.php: Depool db1070

https://gerrit.wikimedia.org/r/342427

Mentioned in SAL (#wikimedia-operations) [2017-03-13T08:27:40Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1070 - T153743 (duration: 00m 41s)

Mentioned in SAL (#wikimedia-operations) [2017-03-13T08:30:42Z] <marostegui> Stop MySQL on db1095 (sanitarium2) to take a backup - T153743

Mentioned in SAL (#wikimedia-operations) [2017-03-13T08:31:30Z] <marostegui> Stop replication on labsdb1009,10 and 11 - T153743

labsdb1009,10 and 11 - replication stopped
db1095 replication stopped and mysql down
data transfer between db1095 and dbstore1001 is now happening.

Mentioned in SAL (#wikimedia-operations) [2017-03-13T08:40:43Z] <marostegui> Compress dewiki - db1070 - T153743

Marostegui moved this task from Next to In progress on the DBA board.Mar 13 2017, 9:46 AM

I just realised that db1070 doesn't have .ibd files because of this: T137191
So I think I will reclone that host from a host that does have file per table and then we can export dewiki and wikidata do sanitarium2.

Probably also do a re-image won't hurt.

Marostegui moved this task from In progress to Next on the DBA board.Mar 13 2017, 10:21 AM

Mentioned in SAL (#wikimedia-operations) [2017-04-03T06:41:21Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1070 to compress it - T153743 (duration: 00m 44s)

I am not going to import anything this week most likely, but I am advancing on the task to import s5, by starting the compression on db1070 before importing its data into db1095

db1070 has been compressed.

Mentioned in SAL (#wikimedia-operations) [2017-04-06T05:41:51Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1070 after compression - T153743 (duration: 00m 51s)

Mentioned in SAL (#wikimedia-operations) [2017-05-29T07:38:30Z] <marostegui> Stop MySQL on db1095 to take a backup - this will make labsdb1009,10 and 11 break replication while it is down - T153743

Change 356027 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1070

https://gerrit.wikimedia.org/r/356027

Change 356027 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1070

https://gerrit.wikimedia.org/r/356027

Mentioned in SAL (#wikimedia-operations) [2017-05-29T13:07:45Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1070 - T153743 (duration: 00m 41s)

Mentioned in SAL (#wikimedia-operations) [2017-05-29T13:10:20Z] <marostegui> Stop replication on db1070 to flush tables for export - T153743

Marostegui moved this task from Next to In progress on the DBA board.Mon, May 29, 2:14 PM

Mentioned in SAL (#wikimedia-operations) [2017-05-29T14:14:32Z] <marostegui> Stop MySQL labsdb1009 to take a backup - T153743

After transferring all the ibd and cfg files and run md5sum over them I am now importing wikidatawiki tablespace on db1095.
Once it is done, I will check all the tables and if everything is fine, will start with dewiki

(As I was expecting) I have seen some issues when importing compressed tablespaces. I am troubleshooting it.

Marostegui added a comment.EditedTue, May 30, 7:58 AM

While troubleshooting db1095 gave a nasty error and I do not trust it much now
I have read in a few posts that compression + transportable spaces isn't a good combination and sometimes you can face issues which I am facing now.
I am going to recover db1095's backup and we should think another way of copying it

  1. Use another non compressed host
  2. mydumper
70530  7:52:50 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.1.23-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=101
max_threads=102
thread_count=4
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 355098 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f6b1a114008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f6e0eacee38 thread_stack 0x30000
*** buffer overflow detected ***: /opt/wmf-mariadb101/bin/mysqld terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x731af)[0x7fe06348c1af]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7fe063511aa7]
/lib/x86_64-linux-gnu/libc.so.6(+0xf6cc0)[0x7fe06350fcc0]
/lib/x86_64-linux-gnu/libc.so.6(+0xf8a17)[0x7fe063511a17]
/opt/wmf-mariadb101/bin/mysqld(my_addr_resolve+0x48)[0x56103c78a7b8]
/opt/wmf-mariadb101/bin/mysqld(my_print_stacktrace+0x1d2)[0x56103c776df2]
/opt/wmf-mariadb101/bin/mysqld(handle_fatal_signal+0x2fd)[0x56103c29cc4d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7fe064cfb890]
/opt/wmf-mariadb101/bin/mysqld(+0x726e19)[0x56103c415e19]
/opt/wmf-mariadb101/bin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x2136)[0x56103c1a3296]
/opt/wmf-mariadb101/bin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x5bc)[0x56103c1ea4ec]
/opt/wmf-mariadb101/bin/mysqld(_Z21mysql_execute_commandP3THD+0x12f1)[0x56103c1112d1]
/opt/wmf-mariadb101/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x275)[0x56103c118e05]
/opt/wmf-mariadb101/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1c43)[0x56103c11b813]
/opt/wmf-mariadb101/bin/mysqld(_Z10do_commandP3THD+0x14e)[0x56103c11bf3e]
/opt/wmf-mariadb101/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x182)[0x56103c1e7872]
/opt/wmf-mariadb101/bin/mysqld(handle_one_connection+0x40)[0x56103c1e7a10]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064)[0x7fe064cf4064]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe06350162d]
======= Memory map: ========
56103bcef000-56103cca9000 r-xp 00000000 08:01 2396096                    /opt/wmf-mariadb101/bin/mysqld
56103cea8000-56103cf62000 r--p 00fb9000 08:01 2396096                    /opt/wmf-mariadb101/bin/mysqld
56103cf62000-56103d016000 rw-p 01073000 08:01 2396096                    /opt/wmf-mariadb101/bin/mysqld
56103d016000-56103d893000 rw-p 00000000 00:00 0
7f6b16400000-7f6be9800000 rw-p 00000000 00:00 0
7f6be9b31000-7f6be9b32000 ---p 00000000 00:00 0
7f6be9b32000-7f6be9b62000 rw-p 00000000 00:00 0
7f6be9b62000-7f6be9b63000 ---p 00000000 00:00 0
7f6be9b63000-7f6be9b93000 rw-p 00000000 00:00 0
7f6be9b93000-7f6be9b94000 ---p 00000000 00:00 0
7f6be9b94000-7f6be9bc4000 rw-p 00000000 00:00 0
7f6be9bc4000-7f6be9bc5000 ---p 00000000 00:00 0
7f6be9bc5000-7f6be9bf5000 rw-p 00000000 00:00 0
7f6be9bf5000-7f6be9bf6000 ---p 00000000 00:00 0
7f6be9bf6000-7f6bea3f6000 rw-p 00000000 00:00 0
7f6bea3f6000-7f6bea3f7000 ---p 00000000 00:00 0
7f6bea3f7000-7f6beabf7000 rw-p 00000000 00:00 0
7f6beabf7000-7f6beabf8000 ---p 00000000 00:00 0
7f6beabf8000-7f6beb3f8000 rw-p 00000000 00:00 0
7f6beb3f8000-7f6beb3f9000 ---p 00000000 00:00 0
7f6beb3f9000-7f6bebbf9000 rw-p 00000000 00:00 0
7f6bebbf9000-7f6bebbfa000 ---p 00000000 00:00 0
7f6bebbfa000-7f6bec3fa000 rw-p 00000000 00:00 0
7f6bec3fa000-7f6bec3fb000 ---p 00000000 00:00 0
7f6bec3fb000-7f6becbfb000 rw-p 00000000 00:00 0
7f6becbfb000-7f6becbfc000 ---p 00000000 00:00 0
7f6becbfc000-7f6bed3fc000 rw-p 00000000 00:00 0
7f6bed3fc000-7f6bed3fd000 ---p 00000000 00:00 0
7f6bed3fd000-7f6bedbfd000 rw-p 00000000 00:00 0
7f6bedbfd000-7f6bedbfe000 ---p 00000000 00:00 0
7f6bedbfe000-7f6bee3fe000 rw-p 00000000 00:00 0
7f6bee3fe000-7f6bee3ff000 ---p 00000000 00:00 0
7f6bee3ff000-7f6beebff000 rw-p 00000000 00:00 0
7f6beebff000-7f6beec00000 ---p 00000000 00:00 0
7f6beec00000-7f6c5bc00000 rw-p 00000000 00:00 0
7f6c5bc2b000-7f6c5bc2c000 ---p 00000000 00:00 0
7f6c5bc2c000-7f6c5bc5c000 rw-p 00000000 00:00 0
7f6c5bc5c000-7f6c5bc5d000 ---p 00000000 00:00 0
7f6c5bc5d000-7f6c5bc8d000 rw-p 00000000 00:00 0
7f6c5bc8d000-7f6c5bc8e000 ---p 00000000 00:00 0
7f6c5bc8e000-7f6c5bcbe000 rw-p 00000000 00:00 0
7f6c5bcbe000-7f6c5bcbf000 ---p 00000000 00:00 0
7f6c5bcbf000-7f6c5bcef000 rw-p 00000000 00:00 0
7f6c5bcef000-7f6c5bcf0000 ---p 00000000 00:00 0
7f6c5bcf0000-7f6c5bd20000 rw-p 00000000 00:00 0
7f6c5bd20000-7f6c5bd21000 ---p 00000000 00:00 0
7f6c5bd21000-7f6c5bd51000 rw-p 00000000 00:00 0
7f6c5bd51000-7f6c5bd52000 ---p 00000000 00:00 0
7f6c5bd52000-7f6c5bd82000 rw-p 00000000 00:00 0
7f6c5bd82000-7f6c5bd83000 ---p 00000000 00:00 0
7f6c5bd83000-7f6c5bdb3000 rw-p 00000000 00:00 0
7f6c5bdb3000-7f6c5bdb4000 ---p 00000000 00:00 0
7f6c5bdb4000-7f6c5bde4000 rw-p 00000000 00:00 0
7f6c5bde4000-7f6c5bde5000 ---p 00000000 00:00 0
7f6c5bde5000-7f6c5be15000 rw-p 00000000 00:00 0
7f6c5be15000-7f6c5be16000 ---p 00000000 00:00 0
7f6c5be16000-7f6c5be46000 rw-p 00000000 00:00 0
7f6c5be46000-7f6c5be47000 ---p 00000000 00:00 0
7f6c5be47000-7f6c5be77000 rw-p 00000000 00:00 0
7f6c5be77000-7f6c5be78000 ---p 00000000 00:00 0
7f6c5be78000-7f6c5bea8000 rw-p 00000000 00:00 0
7f6c5bea8000-7f6c5bea9000 ---p 00000000 00:00 0
7f6c5bea9000-7f6c5bed9000 rw-p 00000000 00:00 0
7f6c5bed9000-7f6c5beda000 ---p 00000000 00:00 0
7f6c5beda000-7f6c5bf0a000 rw-p 00000000 00:00 0
7f6c5bf0a000-7f6c5bf0b000 ---p 00000000 00:00 0
7f6c5bf0b000-7f6c5bf3b000 rw-p 00000000 00:00 0
7f6c5bf3b000-7f6c5bf3c000 ---p 00000000 00:00 0
7f6c5bf3c000-7f6c5bf6c000 rw-p 00000000 00:00 0
7f6c5bf6c000-7f6c5bf6d000 ---p 00000000 00:00 0
7f6c5bf6d000-7f6c5bf9d000 rw-p 00000000 00:00 0
7f6c5bf9d000-7f6c5bf9e000 ---p 00000000 00:00 0
7f6c5bf9e000-7f6c5bfce000 rw-p 00000000 00:00 0
7f6c5bfce000-7f6c5bfcf000 ---p 00000000 00:00 0
7f6c5bfcf000-7f6c5bfff000 rw-p 00000000 00:00 0
7f6c5bfff000-7f6c5c000000 ---p 00000000 00:00 0
7f6c5c000000-7f6e0e800000 rw-p 00000000 00:00 0
7f6e0e822000-7f6e0e823000 ---p 00000000 00:00 0
7f6e0e823000-7f6e0e853000 rw-p 00000000 00:00 0
7f6e0e853000-7f6e0e854000 ---p 00000000 00:00 0
7f6e0e854000-7f6e0e884000 rw-p 00000000 00:00 0
7f6e0e884000-7f6e0e885000 ---p 00000000 00:00 0
7f6e0e885000-7f6e0e8b5000 rw-p 00000000 00:00 0
7f6e0e8b5000-7f6e0e8b6000 ---p 00000000 00:00 0
7f6e0e8b6000-7f6e0e8e6000 rw-p 00000000 00:00 0
7f6e0e8e6000-7f6e0e8e7000 ---p 00000000 00:00 0
7f6e0e8e7000-7f6e0e917000 rw-p 00000000 00:00 0
7f6e0e917000-7f6e0e918000 ---p 00000000 00:00 0
7f6e0e918000-7f6e0e948000 rw-p 00000000 00:00 0
7f6e0e948000-7f6e0e949000 ---p 00000000 00:00 0
7f6e0e949000-7f6e0e979000 rw-p 00000000 00:00 0
7f6e0e979000-7f6e0e97a000 ---p 00000000 00:00 0
7f6e0e97a000-7f6e0e9aa000 rw-p 00000000 00:00 0
7f6e0e9aa000-7f6e0e9ab000 ---p 00000000 00:00 0
7f6e0e9ab000-7f6e0e9db000 rw-p 00000000 00:00 0
7f6e0e9db000-7f6e0e9dc000 ---p 00000000 00:00 0
7f6e0e9dc000-7f6e0ea0c000 rw-p 00000000 00:00 0
7f6e0ea0c000-7f6e0ea0d000 ---p 00000000 00:00 0
7f6e0ea0d000-7f6e0ea3d000 rw-p 00000000 00:00 0
7f6e0ea3d000-7f6e0ea3e000 ---p 00000000 00:00 0
7f6e0ea3e000-7f6e0ea6e000 rw-p 00000000 00:00 0
7f6e0ea6e000-7f6e0ea6f000 ---p 00000000 00:00 0
7f6e0ea6f000-7f6e0ea9f000 rw-p 00000000 00:00 0
7f6e0ea9f000-7f6e0eaa0000 ---p 00000000 00:00 0
7f6e0eaa0000-7f6e0ead0000 rw-p 00000000 00:00 0
7f6e0ead0000-7f6e0ead1000 ---p 00000000 00:00 0
7f6e0ead1000-7f6e0eb01000 rw-p 00000000 00:00 0
7f6e0eb01000-7f6e0eb02000 ---p 00000000 00:00 0
7f6e0eb02000-7f6e0eb32000 rw-p 00000000 00:00 0
7f6e0eb32000-7f6e0eb33000 ---p 00000000 00:00 0
7f6e0eb33000-7f6e0eb63000 rw-p 00000000 00:00 0
7f6e0eb63000-7f6e0eb64000 ---p 00000000 00:00 0
7f6e0eb64000-7f6e0eb94000 rw-p 00000000 00:00 0
7f6e0eb94000-7f6e0eb95000 ---p 00000000 00:00 0
7f6e0eb95000-7f6e0ebc5000 rw-p 00000000 00:00 0
7f6e0ebc5000-7f6e0ebc6000 ---p 00000000 00:00 0
7f6e0ebc6000-7f6e0ebf6000 rw-p 00000000 00:00 0
7f6e0ebf6000-7f6e0ebf7000 ---p 00000000 00:00 0
7f6e0ebf7000-7f6e0f3f7000 rw-p 00000000 00:00 0
7f6e0f3f7000-7f6e0f3f8000 ---p 00000000 00:00 0
7f6e0f3f8000-7f6e0fbf8000 rw-p 00000000 00:00 0
7f6e0fbf8000-7f6e0fbf9000 ---p 00000000 00:00 0
7f6e0fbf9000-7f6e103f9000 rw-p 00000000 00:00 0
7f6e103f9000-7f6e103fa000 ---p 00000000 00:00 0
7f6e103fa000-7f6e10bfa000 rw-p 00000000 00:00 0
7f6e10bfa000-7f6e10bfb000 ---p 00000000 00:00 0
7f6e10bfb000-7f6e113fb000 rw-p 00000000 00:00 0
7f6e113fb000-7f6e113fc000 ---p 00000000 00:00 0
7f6e113fc000-7f6e11bfc000 rw-p 00000000 00:00 0
7f6e11bfc000-7f6e11bfd000 ---p 00000000 00:00 0
7f6e11bfd000-7f6e123fd000 rw-p 00000000 00:00 0
7f6e123fd000-7f6e123fe000 ---p 00000000 00:00 0
7f6e123fe000-7f6e12bfe000 rw-p 00000000 00:00 0
7f6e12bfe000-7f6e12bff000 ---p 00000000 00:00 0
7f6e12bff000-7f6e133ff000 rw-p 00000000 00:00 0
7f6e133ff000-7f6e13400000 ---p 00000000 00:00 0
7f6e13400000-7f7015000000 rw-p 00000000 00:00 0
7f701501a000-7f701501b000 ---p 00000000 00:00 0
7f701501b000-7f701504b000 rw-p 00000000 00:00 0
7f701504b000-7f701504c000 ---p 00000000 00:00 0
7f701504c000-7f701507c000 rw-p 00000000 00:00 0
7f701507c000-7f701507d000 ---p 00000000 00:00 0
7f701507d000-7f70150ad000 rw-p 00000000 00:00 0
7f70150ad000-7f70150ae000 ---p 00000000 00:00 0
7f70150ae000-7f70150de000 rw-p 00000000 00:00 0
7f70150de000-7f70150df000 ---p 00000000 00:00 0
7f70150df000-7f701510f000 rw-p 00000000 00:00 0
7f701510f000-7f7015110000 ---p 00000000 00:00 0
7f7015110000-7f7e1e400000 rw-p 00000000 00:00 0
7f7e1e41a000-7f7e1e41b000 ---p 00000000 00:00 0
7f7e1e41b000-7f7e1e44b000 rw-p 00000000 00:00 0
7f7e1e44b000-7f7e1e44c000 ---p 00000000 00:00 0
7f7e1e44c000-7f7e1e47c000 rw-p 00000000 00:00 0
7f7e1e47c000-7f7e1e47d000 ---p 00000000 00:00 0
7f7e1e47d000-7f7e1e4ad000 rw-p 00000000 00:00 0
7f7e1e4ad000-7f7e1e4ae000 ---p 00000000 00:00 0
7f7e1e4ae000-7f7e1e4de000 rw-p 00000000 00:00 0
7f7e1e4de000-7f7e1e4df000 ---p 00000000 00:00 0
7f7e1e4df000-7f7e1e50f000 rw-p 00000000 00:00 0
7f7e1e50f000-7f7e1e510000 ---p 00000000 00:00 0
7f7e1e510000-7f8c27800000 rw-p 00000000 00:00 0
7f8c2781a000-7f8c2781b000 ---p 00000000 00:00 0
7f8c2781b000-7f8c2784b000 rw-p 00000000 00:00 0
7f8c2784b000-7f8c2784c000 ---p 00000000 00:00 0
7f8c2784c000-7f8c2787c000 rw-p 00000000 00:00 0
7f8c2787c000-7f8c2787d000 ---p 00000000 00:00 0
7f8c2787d000-7f8c278ad000 rw-p 00000000 00:00 0
7f8c278ad000-7f8c278ae000 ---p 00000000 00:00 0
7f8c278ae000-7f8c278de000 rw-p 00000000 00:00 0
7f8c278de000-7f8c278df000 ---p 00000000 00:00 0
7f8c278df000-7f8c2790f000 rw-p 00000000 00:00 0
7f8c2790f000-7f8c27910000 ---p 00000000 00:00 0
7f8c27910000-7f9a30c00000 rw-p 00000000 00:00 0
7f9a30c1a000-7f9a30c1b000 ---p 00000000 00:00 0
7f9a30c1b000-7f9a30c4b000 rw-p 00000000 00:00 0
7f9a30c4b000-7f9a30c4c000 ---p 00000000 00:00 0
7f9a30c4c000-7f9a30c7c000 rw-p 00000000 00:00 0
7f9a30c7c000-7f9a30c7d000 ---p 00000000 00:00 0
7f9a30c7d000-7f9a30cad000 rw-p 00000000 00:00 0
7f9a30cad000-7f9a30cae000 ---p 00000000 00:00 0
7f9a30cae000-7f9a30cde000 rw-p 00000000 00:00 0
7f9a30cde000-7f9a30cdf000 ---p 00000000 00:00 0
7f9a30cdf000-7f9a30d0f000 rw-p 00000000 00:00 0
7f9a30d0f000-7f9a30d10000 ---p 00000000 00:00 0
7f9a30d10000-7fa83a400000 rw-p 00000000 00:00 0
7fa83a41a000-7fa83a41b000 ---p 00000000 00:00 0
7fa83a41b000-7fa83a44b000 rw-p 00000000 00:00 0
7fa83a44b000-7fa83a44c000 ---p 00000000 00:00 0
7fa83a44c000-7fa83a47c000 rw-p 00000000 00:00 0
7fa83a47c000-7fa83a47d000 ---p 00000000 00:00 0
7fa83a47d000-7fa83a4ad000 rw-p 00000000 00:00 0
7fa83a4ad000-7fa83a4ae000 ---p 00000000 00:00 0
7fa83a4ae000-7fa83a4de000 rw-p 00000000 00:00 0
7fa83a4de000-7fa83a4df000 ---p 00000000 00:00 0
7fa83a4df000-7fa83a50f000 rw-p 00000000 00:00 0
7fa83a50f000-7fa83a510000 ---p 00000000 00:00 0
7fa83a510000-7fb643800000 rw-p 00000000 00:00 0
7fb64381a000-7fb64381b000 ---p 00000000 00:00 0
7fb64381b000-7fb64384b000 rw-p 00000000 00:00 0
7fb64384b000-7fb64384c000 ---p 00000000 00:00 0
7fb64384c000-7fb64387c000 rw-p 00000000 00:00 0
7fb64387c000-7fb64387d000 ---p 00000000 00:00 0
7fb64387d000-7fb6438ad000 rw-p 00000000 00:00 0
7fb6438ad000-7fb6438ae000 ---p 00000000 00:00 0
7fb6438ae000-7fb6438de000 rw-p 00000000 00:00 0
7fb6438de000-7fb6438df000 ---p 00000000 00:00 0
7fb6438df000-7fb64390f000 rw-p 00000000 00:00 0
7fb64390f000-7fb643910000 ---p 00000000 00:00 0
7fb643910000-7fc44cc00000 rw-p 00000000 00:00 0
7fc44cc1a000-7fc44cc1b000 ---p 00000000 00:00 0
7fc44cc1b000-7fc44cc4b000 rw-p 00000000 00:00 0
7fc44cc4b000-7fc44cc4c000 ---p 00000000 00:00 0
7fc44cc4c000-7fc44cc7c000 rw-p 00000000 00:00 0
7fc44cc7c000-7fc44cc7d000 ---p 00000000 00:00 0
7fc44cc7d000-7fc44ccad000 rw-p 00000000 00:00 0
7fc44ccad000-7fc44ccae000 ---p 00000000 00:00 0
7fc44ccae000-7fc44ccde000 rw-p 00000000 00:00 0
7fc44ccde000-7fc44ccdf000 ---p 00000000 00:00 0
7fc44ccdf000-7fc44cd0f000 rw-p 00000000 00:00 0
7fc44cd0f000-7fc44cd10000 ---p 00000000 00:00 0
7fc44cd10000-7fd256000000 rw-p 00000000 00:00 0
7fd25601a000-7fd25601b000 ---p 00000000 00:00 0
7fd25601b000-7fd25604b000 rw-p 00000000 00:00 0
7fd25604b000-7fd25604c000 ---p 00000000 00:00 0
7fd25604c000-7fd25607c000 rw-p 00000000 00:00 0
7fd25607c000-7fd25607d000 ---p 00000000 00:00 0
7fd25607d000-7fd2560ad000 rw-p 00000000 00:00 0
7fd2560ad000-7fd2560ae000 ---p 00000000 00:00 0
7fd2560ae000-7fd2560de000 rw-p 00000000 00:00 0
7fd2560de000-7fd2560df000 ---p 00000000 00:00 0
7fd2560df000-7fd25610f000 rw-p 00000000 00:00 0
7fd25610f000-7fd256110000 ---p 00000000 00:00 0
7fd256110000-7fe060c00000 rw-p 00000000 00:00 0
7fe060c0b000-7fe060c0c000 ---p 00000000 00:00 0
7fe060c0c000-7fe060c3c000 rw-p 00000000 00:00 0
7fe060c3c000-7fe060c3d000 ---p 00000000 00:00 0
7fe060c3d000-7fe060c6d000 rw-p 00000000 00:00 0
7fe060c6d000-7fe060c6e000 ---p 00000000 00:00 0
7fe060c6e000-7fe060c9e000 rw-p 00000000 00:00 0
7fe060c9e000-7fe060c9f000 ---p 00000000 00:00 0
7fe060c9f000-7fe060ccf000 rw-p 00000000 00:00 0
7fe060ccf000-7fe060cd0000 ---p 00000000 00:00 0
7fe060cd0000-7fe060d00000 rw-p 00000000 00:00 0
7fe060d00000-7fe060d01000 ---p 00000000 00:00 0
7fe060d01000-7fe060d31000 rw-p 00000000 00:00 0
7fe060d31000-7fe060d32000 ---p 00000000 00:00 0
7fe060d32000-7fe060d62000 rw-p 00000000 00:00 0
7fe060d62000-7fe060d63000 ---p 00000000 00:00 0
7fe060d63000-7fe060d93000 rw-p 00000000 00:00 0
7fe060d93000-7fe060d94000 ---p 00000000 00:00 0
7fe060d94000-7fe060dc4000 rw-p 00000000 00:00 0
7fe060dc4000-7fe060dc7000 r-xp 00000000 08:01 2395956                    /opt/wmf-mariadb101/lib/plugin/semisync_slave.so
7fe060dc7000-7fe060fc6000 ---p 00003000 08:01 2395956                    /opt/wmf-mariadb101/lib/plugin/semisync_slave.so
7fe060fc6000-7fe060fc7000 r--p 00002000 08:01 2395956                    /opt/wmf-mariadb101/lib/plugin/semisync_slave.so
7fe060fc7000-7fe060fc8000 rw-p 00003000 08:01 2395956                    /opt/wmf-mariadb101/lib/plugin/semisync_slave.so
7fe060fc8000-7fe060fc9000 ---p 00000000 00:00 0
7fe060fc9000-7fe0617c9000 rw-p 00000000 00:00 0
7fe0617c9000-7fe0617d4000 r-xp 00000000 08:01 1226707                    /lib/x86_64-linux-gnu/libnss_files-2.19.so
7fe0617d4000-7fe0619d3000 ---p 0000b000 08:01 1226707                    /lib/x86_64-linux-gnu/libnss_files-2.19.so
7fe0619d3000-7fe0619d4000 r--p 0000a000 08:01 1226707                    /lib/x86_64-linux-gnu/libnss_files-2.19.so
7fe0619d4000-7fe0619d5000 rw-p 0000b000 08:01 1226707                    /lib/x86_64-linux-gnu/libnss_files-2.19.so
7fe0619d5000-7fe0619df000 r-xp 00000000 08:01 1226902                    /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7fe0619df000-7fe061bde000 ---p 0000a000 08:01 1226902                    /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7fe061bde000-7fe061bdf000 r--p 00009000 08:01 1226902                    /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7fe061bdf000-7fe061be0000 rw-p 0000a000 08:01 1226902                    /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7fe061be0000-7fe061bf5000 r-xp 00000000 08:01 1226563                    /lib/x86_64-linux-gnu/libnsl-2.19.so
7fe061bf5000-7fe061df4000 ---p 00015000 08:01 1226563                    /lib/x86_64-linux-gnu/libnsl-2.19.so
7fe061df4000-7fe061df5000 r--p 00014000 08:01 1226563                    /lib/x86_64-linux-gnu/libnsl-2.19.so
7fe061df5000-7fe061df6000 rw-p 00015000 08:01 1226563                    /lib/x86_64-linux-gnu/libnsl-2.19.so
7fe061df6000-7fe061df8000 rw-p 00000000 00:00 0
7fe061df8000-7fe061dff000 r-xp 00000000 08:01 1226703                    /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7fe061dff000-7fe061ffe000 ---p 00007000 08:01 1226703                    /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7fe061ffe000-7fe061fff000 r--p 00006000 08:01 1226703                    /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7fe061fff000-7fe062000000 rw-p 00007000 08:01 1226703                    /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7fe062000000-7fe063000000 rw-p 00000000 00:00 0
7fe063001000-7fe063002000 r-xp 00000000 08:01 2395961                    /opt/wmf-mariadb101/lib/plugin/auth_socket.so
7fe063002000-7fe063201000 ---p 00001000 08:01 2395961                    /opt/wmf-mariadb101/lib/plugin/auth_socket.so
7fe063201000-7fe063202000 r--p 00000000 08:01 2395961                    /opt/wmf-mariadb101/lib/plugin/auth_socket.so
7fe063202000-7fe063203000 rw-p 00001000 08:01 2395961                    /opt/wmf-mariadb101/lib/plugin/auth_socket.so
7fe063203000-7fe063219000 r-xp 00000000 08:01 1226416                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fe063219000-7fe063418000 ---p 00016000 08:01 1226416                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fe063418000-7fe063419000 rw-p 00015000 08:01 1226416                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fe063419000-7fe0635ba000 r-xp 00000000 08:01 1226440                    /lib/x86_64-linux-gnu/libc-2.19.so
7fe0635ba000-7fe0637ba000 ---p 001a1000 08:01 1226440                    /lib/x86_64-linux-gnu/libc-2.19.so
7fe0637ba000-7fe0637be000 r--p 001a1000 08:01 1226440                    /lib/x86_64-linux-gnu/libc-2.19.so
7fe0637be000-7fe0637c0000 rw-p 001a5000 08:01 1226440                    /lib/x86_64-linux-gnu/libc-2.19.so
7fe0637c0000-7fe0637c4000 rw-p 00000000 00:00 0
7fe0637c4000-7fe0638c4000 r-xp 00000000 08:01 1226482                    /lib/x86_64-linux-gnu/libm-2.19.so
7fe0638c4000-7fe063ac3000 ---p 00100000 08:01 1226482                    /lib/x86_64-linux-gnu/libm-2.19.so
7fe063ac3000-7fe063ac4000 r--p 000ff000 08:01 1226482                    /lib/x86_64-linux-gnu/libm-2.19.so
7fe063ac4000-7fe063ac5000 rw-p 00100000 08:01 1226482                    /lib/x86_64-linux-gnu/libm-2.19.so
7fe063ac5000-7fe063bb1000 r-xp 00000000 08:01 1508076                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.20
7fe063bb1000-7fe063db1000 ---p 000ec000 08:01 1508076                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.20
7fe063db1000-7fe063db9000 r--p 000ec000 08:01 1508076                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.20
7fe063db9000-7fe063dbb000 rw-p 000f4000 08:01 1508076                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.20
7fe063dbb000-7fe063dd0000 rw-p 00000000 00:00 0
7fe063dd0000-7fe063dd3000 r-xp 00000000 08:01 1226475                    /lib/x86_64-linux-gnu/libdl-2.19.so
7fe063dd3000-7fe063fd2000 ---p 00003000 08:01 1226475                    /lib/x86_64-linux-gnu/libdl-2.19.so
7fe063fd2000-7fe063fd3000 r--p 00002000 08:01 1226475                    /lib/x86_64-linux-gnu/libdl-2.19.so
7fe063fd3000-7fe063fd4000 rw-p 00003000 08:01 1226475                    /lib/x86_64-linux-gnu/libdl-2.19.so
7fe063fd4000-7fe06420a000 r-xp 00000000 08:01 1506105                    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fe06420a000-7fe06440a000 ---p 00236000 08:01 1506105                    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fe06440a000-7fe064428000 r--p 00236000 08:01 1506105                    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fe064428000-7fe064438000 rw-p 00254000 08:01 1506105                    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fe064438000-7fe06443c000 rw-p 00000000 00:00 0
7fe06443c000-7fe06449d000 r-xp 00000000 08:01 1506103                    /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
7fe06449d000-7fe06469c000 ---p 00061000 08:01 1506103                    /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
7fe06469c000-7fe0646a0000 r--p 00260000 08:01 1506103                    /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
7fe0646a0000-7fe0646a7000 rw-p 00264000 08:01 1506103                    /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
7fe0646a7000-7fe0646af000 r-xp 00000000 08:01 1226471                    /lib/x86_64-linux-gnu/libcrypt-2.19.so
7fe0646af000-7fe0648ae000 ---p 00008000 08:01 1226471                    /lib/x86_64-linux-gnu/libcrypt-2.19.so
7fe0648ae000-7fe0648af000 r--p 00007000 08:01 1226471                    /lib/x86_64-linux-gnu/libcrypt-2.19.so
7fe0648af000-7fe0648b0000 rw-p 00008000 08:01 1226471                    /lib/x86_64-linux-gnu/libcrypt-2.19.so
7fe0648b0000-7fe0648de000 rw-p 00000000 00:00 0
7fe0648de000-7fe0648df000 r-xp 00000000 08:01 1226921                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
7fe0648df000-7fe064ade000 ---p 00001000 08:01 1226921                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
7fe064ade000-7fe064adf000 r--p 00000000 08:01 1226921                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
7fe064adf000-7fe064ae0000 rw-p 00001000 08:01 1226921                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
7fe064ae0000-7fe064aeb000 r-xp 00000000 08:01 1505885                    /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
7fe064aeb000-7fe064cea000 ---p 0000b000 08:01 1505885                    /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
7fe064cea000-7fe064ceb000 r--p 0000a000 08:01 1505885                    /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
7fe064ceb000-7fe064cec000 rw-p 0000b000 08:01 1505885                    /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
7fe064cec000-7fe064d04000 r-xp 00000000 08:01 1226429                    /lib/x86_64-linux-gnu/libpthread-2.19.so
7fe064d04000-7fe064f03000 ---p 00018000 08:01 1226429                    /lib/x86_64-linux-gnu/libpthread-2.19.so
7fe064f03000-7fe064f04000 r--p 00017000 08:01 1226429                    /lib/x86_64-linux-gnu/libpthread-2.19.so
7fe064f04000-7fe064f05000 rw-p 00018000 08:01 1226429                    /lib/x86_64-linux-gnu/libpthread-2.19.so
7fe064f05000-7fe064f09000 rw-p 00000000 00:00 0
7fe064f09000-7fe064f29000 r-xp 00000000 08:01 1226431                    /lib/x86_64-linux-gnu/ld-2.19.so
7fe064f29000-7fe064f2a000 ---p 00000000 00:00 0
7fe064f2a000-7fe064f5a000 rw-p 00000000 00:00 0
7fe064f5a000-7fe064f5b000 ---p 00000000 00:00 0
7fe064f5b000-7fe064f8b000 rw-p 00000000 00:00 0
7fe064f8b000-7fe064f8c000 ---p 00000000 00:00 0
7fe064f8c000-7fe064fbc000 rw-p 00000000 00:00 0
7fe064fbc000-7fe064fbd000 ---p 00000000 00:00 0
7fe064fbd000-7fe064fed000 rw-p 00000000 00:00 0
7fe064fed000-7fe064fee000 ---p 00000000 00:00 0
7fe064fee000-7fe06501e000 rw-p 00000000 00:00 0
7fe06501e000-7fe06501f000 ---p 00000000 00:00 0
7fe06501f000-7fe06504f000 rw-p 00000000 00:00 0
7fe06504f000-7fe065050000 ---p 00000000 00:00 0
7fe065050000-7fe065080000 rw-p 00000000 00:00 0
7fe065080000-7fe065081000 ---p 00000000 00:00 0
7fe065081000-7fe0650b1000 rw-p 00000000 00:00 0
7fe0650b1000-7fe0650b2000 ---p 00000000 00:00 0
7fe0650b2000-7fe0650e5000 rw-p 00000000 00:00 0
7fe0650e5000-7fe0650e7000 rw-s 00000000 00:0c 20563040                   /[aio] (deleted)
7fe0650e7000-7fe0650ec000 rw-s 00000000 00:0c 20563039                   /[aio] (deleted)
7fe0650ec000-7fe0650f1000 rw-s 00000000 00:0c 20563038                   /[aio] (deleted)
7fe0650f1000-7fe0650f6000 rw-s 00000000 00:0c 20563037                   /[aio] (deleted)
7fe0650f6000-7fe0650fb000 rw-s 00000000 00:0c 20563036                   /[aio] (deleted)
7fe0650fb000-7fe065100000 rw-s 00000000 00:0c 20563035                   /[aio] (deleted)
7fe065100000-7fe065105000 rw-s 00000000 00:0c 20563034                   /[aio] (deleted)
7fe065105000-7fe06510a000 rw-s 00000000 00:0c 20563033                   /[aio] (deleted)
7fe06510a000-7fe06510f000 rw-s 00000000 00:0c 20563032                   /[aio] (deleted)
7fe06510f000-7fe065114000 rw-s 00000000 00:0c 20563031                   /[aio] (deleted)
7fe065114000-7fe065119000 rw-s 00000000 00:0c 20563030                   /[aio] (deleted)
7fe065119000-7fe065121000 rw-p 00000000 00:00 0
7fe065121000-7fe065123000 rw-s 00000000 00:0c 20563029                   /[aio] (deleted)
7fe065123000-7fe065129000 rw-p 00000000 00:00 0
7fe065129000-7fe06512a000 r--p 00020000 08:01 1226431                    /lib/x86_64-linux-gnu/ld-2.19.so
7fe06512a000-7fe06512b000 rw-p 00021000 08:01 1226431                    /lib/x86_64-linux-gnu/ld-2.19.so
7fe06512b000-7fe06512c000 rw-p 00000000 00:00 0
7ffeb2cb0000-7ffeb2cd1000 rw-p 00000000 00:00 0                          [stack]
7ffeb2def000-7ffeb2df1000 r--p 00000000 00:00 0                          [vvar]
7ffeb2df1000-7ffeb2df3000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Fatal signal 6 while backtracing
Aborted

I have been able to reproduce the issue on my local environment with compressed tables. It always fail.

Tried different things without any success:

  • Switch to Antelope before creating the empty schema without compression and compressing it after its creation and after discarding it.
  • Switch to Antelope and creating the empty schema with compression.
  • Creating the empty schema without compression and with Barracuda, switch to Antelope, compress it and discard it before importing it

We have to also keep in mind that this was the first time we try to move a compressed tablespace. We are moving it between major versions (10.0 and 10.1) and while this didn't give any issue without compression (although it is not ideal), with compression this different versions can totally be playing a role here.

db1095 has been restored from backups and I have started replication there and on labsdb hosts and db1070 let them catchup.
Probably I will use mydumper to import from db1070 to db1095.

Marostegui added a comment.EditedWed, May 31, 8:37 AM

So, given the issue with the compressed tables and if we still want to use db1070 - this is the procedure I have thought to get it on labs.
Most of the steps we'd need to do anyways when we move tablespaces so...there is not much difference here.

  • Stop slaves on db1095 and labsdb1009,10,11
  • Stop MySQL on db1095 and on labsdb1009 and take a backup
  • Stop replication on db1070, get the position and file, and mydumper the dewiki and wikidatawiki tables
  • Double check binlog is disabled on db1095
  • Import the tables on db1095
  • Run redact_sanitarium.sh on db1095 for dewiki and wikidatawiki
  • Run check_private_data.py to make sure everything was cleaned up
  • mydumper those tables and import them on labsdb1009,10 and 11
  • Stop MySQL on db1095
  • Configure replication on db1095 for the s5 channel
  • Start replication on s5 on db1095
  • Start replication on labsdb1009 and make sure it works fine, if it does for some minutes, start it on labsdb1010 and labsdb1011

Don't use mysqldump- mydumper will be faster, and works great. Just make sure you do not backup (or remove later) common dbs like mysql and others. You do not need to stop replication or restart the server during the process (although backups won't hurt)- mydumper by default disables the binlog on the sessions for the restore to be faster -although check the options.

Mentioned in SAL (#wikimedia-operations) [2017-06-05T05:58:55Z] <marostegui> Stop MySQL on db1095 to take a backup - T153743

Mentioned in SAL (#wikimedia-operations) [2017-06-05T07:43:39Z] <marostegui> Stop labsdb1011 to take a backup - T153743

Mentioned in SAL (#wikimedia-operations) [2017-06-05T09:06:40Z] <marostegui> Stop replication on db1070 for maintenance - T153743

I have reloaded dbproxy1011 configuration, heads up in case the wrong db is being pointed at (downtime expired).

Maybe it didn't expire but got lost, but same thing applies.

Mentioned in SAL (#wikimedia-operations) [2017-06-07T08:02:19Z] <marostegui> Run redact_sanitarium on db1095 for dewiki - T153743

The import finished during the weekend on the labs hosts and I have configured replication on db1095 for s5, and it is now flowing. I started replication on the labs hosts during the weekend and they received a second sanitization (coming from db1095).
I am running another check data on them before the views are created to make sure it is all correct.

Marostegui updated the task description. (Show Details)Mon, Jun 12, 8:41 AM

Change 358324 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s5.hosts: Add new labs infra

https://gerrit.wikimedia.org/r/358324

Change 358324 merged by jenkins-bot:
[operations/software@master] s5.hosts: Add new labs infra hosts

https://gerrit.wikimedia.org/r/358324

bd808 added a subscriber: bd808.Tue, Jun 13, 7:29 PM

cloud-services-team feel free to create the views for dewiki and wikidatawiki on the labsdb1009, 1010 and 1011
Thanks!

Yay ! New wikis !
Thanks @Marostegui and teams :)

Mentioned in SAL (#wikimedia-operations) [2017-06-19T14:08:20Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1070 - T153743 (duration: 00m 41s)