Page MenuHomePhabricator

MySQL down on integration-slave-trusty-(1020|1021)
Closed, DuplicatePublic

Description

Steps to reproduce

  • Do recheck in comment.

Actual results

  • Then wait for the test to run and jenkins to report back.

Errror is on slave

slave ( integration-slave-trusty-1021 ) as well as date of event ( 12:16:09 UTC ) and the error "DB connection error: Access denied for user 'jenkins_u0'@'localhost' (using password: YES) (127.0.0.1:3306)"

  • It fails with this error

16:01:26 DB connection error: Access denied for user 'jenkins_u0'@'localhost' (using password: YES) (127.0.0.1:3306).
16:01:26 Check the host, username and password and try again.

https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm/2198/console

I'm not sure weather it happends on other slaves.

Expected results

  • MySQL should successfully install the db and there should not be db access issue.

Event Timeline

Paladox raised the priority of this task from to Needs Triage.
Paladox updated the task description. (Show Details)
Paladox added a subscriber: Paladox.
Paladox renamed this task from MySQL down on integration-slave-trusty-1020 to MySQL down on integration-slave-trusty-(1020|1021).Feb 11 2016, 4:46 PM

this blocks merging a new wikidata build (https://gerrit.wikimedia.org/r/#/c/269990/) which is needed for gate-and-submit (e.g. mediawiki-extensions-qunit) to work again for patches in core and some other extensions

aude triaged this task as High priority.Feb 11 2016, 5:04 PM

This doesn't seem to be a disk space issue:

integration-slave-trusty-1020:~$ df -h
Filesystem                          Size  Used Avail Use% Mounted on
udev                                997M   12K  997M   1% /dev
tmpfs                               201M  416K  200M   1% /run
/dev/vda1                            18G  9.3G  7.6G  56% /
none                                4.0K     0  4.0K   0% /sys/fs/cgroup
none                                5.0M     0  5.0M   0% /run/lock
none                               1001M     0 1001M   0% /run/shm
none                                100M     0  100M   0% /run/user
none                                256M   70M  187M  28% /var/lib/mysql
/dev/mapper/vd-second--local--disk   21G   12G  8.2G  59% /mnt
none                                128M     0  128M   0% /mnt/home/jenkins-deploy/tmpfs

The instances we added yesterday do not have enough memory. That is T126545. I am getting rid of them.

Dont quote me, but I think that whenever the servers is hitting memory limit, the tmpfs misbehave and the database hosted on it ends up disappearing. Weird.