From dmesg:

[ 6872.646908] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 4
[ 6872.646911] {1}[Hardware Error]: It has been corrected by h/w and requires no further action
[ 6872.646913] {1}[Hardware Error]: event severity: corrected
[ 6872.646915] {1}[Hardware Error]:  Error 0, type: corrected
[ 6872.646917] {1}[Hardware Error]:  fru_text: B3
[ 6872.646918] {1}[Hardware Error]:   section_type: memory error
[ 6872.646920] {1}[Hardware Error]:   error_status: 0x0000000000000400
[ 6872.646921] {1}[Hardware Error]:   physical_address: 0x00000077b4ea7800
[ 6872.646925] {1}[Hardware Error]:   node: 1 card: 2 module: 0 rank: 0 bank: 2 row: 45906 column: 992
[ 6872.646927] {1}[Hardware Error]:   error_type: 2, single-bit ECC
[ 6872.646949] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 6872.646951] EDAC sbridge MC0: CPU 0: Machine Check Event: 0 Bank 255: 940000000000009f
[ 6872.646952] EDAC sbridge MC0: TSC eb28091adcc
[ 6872.646954] EDAC sbridge MC0: ADDR 77b4ea7800
[ 6872.646956] EDAC sbridge MC0: MISC 0
[ 6872.646958] EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1555188932 SOCKET 0 APIC 0
[ 6872.646982] EDAC MC1: 0 CE memory read error on CPU_SrcID#1_Ha#1_Chan#0_DIMM#0 (channel:4 slot:0 page:0x77b4ea7 offset:0x800 grain:32 syndrome:0x0 -  area:DRAM err_code:0000:009f socket:1 ha:1 channel_mask:1 rank:0)
[ 7164.888310] mce: [Hardware Error]: Machine check events logged

A more complete version of what I copied out of the console before I rebooted can be found at P8394

After forcing powercycle on the console, I have been evacuating things that are more important. Once the last three finish, it will be some k8s workers (one paws worker) and a bunch of toolsbeta stuff. I'll leave it at that for testing and such, but the tools-workers are depooled at this time (most are also still down after the reboot). The paws worker is off as well.

In T166949#3312721, @bd808 wrote:

Restarting the kubernetes webservice process seems to have fixed the app. It looks like the PHP get_current_user() function started returning an empty string. get_current_user() checks the ownership of the script, so maybe this was caused by some sort of NFS communication error.

Possibly an LDAP issue as well?

That said, I can get the nfs up soon within the project (maybe even next week--because I am off tomorrow).

The NFS bit is partly about moving to a new project (as it is staged now), which is blocked by a bug still from cert manager and a lot of puppetization. So if we complete the NFS part, it won't show up in PAWS for a while anyway. The NFS piece is the easy bit. At this time, I don't think there is any way I'll have time to do the rest of it by then. Anything can happen, but I don't think we should plan on that with a hard timeline.

Credentials created. Sorry for the wait!

Looking back through the log yet again, I think I've got this sorted out.

Yeah, mediaviews-api needs a venv rebuild, I'm fairly sure. This is working perfectly around that in some ways. What baffled me earlier was it complaining about being unable to restart things that are actually up and working. It might not be anything around webservice, but it seemed strange enough to dive down the rabbit hole.

Ok. I can now confirm after playing with it that is mostly working at this time. There is something weird about the behavior when it is trying to restart a webservice, though.

It appears to also completely fail to actually restart anything. Since it is failing, it may not actually be terribly bad that it's broken. Hard to tell. It is definitely growing log sizes for tools, though.

This is trying to restart things that aren't broken and just misbehaving. I'll try to sort it out.

Looking through, this is actually incorrect because of an unrelated error in tooling. My apologies.

@Zppix The project should be good to go in horizon now

All set from WMCS end. Confirmed I can run queries from Toolforge.

Removing some subscribers to not spam the world if not needed.

Just talked to @chasemp, who informed me that the original discussion included the idea of placing a floating IP on there that was iptables-restricted entirely to just tendril. We can certainly move to that, if that is the expectation and considered the best solution. That would mean we still don't have cloud clients connecting over that IP. I don't like it much because of the additional, not-so-well-monitored layer of depending on ferm, but I wanted to mention that here as an option that was previously discussed and can be done if it all comes down to that.

I confirmed with @ayounsi that it's blocked by network restrictions going into the cloud. The issue would be that return traffic from these instances to the tendril server would have to be allowed. This leads me to three questions for the DBAs:

1. Would we ever want to do that?
2. How bad is it to just not monitor it with tendril in the case of toolsdb?
3. Would it make sense to establish an instance of tendril (or similar) inside Cloud VPS for this and any future cases of virtualized DBs?

I'm fully expecting that the prometheus stuff is going to be on the labs side. That's mostly about making sure we set up the right dashboards for decent monitoring.

The other instance for this is 172.16.5.119, the secondary/slave.

I can confirm that the firewalling isn't at the instance/security group level.

Ping in general likely won't work into the cloud from anything outside it. However, I'm poking around at getting telnet 172.16.7.153 3306 working.

In T220530#5098513, @aborrero wrote:

Do you remind why we decided not to have floating IPs on those instances?

@aborrero mentioned https://gerrit.wikimedia.org/r/c/operations/puppet/+/499516 in meeting today, but I think (especially since that may not be complete), that should be added to in the course of this rather than being a blocker for getting this in shinken. Unmonitored DBs is very bad.

The full name is clouddb1001.clouddb-services.eqiad.wmflabs, but you almost certainly need to use 172.16.7.153 instead unless we can resolve the DNS outside cloud in the future.

That's part of the thing to figure out :)

Also: because of weaknesses in the old system, we probably will want to start with Luminous or Mimic--especially since no other releases are actually supported. Bluestore-as-default is one of the biggest benefits.

Some notes that also will apply to T90364:

• Ceph is a very high-latency system without lots of grooming and love when used as anything but a straight-up object store like Swift. With proper tuning it can be somewhat faster than NFSv4 in sync mode (which we use).
• 10G network on all nodes and clients should be viewed as a requirement except where we basically don't care at all about speed
• If OSD nodes are large with lots of disks, a failure and rebuild could collapse the system by overstraining the OSDs and creating instability. Smaller, more numerous nodes allow for more resiliency and higher availability and performance.
• The better the disk, the faster the processor needs to be...and single socket can outperform dual socket motherboards for ceph with the same processors--multiple cores good.
• Ceph doesn't do comprehensive testing and development on Debian, though they do package for it with a basic install and check test. They also recommend upgrading stock Debian kernels. They do full comprehensive support on CentOS and Ubuntu. No mimic packages are available for Debian until Buster because of needing gcc8.
• IO throughput requirement testing needs to be done so that we can tune things. I'm digging around in prometheus to find good metrics to watch and compare.
• Reviewing the network architecture around Ceph is recommended to avoid collapsing the system because of network changes and unrecommended config

Apr 5 19:12:11 labstore1005 nfs-exportd[7797]: exportfs: Failed to stat /exp/project/paws: No such file or directory

Thanks for the context, @Kolossos. You were the only easily-found person in phabricator who has access to the project.
Is the project effectively abandoned and in need of a maintainer? I see plenz is on there, but I am not sure who they are in Phabricator, if they are on here.

I know I'm very late to the game on this. I'm just curious. If we think what I'm suggesting is good, I can shuffle things.

Do we want this stuff under mariadb profile or under wmcs more like modules/profile/manifests/wmcs/services/toolsdb_primary.pp (honest question)? Like modules/profile/wmcs/cloudinfra/ or something?

osmdb is now on VMs.

@TheDJ and others, sorry the database crashed at one point because of permissions on a failover file. I'll see about fixing that in puppet and documenting a warning for future failovers. Some maps services may need a restart.

Now we wait 5min at least.

That tooks a bit because it required local rebase fussing to merge.

Starting on the DNS change

I'm leaving this more or less for @aborrero and @Andrew to edit :)

Sent email to the user registered at that address in LDAP about the use of the account to see if it is needed, etc.

@Halfak, I've set up the alias. So you should be able to connect to wikilabels.db.svc.eqiad.wmflabs instead (which is pointed at clouddb1002). This will hopefully make life a tad easier in the future.

Setting schedule for change on Thursday with announcement going out today, then.

In T167086#5073704, @Chicocvenancio wrote:

How would they be exposed exactly? The same as Toolforge, (two NFS shares mounted with one them pointed from a symbolic link)? IE, will the hack defined in T192214 still work?

Great! Thanks.

So I'm going to revert and then resubmit once tools-checker-01 and -02 are down

This should work now.
Before:

@Marostegui it was supposed to be down. It needed a kill -9.

Database services (postgres and mariadb) are now shut off, and the spare role is applied.

Good, I'll remove that...carefully.