nginx accidentaly mounted on tmpfs in WMCS
Closed, InvalidPublicBUG REPORT
Actions

Assigned To

None

Authored By

	jhathaway
	Sep 26 2023, 9:18 PM

Description

A puppet patch I added accidentally broke puppet compiles on WMCS, https://gerrit.wikimedia.org/r/c/operations/puppet/+/959226, unfortunately the subsequent patch to fix the issue, https://gerrit.wikimedia.org/r/c/operations/puppet/+/960708, flipped the state in wmcs from false to true, causing /var/lib/nginx to be mounted on tmpfs. This may cause issues, so we should consider reverting the change, possibly with these steps:

Change profile::nginx::lib_on_tmpfs: true to false in cloud.yaml
Apply puppet
Stop nginx
Umount /var/lib/nginx
Start nginx

Related Objects
Search...

Status	Subtype	Assigned	Task
Invalid	BUG REPORT	None	T347432 nginx /var/lib/nginx accidentaly mounted on tmpfs in WMCS
Resolved		fnegri	T347428 cumin and cloud-vps instances not working
Resolved		fnegri	T340241 cloudcumin is slow when targeting many cloud VMs
Stalled	BUG REPORT	fnegri	T346453 [cumin] [openstack] Openstack backend fails when project is not set

Event Timeline

jhathaway created this task.Sep 26 2023, 9:18 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 26 2023, 9:18 PM

jhathaway added a subtask: T347428: cumin and cloud-vps instances not working.Sep 26 2023, 9:18 PM

jhathaway added a project: Cloud-Services.

jhathaway updated the task description. (Show Details)

jhathaway added subscribers: Andrew, taavi.

The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!

JJMC89 edited projects, added Cloud-VPS; removed Cloud-Services.Sep 26 2023, 9:21 PM

After some discussion we (mostly @taavi ) think that it's probably OK to leave this set to 'true'. It would be good to do a cloud-wide restart of nginx servers to make sure that the new setting is working properly, which we can do if/when cumin starts working again.

fnegri changed the status of subtask T347428: cumin and cloud-vps instances not working from Open to In Progress.Sep 27 2023, 10:05 AM

fnegri changed the status of subtask T347428: cumin and cloud-vps instances not working from In Progress to Stalled.Sep 28 2023, 1:18 PM

fnegri closed subtask T347428: cumin and cloud-vps instances not working as Resolved.Apr 11 2024, 10:37 AM

@Andrew do you still want to do a restart of nginx servers? Cumin is now working fine (though only thanks to your manually-applied patch, upstream Cumin is still broken).

I think this is moot at this point.

In T347432#9706520, @taavi wrote:

I think this is moot at this point.

yep, agreed

nginx /var/lib/nginx accidentaly mounted on tmpfs in WMCSClosed, InvalidPublicBUG REPORTActions

Description

Related ObjectsSearch...

Event Timeline

nginx /var/lib/nginx accidentaly mounted on tmpfs in WMCS
Closed, InvalidPublicBUG REPORT
Actions

Related Objects
Search...