Page MenuHomePhabricator
Create Task
Maniphest T366357

cloud-vps puppetservers filling up / with puppetserver reports
Closed, ResolvedPublic
Actions

Description

When looking at a failed fullstack VM, I see this error in the logs:

Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: No space left on device - /var/lib/puppetserver/server_data/facts/fullstackd-20240530223742.admin-monitoring.eqiad1.wikimedia.cloud.json

Indeed, / is full on the puppetserver, due to a huge number of files in /var/lib/puppetserver/reports.

We need to either stop generating those reports entirely, or purge them periodically to keep our puppetservers from degrading.

Details

SubjectRepoBranchLines +/-
puppetdb: Remove 'none' from puppet reports config when adding 'puppetdb'operations/puppetproduction+7 -1
puppetserver 'report' enum: allow 'none' as a valueoperations/puppetproduction+1 -1
cloud-vps: turn off puppet report storage on cloud-vps puppetserversoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects

Event Timeline

Andrew created this task.May 31 2024, 2:36 PM
Restricted Application added a project: cloud-services-team. · View Herald TranscriptMay 31 2024, 2:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
gerritbot added a comment.May 31 2024, 3:24 PM

Change #1037812 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloud-vps: turn off report storage on cloud-vps puppetservers

https://gerrit.wikimedia.org/r/1037812

gerritbot added a project: Patch-For-Review.May 31 2024, 3:24 PM
• OKJ04 added a subtask: T366433: CentralAuth tests broken unless you run them inside Quibble.Jun 3 2024, 1:39 AM
JJMC89 removed a subtask: T366433: CentralAuth tests broken unless you run them inside Quibble.Jun 3 2024, 1:44 AM
dcaro merged a task: T366406: [cloudvps] 2024-05-01 cloudinfra puppetserver got out of space .Jun 3 2024, 2:20 PM
dcaro subscribed.
gerritbot added a comment.Jun 3 2024, 4:00 PM

Change #1037812 merged by Andrew Bogott:

[operations/puppet@production] cloud-vps: turn off puppet report storage on cloud-vps puppetservers

https://gerrit.wikimedia.org/r/1037812

gerritbot added a comment.Jun 3 2024, 4:07 PM

Change #1038381 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] puppetserver 'report' enum: allow 'none' as a value

https://gerrit.wikimedia.org/r/1038381

gerritbot added a comment.Jun 3 2024, 4:13 PM

Change #1038381 merged by Andrew Bogott:

[operations/puppet@production] puppetserver 'report' enum: allow 'none' as a value

https://gerrit.wikimedia.org/r/1038381

gerritbot added a comment.Jun 3 2024, 4:23 PM

Change #1038387 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] puppetdb: Remove 'none' from puppet reports config when adding 'puppetdb'

https://gerrit.wikimedia.org/r/1038387

gerritbot added a comment.Jun 3 2024, 4:40 PM

Change #1038387 merged by Andrew Bogott:

[operations/puppet@production] puppetdb: Remove 'none' from puppet reports config when adding 'puppetdb'

https://gerrit.wikimedia.org/r/1038387

Andrew closed this task as Resolved.Jun 3 2024, 4:55 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 3 2024, 5:30 PM
taavi subscribed.Jun 4 2024, 10:47 AM

@Andrew Doesn't the PCC updater rely on the facts report directory? Also there should be a systemd timer cleaning up old reports to free up space.

dcaro added a comment.Jun 4 2024, 10:53 AM
In T366357#9858847, @taavi wrote:

@Andrew Doesn't the PCC updater rely on the facts report directory? Also there should be a systemd timer cleaning up old reports to free up space.

The merged duplicate task was changing that timer to run every hour instead of the default 8h (in case the reports are still needed).

taavi added a comment.Jun 4 2024, 10:54 AM

And https://gerrit.wikimedia.org/r/c/operations/puppet/+/1037812 was merged yesterday disabling the reports entirely?

dcaro added a comment.Jun 4 2024, 10:57 AM
In T366357#9858898, @taavi wrote:

And https://gerrit.wikimedia.org/r/c/operations/puppet/+/1037812 was merged yesterday disabling the reports entirely?

Yes, as we were not aware they are needed for pcc (I wasn't at least).

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits