This happened again over the weekend. I've restarted it.

And again today.

Another restart on 2020-08-03: https://sal.toolforge.org/log/n4IAtXMBj_Bg1xd3rkvT

Restarted again: https://sal.toolforge.org/log/aexZA3QBj_Bg1xd3wLvx (I'll reference this task next time, now that I know it exists.)

Adding Restart=always to the systemd unit would also mitigate having to wait for someone to manually restart it.

There appears to be an unused configuration file which includes this here:

https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/puppetdb/templates/puppetdb.service.erb

It doesn't seem to be referenced where the service is defined:

https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/puppetdb/manifests/app.pp

Mentioned in SAL (#wikimedia-releng) [2020-08-25T18:03:37Z] <dpifke> Restarted puppetdb on deployment-puppetdb03 (T248041)

Mentioned in SAL (#wikimedia-releng) [2020-08-27T23:19:47Z] <dpifke> Restarted puppetdb on deployment-puppetdb03 (T248041)

Mentioned in SAL (#wikimedia-operations) [2020-09-08T13:25:49Z] <mateusbs17> Restarted puppetdb on deployment-puppetdb03 (T248041)

Mentioned in SAL (#wikimedia-releng) [2020-09-09T08:43:28Z] <hashar> Restarted puppetdb on deployment-puppetdb03 (T248041)

The instance only has 2GB RAM. Maybe the instance flavor can just be changed to get more RAM and then restarted, else we would need to rebuild it from scratch :\

Change 626109 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] deployment-prep: reduce puppetdb memory usage

https://gerrit.wikimedia.org/r/626109

Change 626109 abandoned by Hashar:
[operations/puppet@production] deployment-prep: reduce puppetdb memory usage

Reason:
And looking at the instance it has:

/etc/postgresql/11/main/tuning.conf:shared_buffers = 600MB

Which is set in Horizon https://horizon.wikimedia.org/project/instances/05f311e9-1ef4-4acd-b467-adb59f6c2f93/

profile::puppetdb::database::shared_buffers: 600MB

https://gerrit.wikimedia.org/r/626109

The postgresql tunings are:

maintenance_work_mem = 1GB
checkpoint_completion_target = 0.9
effective_cache_size = 8GB
work_mem = 192MB
wal_buffers = 8MB
shared_buffers = 600MB
max_connections = 120

Mentioned in SAL (#wikimedia-releng) [2020-09-09T09:03:48Z] <hashar> deployment-puppetdb03 set profile::puppetdb::jvm_opts: -Xmx256m via Horizon and restarted Puppetdb # T248041

The java process is now running with -Xmx256m, was -Xmx4G, that should help

In T248041#6445873, @hashar wrote:

The java process is now running with -Xmx256m, was -Xmx4G, that should help

If this tuning by @hashar works, we should probably consider changing the defaults for Cloud VPS deploys via ops/puppet.git:hieradata/cloud.yaml.

Based on dmesg the oom hasn't happened since beginning of October so that is an improvement. Some request to postgre required more memory than available apparently which triggered the oom killer.

The deployment-puppetdb03 instance has just 2 GB of memory, I guess we can get it resized to a slightly larger flavor? Do note there is also a deployment-puppetdb04 instance but it comes with 4GB of memory. I have no idea what those instances are for though.

Mentioned in SAL (#wikimedia-releng) [2020-12-14T08:34:12Z] <hashar> deployment-prep restart puppetdb process on deployment-puppetdb03 # T248041

Mentioned in SAL (#wikimedia-releng) [2020-12-17T18:32:43Z] <dpifke> Restart puppetdb on deployment-puppetdb03 (T248041)

The deployment-puppetdb03 instance has just 2 GB of memory, I guess we can get it resized to a slightly larger flavor?

Requested via T270420 since that is apparently possible :]