Page MenuHomePhabricator
Create Task
Maniphest T207706

LibreNMS upgrade to 1.51
Closed, ResolvedPublic
Actions

Description

I tried upgrading LibreNMS to 1.44 for T207481 but noticed after deploying that they started to use Composer to LibreNMS dependencies.
No mention of this major change in the changelog.
I rolled back and everything is working again.

Not sure yet on the next step forward to be able to upgrade LibreNMS? Is it possible to manually download and freeze the requirements like we do with Python apps?

See https://github.com/librenms/librenms/blob/master/composer.json

Edit: Upstream is now at 1.49, so we're getting quite behind.
Upgrade is needed for bug fixes (eg. we don't have email alerting anymore), security fix, and new features.

Details

Show related patches Customize query in gerrit

Event Timeline

ayounsi triaged this task as Medium priority.Oct 22 2018, 9:26 PM
ayounsi created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 22 2018, 9:26 PM
Dzahn added a project: SRE.Oct 22 2018, 9:27 PM
ayounsi added subscribers: Volans, Joe.Oct 22 2018, 11:43 PM
MoritzMuehlenhoff added a subscriber: MoritzMuehlenhoff.Dec 10 2018, 5:35 PM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:33 PM
ayounsi renamed this task from LibreNMS upgrade to 1.44 to LibreNMS upgrade to 1.49.Mar 12 2019, 5:21 PM
ayounsi updated the task description. (Show Details)
Stashbot added a comment.Apr 30 2019, 4:36 PM

Mentioned in SAL (#wikimedia-operations) [2019-04-30T16:36:43Z] <ayounsi@deploy1001> Started deploy [librenms/librenms@2094575]: Upgrade LibreNMS to 1.51 - T207706

Stashbot added a comment.Apr 30 2019, 4:36 PM

Mentioned in SAL (#wikimedia-operations) [2019-04-30T16:36:52Z] <ayounsi@deploy1001> Finished deploy [librenms/librenms@2094575]: Upgrade LibreNMS to 1.51 - T207706 (duration: 00m 11s)

gerritbot added a comment.Apr 30 2019, 5:54 PM

Change 507391 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Add PHP 7.2 to LibreNMS

https://gerrit.wikimedia.org/r/507391

gerritbot added a project: Patch-For-Review.Apr 30 2019, 5:54 PM
ayounsi added a comment.Apr 30 2019, 6:12 PM

Added doc on how to upgrade LibreNMS https://wikitech.wikimedia.org/wiki/LibreNMS#Upgrade_LibreNMS

ayounsi renamed this task from LibreNMS upgrade to 1.49 to LibreNMS upgrade to 1.51.Apr 30 2019, 6:12 PM
gerritbot added a comment.Apr 30 2019, 6:49 PM

Change 507391 merged by Dzahn:
[operations/puppet@production] Add PHP 7.2 to LibreNMS

https://gerrit.wikimedia.org/r/507391

gerritbot added a comment.Apr 30 2019, 7:15 PM

Change 507402 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] netmon: switch from PHP 7.0 to PHP 7.2 for LibreNMS upgrade

https://gerrit.wikimedia.org/r/507402

gerritbot added a comment.Apr 30 2019, 7:22 PM

Change 507402 merged by Dzahn:
[operations/puppet@production] netmon: switch from PHP 7.0 to PHP 7.2 for LibreNMS upgrade

https://gerrit.wikimedia.org/r/507402

gerritbot added a comment.Apr 30 2019, 8:49 PM

Change 507487 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] librenms: ensure php7.2-ldap is installed

https://gerrit.wikimedia.org/r/507487

gerritbot added a comment.Apr 30 2019, 8:51 PM

Change 507487 merged by Dzahn:
[operations/puppet@production] librenms: ensure php7.2-ldap is installed

https://gerrit.wikimedia.org/r/507487

Dzahn added a subscriber: Dzahn.EditedApr 30 2019, 9:24 PM

switching netmon from PHP 7.0 to PHP 7.2

  • first change installed the 7.2 packages but did not change which Apache module was loaded, so still used 7.0 while both packages were installed in parallel
  • added a phpinfo.php to confirm which version is actually used because both mod_php7.0 and mod_php7.2 were loaded at the same time which normally should not happen and isn't considered a sane config
  • second change switched the mod_php version but LibreNMS just silently died with a 500 on 7.2
  • stopped puppet and manually edited /etc/php/7.2/apache2/php.ini changed values for error_reporting and display_errors from Production value to Development value to get an actual error besides just the 500 apache logs.
  • logs now showed the issue was the PHP LDAP module was missing in 7.2 (/etc/php/7.2/mods-available vs. the same for 7.0). It got dropped in the first change.
  • third change added the php-ldap package which made LibreNMS work
  • reverted the manual change to php.ini; deleted the phpinfo.php file
  • removed with --purge all php7.0* packages; installed pending upgrades for php-common php-pear on both netmon2001 and netmon1002.. to remove all uncertainty about it
  • only netmon1002 is production .. done!
ayounsi added a comment.Apr 30 2019, 11:22 PM

I tried to deploy it once again.

1/ They replaced log_file with log_dir, this will need a puppet change
I temporarily worked around it but:

2/ App is not loading and this is showing up in the logs:

[2019-04-30 23:15:36] production.ERROR: Uncaught ReflectionException: Class files does not exist in /srv/deployment/librenms/librenms-cache/revs/2094575e580c045fffe603495d3c12952ad22aec/vendor/laravel/framework/src/Illuminate/Container/Container.php:779
Stack trace:
#0 /srv/deployment/librenms/librenms-cache/revs/2094575e580c045fffe603495d3c12952ad22aec/vendor/laravel/framework/src/Illuminate/Container/Container.php(779): ReflectionClass->__construct('files')
#1 /srv/deployment/librenms/librenms-cache/revs/2094575e580c045fffe603495d3c12952ad22aec/vendor/laravel/framework/src/Illuminate/Container/Container.php(658): Illuminate\Container\Container->build('files')
#2 /srv/deployment/librenms/librenms-cache/revs/2094575e580c045fffe603495d3c12952ad22aec/vendor/laravel/framework/src/Illuminate/Container/Container.php(609): Illuminate\Container\Container->resolve('files', Array)
#3 /srv/deployment/librenms/librenms-cache/revs/2094575e580c045fffe603495d3c12952ad22aec/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(759): Illuminate\Contai {"exception":"[object] (Symfony\\Component\\Debug\\Exception\\FatalErrorException(code: 1): Uncaught ReflectionException: Class files does not exist in /srv/deployment/librenms/librenms-cache/revs/2094575e580c045fffe603495d3c12952ad22aec/vendor/laravel/framework/src/Illuminate/Container/Container.php:779
Stack trace:
#0 /srv/deployment/librenms/librenms-cache/revs/2094575e580c045fffe603495d3c12952ad22aec/vendor/laravel/framework/src/Illuminate/Container/Container.php(779): ReflectionClass->__construct('files')
#1 /srv/deployment/librenms/librenms-cache/revs/2094575e580c045fffe603495d3c12952ad22aec/vendor/laravel/framework/src/Illuminate/Container/Container.php(658): Illuminate\\Container\\Container->build('files')
#2 /srv/deployment/librenms/librenms-cache/revs/2094575e580c045fffe603495d3c12952ad22aec/vendor/laravel/framework/src/Illuminate/Container/Container.php(609): Illuminate\\Container\\Container->resolve('files', Array)
#3 /srv/deployment/librenms/librenms-cache/revs/2094575e580c045fffe603495d3c12952ad22aec/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(759): Illuminate\\Contai at /srv/deployment/librenms/librenms-cache/revs/2094575e580c045fffe603495d3c12952ad22aec/vendor/laravel/framework/src/Illuminate/Container/Container.php:779)
[stacktrace]
#0 {main}
"}
gerritbot added a comment.Apr 30 2019, 11:43 PM

Change 507526 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] LibreNMS, add log dir

https://gerrit.wikimedia.org/r/507526

gerritbot added a comment.Apr 30 2019, 11:47 PM

Change 507526 merged by Ayounsi:
[operations/puppet@production] LibreNMS, add log dir

https://gerrit.wikimedia.org/r/507526

ayounsi added a comment.May 1 2019, 12:10 AM

CR above solved issue #1, but also needed:
sudo chown www-data:librenms /var/log/librenms.log

sudo chmod a+rw bootstrap/cache/ solved issue #2

Now getting the following:

Running the following commands will fix the issue most of the time:

sudo chown -R librenms:librenms /srv/deployment/librenms/librenms-cache/revs/2094575e580c045fffe603495d3c12952ad22aec

sudo setfacl -d -m g::rwx /srv/deployment/librenms/librenms-cache/revs/2094575e580c045fffe603495d3c12952ad22aec/bootstrap/cache /srv/deployment/librenms/librenms-cache/revs/2094575e580c045fffe603495d3c12952ad22aec/storage /var/log/librenms /srv/librenms/rrd

sudo chmod -R ug=rwX /srv/deployment/librenms/librenms-cache/revs/2094575e580c045fffe603495d3c12952ad22aec/bootstrap/cache /srv/deployment/librenms/librenms-cache/revs/2094575e580c045fffe603495d3c12952ad22aec/storage /var/log/librenms /srv/librenms/rrd

usermod -a -G librenms www-data

Which translate to the following when removing the scap cache:

sudo chown -R librenms:librenms /srv/deployment/librenms/librenms  (curently have deploy-librenms:deploy-librenms)

sudo setfacl -d -m g::rwx /srv/deployment/librenms/librenms/bootstrap/cache /srv/deployment/librenms/librenms/storage /var/log/librenms /srv/librenms/rrd

sudo chmod -R ug=rwX /srv/deployment/librenms/librenms/bootstrap/cache /srv/deployment/librenms/librenms/storage /var/log/librenms /srv/librenms/rrd

usermod -a -G librenms www-data

I'm not sure all of them do, if they're needed/sane and how to translate them to Puppet.

ayounsi added a comment.EditedMay 1 2019, 1:15 AM

I managed to get it working with some live hacking that need to be puppetized:
For the above error, the specific error was:

[Wed May 01 00:33:06.878169 2019] [php7:error] [pid 12465] [client 208.80.154.84:47290] PHP Fatal error:  Uncaught ErrorException: file_put_contents(/srv/deployment/librenms/librenms-cache/revs/2094575e580c045fffe603495d3c12952ad22aec/storage/framework/views/2adada3c0feaebaf338515d737e5fc57b56fddc9.php): failed to open stream: Permission denied in /srv/deployment/librenms/librenms-cache/revs/2094575e580c045fffe603495d3c12952ad22aec/vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php:122

Solved with a chmod -R a+rw storage
In our case we need at leat www-data to be able to write to that folder.

Then:

[2019-05-01 00:35:26] production.ERROR: No application encryption key has been specified. {"exception":"[object] (RuntimeException(code: 0): No application encryption key has been specified. at /srv/deployment/librenms/librenms-cache/revs/2094575e580c045fffe603495d3c12952ad22aec/vendor/laravel/framework/src/Illuminate/Encryption/EncryptionServiceProvider.php:44)

This is because running composer install also created a .env file that needs to reside at the root of librenms folder:

APP_KEY=<secret>

#DB_HOST=
#DB_DATABASE=
#DB_USERNAME=
#DB_PASSWORD=

#APP_URL=
NODE_ID=<is that secret?>  (EDIT: Not needed)

Last, LibreNMS still tries to write logs to /srv/deployment/librenms/librenms/logs/librenms.log so I changed its rights so it doesn't fail silently.

gerritbot added a comment.May 1 2019, 4:02 PM

Change 507605 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] LibreNMS, remove email_from from config.php

https://gerrit.wikimedia.org/r/507605

gerritbot added a comment.May 1 2019, 4:06 PM

Change 507605 merged by Ayounsi:
[operations/puppet@production] LibreNMS, remove email_from from config.php

https://gerrit.wikimedia.org/r/507605

gerritbot added a comment.May 1 2019, 4:38 PM

Change 507610 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] LibreNMS, run irc-bot as user deploy-librenms

https://gerrit.wikimedia.org/r/507610

gerritbot added a comment.May 1 2019, 4:48 PM

Change 507610 merged by Ayounsi:
[operations/puppet@production] LibreNMS, run irc-bot as user deploy-librenms

https://gerrit.wikimedia.org/r/507610

gerritbot added a comment.May 1 2019, 5:14 PM

Change 507616 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] LibreNMS, set IRC alerting to single line messages

https://gerrit.wikimedia.org/r/507616

gerritbot added a comment.May 1 2019, 5:15 PM

Change 507616 merged by Ayounsi:
[operations/puppet@production] LibreNMS, set IRC alerting to single line messages

https://gerrit.wikimedia.org/r/507616

gerritbot added a comment.May 1 2019, 6:31 PM

Change 507624 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] LibreNMS, alerting syntax change (REGEX instead of ~)

https://gerrit.wikimedia.org/r/507624

gerritbot added a comment.May 1 2019, 6:37 PM

Change 507624 merged by Ayounsi:
[operations/puppet@production] LibreNMS, alerting syntax change (REGEX instead of ~)

https://gerrit.wikimedia.org/r/507624

gerritbot added a comment.May 1 2019, 7:13 PM

Change 507629 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] LibreNMS, fix typo (REGEX -> REGEXP)

https://gerrit.wikimedia.org/r/507629

gerritbot added a comment.May 1 2019, 7:14 PM

Change 507629 merged by Ayounsi:
[operations/puppet@production] LibreNMS, fix typo (REGEX -> REGEXP)

https://gerrit.wikimedia.org/r/507629

gerritbot added a comment.May 1 2019, 10:40 PM

Change 507715 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[labs/private@master] Add librenms laravel_app_key fake public key

https://gerrit.wikimedia.org/r/507715

gerritbot added a comment.May 1 2019, 10:46 PM

Change 507716 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] LibreNMS, file files permission, add app key, add logrotate

https://gerrit.wikimedia.org/r/507716

gerritbot added a comment.May 2 2019, 11:39 PM

Change 507715 merged by Dzahn:
[labs/private@master] Add librenms laravel_app_key fake private key

https://gerrit.wikimedia.org/r/507715

gerritbot added a comment.May 3 2019, 12:34 AM

Change 507716 merged by Ayounsi:
[operations/puppet@production] LibreNMS, file files permission, add app key, add logrotate

https://gerrit.wikimedia.org/r/507716

ayounsi closed this task as Resolved.May 3 2019, 12:44 AM
ayounsi claimed this task.

Everything is done. Future upgrades should be much smoother.

gerritbot added a comment.May 3 2019, 4:25 AM

Change 507909 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] LibreNMS, change $install_dir to group librenms

https://gerrit.wikimedia.org/r/507909

gerritbot added a comment.May 3 2019, 7:22 PM

Change 507909 merged by Ayounsi:
[operations/puppet@production] LibreNMS, change $install_dir to group librenms

https://gerrit.wikimedia.org/r/507909

gerritbot added a comment.May 3 2019, 7:40 PM

Change 508014 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] LibreNMS, follow the symlink to $install_dir

https://gerrit.wikimedia.org/r/508014

gerritbot added a comment.May 3 2019, 7:44 PM

Change 508014 merged by Ayounsi:
[operations/puppet@production] LibreNMS, follow the symlink to $install_dir

https://gerrit.wikimedia.org/r/508014

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL