Page MenuHomePhabricator

puppetize and configure fruec banner log parser/db injector script
Closed, ResolvedPublic

Description

https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/fundraising/FRUEC/
(branch should be 'deployment')

https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/fundraising/FRUEC/+/master/config_example.yaml
(decide whether to puppetize this or put it in localsettings)

Also see T196564 for database setup.

Event Timeline

Jgreen created this task.Aug 26 2019, 7:50 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 26 2019, 7:50 PM
Jgreen updated the task description. (Show Details)Aug 26 2019, 7:54 PM
  • frdeploy configuration is puppetized and tested, needs to be adjusted to 'deployment' branch once that branch exists
  • /etc/fruec.yaml is puppet-template-deployed
  • python-mysql.connector is puppet-installed
  • python-yaml is puppet-installed

@AndyRussG checking in about these configuration settings:

central_notice_directory: ''
landing_page_directory: ''

The way this works in production is that incoming logs are written to /srv/archive/banner_logs/{year}/* to keep the total file count sane in each directory. Does the script know to look in the ./{year} subdirectory the way the old one does? Otherwise I should be able to make puppet automatically adjust that in the template based on the current system date, but that could result in the configuration changing at an awkward time on New Years Eve relative to when the script runs.

fruec database is created per sql/create_tables.sql

The db portion of /etc/fruec.yaml is:

db_settings:

user: 'fr_stats'
password: '(nope!)'
host: 'fundraisingdb-write.wmnet'
database: 'fruec'

Grants for the fruec database are the same as for the pgehres database.

frdeploy configuration is puppetized and tested, needs to be adjusted to 'deployment' branch once that branch exists

Cool! K, I created that branch! :)

Does the script know to look in the ./{year} subdirectory the way the old one does?

Yep! It will traverse the directory tree, so files can be in the directory indicated or any subdirectories.

You can get more info about the command-line options by running the command (bin/fruec) with the --help option, or you can look at the help text directly in the code here.

Thanks much!!!

frdeploy configuration is puppetized and tested, needs to be adjusted to 'deployment' branch once that branch exists

Cool! K, I created that branch! :)

@AndyRussG Ok I adjusted the deploy config. I think we should be ready to go to start testing and look at setting up the process-control job. Can you ping me when you have time to collaborate on that?

Jgreen closed this task as Resolved.Sep 4 2019, 4:42 PM
Jgreen moved this task from In Progress to Done on the fundraising-tech-ops board.

Hi! We were able to deploy this fine! :) (Thanks much to @Ejegg for the help with that.)

python-mysql.connector is puppet-installed

So we tried to run FRUEC manually on civi1001, but had issues with python libraries. Here is the PYTHONPATH we tried:

PYTHONPATH=/srv/fruec/:/usr/lib/python3

Does that setting seem right? With that setting, python could find the fruec and yaml modules. However, it couldn't find the mysql module.

I do see pymysql in /usr/lib/python3/dist-packages/, so maybe the wrong package was installed? This is the one we need, as per setup.py.

Thanks much!!