Today, @RKemper noticed that all new WDQS Bullseye hosts have Icinga alerts for load-categories-daily.service and load-dcatap-weekly.service .
Creating this ticket to investigate why these services are failing and fix the problem.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T291916 Tracking task for Bullseye migrations in production | |||
| Resolved | Gehel | T323921 [Epic] Migrate all Search Platform servers to Debian Bullseye | |||
| Resolved | bking | T332314 Service implementation for wdqs20[13-22] | |||
| Resolved | bking | T331300 Ensure WDQS stack works on Bullseye | |||
| Resolved | Gehel | T342060 Investigate WDQS categories update failures on Bullseye hosts | |||
| Resolved | dancy | T342162 "scap deploy"'s config-deploy should check for broken symlinks |
Event Timeline
Comment Actions
Investigation so far:
load-categories-daily.service calls
/usr/local/bin/loadCategoriesDaily.sh wdqs
/usr/local/bin/loadCategoriesDaily.sh calls /usr/local/bin/cronUtils.sh which sources variables from either /etc/wdqs/gui_vars.sh or /etc/wdqs/vars.sh.
/etc/wdqs/vars.sh is a broken symlink to ../../srv/deployment/wdqs/wdqs-cache/revs/${HASH}/.git/config-files/etc/query_service/vars.sh
/etc/wdqs/ldf-config.json is also a broken symlink. It doesn't look scap is uploading them during the deployment process. Will investigate further tomorrow.
Comment Actions
Per Tuesday's pairing session, if we use the --force flag when we scap deploy , we can fix the issues with incomplete deployment and finally get these things production-ready.