Tl;dr. FIXED! This indeed was an out-of-disk space issue.

The trick part was that humaniki's main disk-space was not full the network-file system, where we store some temporary files was. I have implemented a fix to do delete those temporary files after 90 days.

In fact the main humaniki database still has 500GB remaining, which at the current rate should last at least another year.

I cleaned up the strange ½ fill that occurred when the disk ran out of space on 2022-02-28. I confirmed that the filling process can run again on 2022-02-28 and 2022-03-07. I believe, as humaniki runs once per day, we should slowly come back up to do within the week.

Need post-push code review /audit if there is a volunteer. Commits:
https://github.com/notconfusing/humaniki-schema/commit/7a023a0e3c5df8b9dbbb5a34731806449e54cd21
https://github.com/notconfusing/humaniki-schema/commit/ac490c5d52a07b7bd12e936597d9527e07c18e60

Remaining to-dos. (logged in phabricator)

1. #TODO Catch the WDTK error java.io.IOException: No space left on device and handle it in the humanik-schema orchestration layer, through emailing.
2. #TODO start thinning out the database to save on space in the future. I believe it would be a good retention to policy to, for instance, on keep the 1st fill of the month, for fills older than 1 year. And maybe 1 fill per quarter for fills older than 2 years. That way we may be able to make the remaining 500GB last us quite some time.

Folowing message from notconfusing, I have tried using Humaniki and I still got the message API reachable but : ["Query returned 0 results"] so the problem does not look solved yet. I reopen the task.