There are failures we expect (data inconsistent with what we consider valid data) and data we check against/filter out. This has come up a few times as we work with real files and test data. In general, if it seems like a value could come through that would be valid but that we were conservative against (eg. a new value in the enumeration), we should log that so a human can check the logs and make a decision about it. If we're stripping out false/null values, that's intentional (because we care about positive results not negation results) and therefore there's no need to log that.
For instance:
- The test data didn't return an enumeration we expected and it correctly threw an error but we shouldn't do anything about it (because it's test-generated data)
- The real data didn't return an enumeration we expected and it correctly threw an error but we should log it here so a human can verify that this is or is not a valid value we should update the code for
Please audit the node scripts that import data (import-data.js) and ensure that our logging of unexpected data is consistent:
- Whenever we do a validity check (eg. against an enumeration), log the value if the data exists but doesn't match the enum
- If the validity check is against empty values, there's no need to log that