It seems that we want to solve two problems - one being classifying good faith vs bad faith, another being the scoring itself which indicates the degree of it

• In this case, a simple algorithm like logreg would be a good staring point, as the algorithm will set weights to each variable; the algorithm will also output a interpreted probability as the likelihood of a given record is being either 1 (good faith) or 0 (bad faith)
• The scoring can then be taken from the likelihood directly

As for validation, improving accuracy part,

• We could start with, as @kostajh suggested, look into these attributes in a sample of accounts with the model outputted weights, and compare with our intuition from past experience, and make a call on if such algorithm provides intuitive explanations.
• Check the scoring distribution from the algorithm - a good sign would be having scores clustered in a few spots, instead of being all over the place
• Then we could try to set thresholds for, good, bad, neutral, etc... and evaluate accuracy from here.

Some further thoughts:
Do we care more about scoring being shown to the user eventually? Or more about correctly classified binary outcomes - good vs bad faith?

• If we care more about scoring and let moderators set thresholds to look at (e.g. user-defined top X), then our problem statement would need to be closer to ranking than accuracy of the classifier.
• In this case, we would need to consider the correctness of ranking - which is not easy to set up ground truth data collection, and potentially a much more complex problem
• Then, the metric of the algorithm would change to MAP, DCG, NDCG, Spearman's rho, etc…
• We would need then to really understand what types of data we can collect and infer

If we care more about good vs bad faith in a binary sense, then having a straightforward binary classification task should suffice.