So its impossible to tell if a program is evil or not, definitively (Assuming you have some technical definition of malicious, then I believe this is implied by Rice's Theorem. If you don't have a rigours definition of evil, then its doubly impossible as you're going to need strong AI just to match the squishy human definition of what is evil).
We could certainly use heuristics. Static analysis is basically (non-statistical) AI applied to this field, but there's some catches:
- We could ban things that fail the test. In theory it is even possible to make a perfect test that has no false negatives, so we'd be safe (If someone gives a firm definition of what behaviours are evil). But then the test will have lots of false positives, and possibly totally ban wanted behaviours
- We could have a test that has less false positives, at the cost of some false negatives. This might even be useful against certain classes of attackers. But most of these things don't assume a malicious attacker, and I'm not sure how useful it would be against a sophisticated attacker. Particularly if the source code (and training data if applicable) were publicly available. But even without that, I'm sure a smart person could bypass such a system. And after all we aren't worried about the 80% bottom percent of attackers. We are worried about the one who gets in, as it only takes one.
- We could mark things for review if they fail the test. But what is the review system? Who is the reviewers? These are already hard political & technical questions that need to be solved first. However, they are probably of independent interest.
Good notes, @Bawolff
But what is the review system?
Special:RecentChanges filtered to JS pages
Who is the reviewers?
I think that setting up an expectation that anyone with those with interface-editor do some patrolling is reasonable. We might even want to add all MW namespace JS pages to their watchlist by default.
This would require some program management work and consultation, for sure. But the motivation is there. People are certainly worried about the privacy and security of their wikis. In the meantime, if we can build an effective model, then pitching a smaller reviewing workload is much easier. In fact, I expect this workload to be so small that I'd like to see a notification of "a JS edit needs review" to pull people in.
I'm not quite sure how to manage this across wikis. We really need a generalized cross-wiki notification/watchlist system to build on top of.