Are you sure that you have *enabled* this comparator? For performance reasons, it is not available by default, and you have to set something like

$smwgQComparators = '<|>|!|~';

in your LocalSettings.php. In any case, there is no error message, since the ~ would be considered as a part of the string you are querying for. I have checked the ~ comparator on my sites, and it worked without problems.