It's currently possible to search incategory:X or -incategory:X, where X is a category name, to include or exclude results categorised in Category:X. However, this does not cover subcategories. This would obviously be useful, particularly on Commons where search is an important way to find media.
This could be a separate keyword, say incategorysub:X. We could also, in addition, extend incategory to specify recursion levels, with incategory:X defaulting to incategory:0:X (just the category specified - no recursion), and incategory:5:X meaning that we want to traverse 5 levels deep, and perhaps incategory:-1:X meaning the same as incategorysub:X.
Obviously, the usefulness of this feature would be enhanced by also addressing Bug 21988 (basically, providing GUI access to advanced search features).