Core supports the term_access query tag to limit access to taxonomy terms, similar to node_access, just without the grants system behind it.
However, core doesn't use it nor does it provide any tests for it. So I'm quite sure that it's broken right now, let's add some tests to prove that ;)