Posting this here for now, will probably open a follow-up issue.
I've been working on updating token.module and adjusting functionality/tests for this. Which is probably something we should have done before committing this, to make sure that this works for more than the few use cases that core has.
I think there is at least one example there why supporting some sort of sanitize => FALSE is useful and important.
The basic use case is when you have user-provided, unsafe input and want it to be continue unsafe and un-escaped, because you then rely on autoescape.
One example in token.module is the block label, it has this code:
function token_block_view_alter(&$build, BlockPluginInterface $block) {
$config = $block->getConfiguration();
$label = $config['label'];
if ($label != '<none>') {
// The label is automatically escaped, avoid escaping it twice.
$build['#configuration']['label'] = \Drupal::token()->replace($label, array(), array('sanitize' => FALSE));
}
}
The problem is that now the block label tokens are escaped twice. There's a test that is creating a node with a ' in it, and right now, that is getting escaped twice (which is exactly what this code is testing), since we force-escape all token return values and then escape the whole string again.
I don't see a proper way to fix this right now. What technically works is using PlainTextOutput::renderFromHtml() but clearly it is not correct to use that in non-plaintext output.
We don't have to pass it to hook implementations, but I really think we need a flag to prevent auto-escaping. We even document:
The caller is responsible for choosing the right escaping / sanitization
but don't actually allow to caller to do that, at least not for token values. But if the token input is untrusted and will be escaped later, we must treat token replacements as untrusted too or we are guaranteed to have double-escaping problems?
(See also the following comments about some discussion and why various things don't work IMHO)
Add an option that does nothing but skip the Html::escape() and always returns the token value as-is.
It does *not* pass that through to hook implementations. To make it clear that it is not the same as the old sanitize option, it should probably be named differently, e.g. 'escape' => TRUE/FALSE, defaulting to FALSE.