Currently, the TempStore implementation resides in the user module. I can see why the factory was placed there (it assumes that there is a global user as it tries to fall back to the global user id if no owner has been definide explicitly). However, in the long run, I think it makes sense to move it to Drupal\Core.
At some point we can probably even inject our refactored session into the factory and retrieve the user id from that, completely untying it from the user module.