Problem/Motivation
If the user enters the quick-edit mode by clicking on "Quick edit" in the contextual links of a block (either in normal mode or editing mode), but then doesn't do any changes and just clicks on "Editing" at the top left corner to go back to the site. Then the pen icons stay visible.
Clicking on "Edit" again, makes the borders around blocks appear, but the pen icons disappear.
The only way to return to a state without pen icons is to click on any "Quick edit" contextual link and then close the black box by clicking on the X at the top right hand corner, or randomly saving a block.
The expected behaviour might be that users will only open a block if they are sure that they want to edit it, but users who are trying to find a configuration that is not available in the settings tray, or newer users who are looking around end up with a confusing state of appearing and disappearing pen icons that they don't seem to be able to get rid of.
Proposed resolution
The pen icons need to disappear when the quick editing mode is left, independent on whether the user saved something or not.