_filter_autop() breaks links around the <drupal-media> tag

Problem/Motivation

The Drupal link CKEditor button can be used to add links to embedded media items. However, we can't guarantee that the rendered representation of that media item will be a single HTML element, or one that can be safely wrapped with <a> tags. As a result, editors can create links that have arbitrary HTML elements inside them.

Steps to reproduce:

1) Go to simplytest.me and spin a vanilla install of Drupal 8.8.x. Log in with admin/admin, install the "Media" and "Media Library" modules.
2) Go to /admin/config/content/formats/manage/basic_html?destination=/admin/config/content/formats and drag the "Insert from Media Library" button into the CKEditor toolbar. Enable the "Embed media" filter, and click "Save configuration".
3) Go to /admin/structure/media/manage/image/display and drag the "Authored by" field up into the visible fields area. Click "Save" to save this view display settings.
4) Go to /node/add/article to open the article node form. In the wysiwyg, click the "Insert from Media Library" button. Upload a file into the upload zone, give it an ALT text. Save and insert. At this point you will have something like this:

5) Click the media item to select it, then click the "Link" button from the CKEditor toolbar. Give it a random URL (such as https://drupal.org)
6) At this point, if you view the source markup you will have something similar to this:

<p>text before media</p>
<a href="https://drupal.org">
<drupal-media data-align="center" data-entity-type="media" data-entity-uuid="94a3ff34-a958-427c-ac51-127259cbc1e2"></drupal-media>
</a>

<p>text after media</p>

7) Fill in the required fields and save the node.
8) Inspect the markup of the node body, you will find something like this:

Proposed resolution

TBD

Remaining tasks

- Discuss and agree on a reasonable approach
- Implement it
- Review & commit

User interface changes

TBD

API changes

TBD

Data model changes

TBD

Release notes snippet

TBD

Original report from @wrd :

With the progress being made on #2801307: [META] Support WYSIWYG embedding of media entities, I've been playing with the new capabilities and have run into what seems like an important issue for typical users: after embedding some media entities, particularly images, a user will probably expect to be able to link the media to another page. At present, this doesn't really work.

1. Perform a Standard install.
2. Enable the Media and Media Library modules.
3. Edit the Basic HTML editor. Enable the "Embed media" filter, remove the "Image" button, add the "Insert from Media Library" button. <drupal-media data-entity-type data-entity-uuid> is added to the allowed HTML tags automatically.
4. Create a new Image media entity in the Media Library.
5. Create a new Basic Page.
6. Use the "Insert from Media Library" button to embed an image in the body field.
   
   
7. Click on the embedded image to select it.
8. Click the Link button to link the image to a destination. I'll link it to https://drupal.org.
   
   
9. When done, note that there's no visual indication that the embedded image has been linked to anything.
   
   
10. Save and publish the node, and view it.

You'll see that the image is, indeed, linked to https://drupal.org. However, if you inspect the element, the <a> tag has been wrapped around the entire media entity:

<article data-history-node-id="1" role="article" about="/node/1" typeof="schema:WebPage" class="node node--type-page node--view-mode-full clearfix">
  <header>
    <span property="schema:name" content="Test Page 1" class="rdf-meta hidden"></span>
  </header>
  <div class="node__content clearfix">
    <div property="schema:text" class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item">
      <p>This is a test. </p>
      <a href="https://drupal.org">
        <article class="media media--type-image media--view-mode-full">
          <div class="field field--name-field-media-image field--type-image field--label-visually_hidden">
            <div class="field__label visually-hidden">
              Image
            </div>
            <div class="field__item">
              <a href="https://drupal88.dd:8443/sites/default/files/2019-08/pears_1.jpg">
                <img src="/sites/default/files/2019-08/pears_1.jpg" width="640" height="480" alt="Pears" typeof="foaf:Image" />
              </a>
            </div>
          </div>
        </article>
      </a>
      <p>This is only a test. </p>
    </div>  
  </div>
</article>

While we can wrap block elements in <a> tags in HTML5, the result here is problematic. The image itself is (by default) linked to the raw image file, so we've got nested hyperlinks, which AFAIK isn't permitted in HTML5. And browsers (Safari, anyway) will ultimately render this as multiple separate <a> tags, most of them empty or wrapping whitespace.

It seems like the expected default behavior for a typical user would be that after using the Link button, there would be some kind of visual indication in CKEditor that the entity is linked, and the rendered page would simply have the image linked to the destination specified using the Link button.

However, to complicate matters, we have to anticipate the possibility that the media entity being embedded is more complex than just an image. For example, what if I have a custom media type that looks like the Image type, but has a few more fields:

• Copyright (date field)
• Location (geolocation field)
• Photographer (entity reference field)

...and a default view mode that displays the image, the copyright date, the location, and a link to the Photographer entity.

If I embed this entity, and then use the link button, the desired behavior is probably for the image to link to the URL entered using the link button, while leaving the link to the Photographer node untouched so that it still works.

I'm not sure if this more complex case can be handled in any reasonably generalized fashion that'll work for most users. However, the more basic behavior of simply linking an embedded image entity should probably work out of the box, shouldn't it?

I hope this is useful! Let me know if I can clarify anything.