Using the layout builder to customize the layout of an individual node, trying to select a new block to include is quite slow. This appears to be due entirely to filtering the available blocks by context. I originally discovered this issue in Panels, and Layout Builder utilizes the same methods in the Block Manager service.
For example, after installing the umami demo profile, enabling the layout builder module, and configuring the page content type to be individually customizable. Clicking the link to add a block to a page can take 1-3 seconds to return and open the off-canvas dialog. The blocks are retrieved in the page callback ChooseBlockController::build() by calling BlockManager::getFilteredDefinitions; from 161 possible blocks, 58 are returned after filtering. The blocks not returned include a block for every field on every bundle of every entity type, so a complex site will take increasingly longer (e.g. node types, taxonomy vocabularies, media types, etc).
When profiling, the majority of the execution time (over 90%) appears to be spent in \Drupal\Core\TypedData\Validation\RecursiveContextualValidator::ValidateNode()
Relevant methods in the stack trace:
\Drupal\layout_builder\Controller\ChooseBlockController::build()
\Drupal\Core\Block\BlockManager::getFilteredDefinitions()
\Drupal\Core\Block\BlockManager::getDefinitionsForContexts()
\Drupal\Core\TypedData\Validation\RecursiveContextualValidator::ValidateNode()