Method resetSize() in dialog.position.js is responsible to reposition and resize jQuery UI dialog on dialog content loading and later on screen resize events, etc.
The problem is that some dialog implementations (like entity_browser) are only setting maxHeight parameter and not the height parameter, and in that combination resetSize() method with fail to set maxHeight on dialog because of missing height parameter.
If we take a look at jQuery UI dialog source _size() method on line 792 we will see that if height option is not set to auto then min/maxHeight options will be ignored.