Problem
_form_builder_handle_input_element() retrieves element values from the top of the form array hierarchy to the bottom, as it is called by form_builder(), which moves through the hierarchy in that direction because it needs to.
The problem with value retrieval is that a parent's value is retrieved before that if its children, which means that if the children have value callbacks, the parent receives the raw values of its children, which in turn causes its own value to be incorrect.
Proposed solution
Create a form_builder_bottom_to_top() that is called after form_builder() and builds all elements similarly to form_builder(), but from the bottom of the hierarchy up, and only for operations that need to be performed in that order. This will also ensure value retrieval happens after element processing to make sure all child elements exist.