I have been finding out many users who are having trouble with running #ajax with tableselect, and it seems the problem is specifically when one uses checkboxes.
When a tableselect is created with multiple = false, which means radios buttons on the left column, the #ajax attribute works fine. However with multiple = true, the #ajax attribute doesn't do anything.