Nothing will happen.
From QC point of view the records will not be messed-up (no referential integrity in this respect).
From process point of view they might be orphaned, so it's worth checking if there are any items that need to be re-assigned to existing users (by whomever, whenever).
It's a little similar to changing projects lists - project entities with assigned values don't change, and it's only if re-validation is triggered that any issues will pop up.
If you delete a Site User, they'll also be removed from any projects they were assigned to, however, any entities assigned to the user will still be assigned to them - if any automail rules are triggered QC will try to email that user and generate an error in the SA or QC log file (I forget which one).
So other than errors in the log, you won't see anything and nothing will 'break'.