Hm. I'll feel pretty embarrassed if this pause is normal and has nothing to do with my addon since it is only a second at the most. Maybe I just haven't noticed it till now. I swapped spec probably 40 times yesterday testing this thing.
I'll report back.
EDIT: No it's definitely the addon because it doesn't happen after the first respec when I delete my saved variables.
EDIT2: I put time prints at the beginning and end of the function and with all my addons enabled and with just my addon enabled I at most get one second differences. half of the time it doesn't even register a difference. I guess that means it's nothing to worry about.