I cannot reproduce your issue.
Attached is a test script.
Your code is very weird btw, why are you getting and setting groups collection? It is passed by reference so you can just have something like $role->addGroup() and $role->removeGroup() and encapsulate the logic?
Also your tests are pretty useless, you check if two variables which are the same reference to the same collection are the same. Which should always be true.
Can you provide more details? I cant verify this without more details.