Ah. Ok. Now I got it.
In fact I
do rotate the frame itself (the Default Blizzard ActionButton frames to be exact) via animation groups. It must be the frame that is rotated. The texture is not enough. It's an action button and it clickable. Rotating the texture only would create a really bad user experience with clicking the buttons.
But, as I sayed, the animation feature has bugs. Some child objects are not rotated or not as expected (framestring), some are rotated to somewhere (everthing that is not anchored to the center of the parent object), and so on. It's a great mess.
Additionally to rotating the button frame I would like to change the shape of the (already rotated) button textures. That's what I'm using SetTexCoord for and what's not working as intended.
I'm almost sure there are no other animation types than rotate, scale and linear transform. Or did I missed something? Is there an animation type for custom tranformations?