Hi, in my experience with masking in wow it doesn't..... some code I just tested using a mask created in gimp that turns any texture in to a circle works fine.
Code:
local tex = Frame:CreateTexture(nil, "ARTWORK", nil, 0);
tex:SetTexture(some texture);
tex:SetAllPoints(Frame);
tex:SetMask(Some Path to CircularMask.tga")