Since i'm pretty sure PlayerModel derived objects can't be masked, i'm assuming you're working with non-animated 3D portraits, which iirc only come from SetPortraitTextureFromCreatureDisplayID, which will discard any set masks and use a standard circle, and also comes with an undefined background tint.
so
Lua Code:
texture:SetMask('Interface/ARCHEOLOGY/Arch-Keystone-Mask');
SetPortraitTextureFromCreatureDisplayID(texture, 83136);
would still result in a circled texture, most likely with a non-black background.
It's quite probable that the hexagonal border in that example is just covering the circle edge.