Hm. Interesting.
I tried something similar a while back. Made a texture with ragged edges like torn paper, set it as a mask, then lowered the alpha of the Minimap so it would look really Diablo'ish. I ran into a problem when being indoors, because it turns out the Minimap itself doesn't respond to alpha changes indoors, anything less than 1 is treated as 0!
BUT... that was the Minimap, I don't think I tried to simply adjust the transparency of the entire mask texture. That might very well be the workaround I was looking for. Because your screenshots are taken indoors, aren't they?