this turned out to be a group unitid propagation issue. took a little work to sort out the data lifecycle issues that were causing it, but things now seem to be working (in the sense that this option generally has the desired effect).
only one artifact remains, and it may be intractable, at least at my level of WoW/LUA development expertise: when Aloft disables a particular nameplate, it hides it and quits tracking it. when you invite someone into your group while their nameplate *should* be visible (would be visible if you enabled "all" friendly players), you have to go out of range (such that Blizzard recycles the graphics objects) and then return into range to get the nameplate to appear. i have some ideas i want to try, to work around this, but i am going to release with this artifact still active.
as for the rest of it: fiddling the option while affected nameplates are in range, having a visible char leave the group while the option is active, and etc... all of that now works. the nameplate is not permanently "lost" never to be seen again.