IsInGroup works and the only reason why it's only working for you on a reload is you need to register the event PLAYER_ENTERING_WORLD / GROUP_ROSTER_UPDATE and once either of those are fired then you check to see if you're in a group.
Code:
local smallcircle5 = CreateFrame("Frame", "SmallCircleTopCenter")
smallcircle5:SetFrameLevel(self.Health:GetFrameLevel() + 5)
local smallcircle5texture = smallcircle5:CreateTexture(nil, "ARTWORK")
smallcircle5texture:SetTexture(m.textures.smallcircle)
smallcircle5texture:SetPoint('CENTER', self, 46, 35)
smallcircle5texture:SetSize(26, 26)
smallcircle5texture:SetTexCoord(0, 1, 0, 1)
smallcircle5:SetAlpha(0)
smallcircle5:RegisterEvent("PLAYER_ENTERING_WORLD")
smallcircle5:RegisterEvent("GROUP_ROSTER_UPDATE")
smallcircle5:SetScript("OnEvent", function(self, event, ...)
if IsInGroup() then
self:SetAlpha(1)
else
self:SetAlpha(0)
end
end)