[zork]

I spent some time now testing. Doing dungeon in the dungeonfinder. I have to push another fix my first post was OK, but incomplete. Swapping between questionmark and model bugs so I had to fix it.

This works for me currently in any situation (going from mark to model or from model to mark or model to model)

local Update = function(self, event, unit)
    if(not unit or not UnitIsUnit(self.unit, unit)) then return end
 
    local portrait = self.Portrait
    if(portrait.PreUpdate) then portrait:PreUpdate(unit) end
 
    if(portrait:IsObjectType'Model') then
        local guid = UnitGUID(unit)
        if(not UnitExists(unit) or not UnitIsConnected(unit) or not UnitIsVisible(unit)) then
            portrait:ClearModel()
            portrait:SetModel('interface\\buttons\\talktomequestionmark.m2')
            portrait:SetCamDistanceScale(0.25)
            portrait:SetPortraitZoom(0)
            portrait:SetPosition(0,0,0.5)           
            portrait.guid = nil
        elseif(portrait.guid ~= guid or event == 'UNIT_MODEL_CHANGED') then
            portrait:ClearModel()
            portrait:SetUnit(unit)
            portrait:SetCamDistanceScale(1)
            portrait:SetPortraitZoom(1)
            portrait:SetPosition(0,0,0)
            portrait.guid = guid
        end
    else
        SetPortraitTexture(portrait, unit)
    end

The else condition can be removed entirely.
What was important is to reset the GUID on questionmark (because it would not change back if you had the GUI, got mark later. It would stuck in mark)
To make the model values apply the model must be cleared. Doesn't work otherwise.