[Duugu]

Ok, this is finaly working. If someone has ever to add a flyout functionality to an action button without using type="action" ... this is how it could be done:

Query the players spellbook for all flyout 'spells' if the addon loads (code from Phanx above this post) and store the flyoutid and all the spells for each flyoutid.
Create your secure action button frames.
Create a set of secure action buttons for the Pop up flyout bar (like 10 should be enough).
Use SetFrameRef to add to each of your standard action buttons a reference to all of the 10 Pop up flyout buttons. Like this

for x = 1, 10 do
        myBtn:SetFrameRef("PopupFlyoutButton"..x, _G[("PopupFlyoutButton"..x])
    end

Add an attribute to all of your action buttons for each of the "flyout sets" you've queried in step one. Like this as an example for flyoutid 11 that contains the spells 1456, 1457 and 1458:

myBtn:SetAttribute("FlyoutSetInfo11", "1456,1457,1458")

Handle the type "flyout" in the "ReceivingDrag" wrap script for your action buttons as usual and set up a valid texture, etc for your action button. Set the type to "myflyout" and "spell" to the flyoutID (is provided with value or extra value ...).

Now use everything within the "OnClick" wrapscript of your action buttons, and set the flyout pop up buttons. Like this:

if self:GetAttribute("type") == "myflyout" then
    local x = 1 
    for k, _ in string.gmatch(self:GetAttribute("FlyoutSetInfo"..self:GetAttribute("spell")), "(%d+),") do
        local tFlyoutBtn = self:GetFrameRef("PopupFlyoutButton"..x)
        tFlyoutBtn:SetAttribute("type", "spell")
        tFlyoutBtn:SetAttribute("spell", k)
        tFlyoutBtn:RunAttribute("UpdateState", self:GetAttribute("state"))
        --<add your code to update the visual stuff of the flyout pop up button (texture, etc).
        --set the flyout pop up buttons position
        tFlyoutBtn:ClearAllPoints()
        tFlyoutBtn:SetPoint("CENTER", tFlyoutBtn:GetParent(), "CENTER", 0, 50 * x)
        x = x + 1
    end
end

I don't know if this is the only way. Or if it is unreasonably complicated. But this is the only way I could figure out to to this. If you have any improvements or whatever I would love to hear it.

[e]
There's still one point where this flyout is different from the standard flyout: it can be dragged from your button in combat, but it's not possible to set the dragged button to the cursor (the secure codes pickup don't handle "flyout" actions and doing it myself with PickupSpellBookItem would cause taint again). :/