Issue with iterating through values with metatables

[Mayron]

Hi,

I don't know exactly how the pairs, ipairs, and next iterator functions in Lua work, only what they do but still I thought they have to access index's of a table but they seem to have strange results on metatable's which is very annoying to work with:

-- test:
local realTable = {
    "hello", 
    "something",
    aValue = true,
}
 
local someTable = setmetatable({}, {__index = realTable})
for key, value in pairs(someTable) do
    print(key) -- never gets printed
end

or:

-- test:
local realTable = {
    "hello", 
    "something",
    aValue = true,
}
 
local sometable = setmetatable({}, {__index = realTable})
for id, value in ipairs(sometable) do
    print(value) -- never gets printed
end
 
print(sometable[1]) -- prints "hello" as expected

These test examples show my point. Both ipairs and pairs, no matter what is in the "realTable" (index/value pairs or key/value pairs) will never access the meta method __index to go to the realTable. I can sort of understand why pairs will not work but ipairs tries to access the first index [1] of someTable which does not exist so shouldn't it attempt to call the __index meta method?
Also if I replace pairs/ipairs with the "next" function I get an "attempt to call a nil value" Lua error instead which seems even more strange to me.

I understand that pairs does not directly look for "aValue" in the someTable but is it possible to achieve the result of looking inside the realTable instead using these iterator functions as a way around this?

Thanks for reading


EDIT: I have read the documentation from lua.org about how ipairs works and still cannot understand why the __index method is not triggered:

Originally Posted by http://www.lua.org/pil/7.3.html function iter (a, i)     i = i + 1     local v = a[i] -- here someTable (a) is being directly indexed by i so it's meta method __index should be called but it doesn't!     if v then         return i, v     end end   function ipairs (a) -- In my case, a would be someTable     return iter, a, 0 end When Lua calls ipairs(a) in a for loop, it gets three values: the iter function as the iterator, a as the invariant state, and zero as the initial value for the control variable. Then, Lua calls iter(a, 0), which results in 1, a[1] (unless a[1] is already nil). In the second iteration, it calls iter(a, 1), which results in 2, a[2], and so on, until the first nil element.

Shouldn't line "local v = a[i]" call this method? Because this should equivalate to "local v = someTable[i]" which is an attempt to access an index which does not exist in someTable which should mean it calls __index to check the realTable instead. But it doesn't do this!

_

[jeffy162]

I don't know, but, this stuff makes my head hurt.

[Resike]

I'm just not sure what are you trying to accomplish, for me it's seems like you are creating an empty metatable.

If you change your code like this, then it should work:

local realTable = {
    "hello",
    "something",
}
 
local sometable = setmetatable(realTable, {__newindex = function(t, index, value)
    rawset(t, index, value)
end})
 
for id, value in ipairs(sometable) do
    print(id, value)
end

[Mayron]

Originally Posted by Resike I'm just not sure what are you trying to accomplish, for me it's seems like you are creating an empty metatable. If you change your code like this, then it should work: local realTable = {     "hello",     "something", }   local sometable = setmetatable(realTable, {__newindex = function(t, index, value)     rawset(t, index, value) end})   for id, value in ipairs(sometable) do     print(id, value) end

It's hard explaining without showing the real code but doesn't the __index method only trigger if it cannot find the value in the normal table and then checks another table (the parent one like inheritance)?

Because basically I want the __index function to ALWAYS trigger so to do this I use __newindex to add entries to the parent table rather than the normal table so every time a value is indexed using the normal table, it has to use the __index function which returns the value from the parent table, as well as perform other stuff.

Here's an abbreviated example of the real code I am working on:

local parent[mode] = {}
-- do stuff
modules[mod] = setmetatable({}, {
    __index = function(_, key)              
        if (type(parent[mod][key]) == "function") then                  
            -- do stuff
        end
        return parent[mod][key];
    end,
    __newindex = function(_, key, value)
        if (type(value) == "function") then
            -- do stuff
        else
            parent[mod][key] = value;
        end
    end,
})

[Resike]

Originally Posted by Mayron It's hard explaining without showing the real code but doesn't the __index method only trigger if it cannot find the value in the normal table and then checks another table (the parent one like inheritance)? Because basically I want the __index function to ALWAYS trigger so to do this I use __newindex to add entries to the parent table rather than the normal table so every time a value is indexed using the normal table, it has to use the __index function which returns the value from the parent table, as well as perform other stuff. Here's an abbreviated example of the real code I am working on: local parent[mode] = {} -- do stuff modules[mod] = setmetatable({}, {     __index = function(_, key)                      if (type(parent[mod][key]) == "function") then                              -- do stuff         end         return parent[mod][key];     end,     __newindex = function(_, key, value)         if (type(value) == "function") then             -- do stuff         else             parent[mod][key] = value;         end     end, })

I think this is what you need then:

http://www.lua.org/pil/13.4.4.html

[Seerah]

Your sometable table doesn't have anything in it, that's why your pairs() loop doesn't see anything. If you were actually trying to index something (doing sometable[key] or sometable.key) and the key does not exist in your table, *that's* when it will look it up in the __index in the metatable to get a value.

In my experience, metatables work better with key=value pairs, rather than an "array"-type table (with numerical indices). Though I don't know how you're constructing your table... Here's where it can go wrong:

defaults = {"a", "b", "c"}
mytable = {}
setmetatable(mytable, {__index = defaults})
for i = 1, 3 do
     print(mytable[i])
end
--prints out:   a   b   c
 
for k, v in pairs(mytable) do
     print(v)
end
--prints out:       (because it's empty - we were just looking stuff up in the __index table)

Okay, so then if you do this...

table.insert(mytable, "d")
for i = 1, 4 do
     print(mytable[i])
end
-- prints out:   d   b   c   nil
 
for k, v in pairs(mytable) do
     print(v)
end
--prints out:   d   (because we inserted something into mytable)

Where did "a" go?

[Mayron]

Originally Posted by Resike I think this is what you need then: http://www.lua.org/pil/13.4.4.html

Thanks, that pretty much sums up exactly what I'm trying to do but unfortunately it says:

(Notice that, unfortunately, this scheme does not allow us to traverse tables. The pairs function will operate on the proxy, not on the original table.)

and doesn't solve this problem which is the same problem I'm having so I assume it's impossible to fix this. As I previously mentioned in the original post, I don't understand how the ipairs function does not trigger the __index method but it seems it doesn't so I will have to live with this

[Mayron]

Originally Posted by Seerah Your sometable table doesn't have anything in it, that's why your pairs() loop doesn't see anything. If you were actually trying to index something (doing sometable[key] or sometable.key) and the key does not exist in your table, *that's* when it will look it up in the __index in the metatable to get a value.

That's where I got confused as according to the code from lua.org, it shows that the ipairs function (not entirely sure about the pairs function) actually does try to index sometable[n] where n is an index number so I thought that would trigger __Index if index n doesn't exist in the table with nothing in it, but strangely it doesn't.

But I do understand how the __Index method works and how it looks at the original table before looking at the "inherited" one I just wish that for loop iterator functions would trigger the __index method when it tries to index something that does not exist in the original table but from reading the link Resike sent I just have to assume its impossible.

[Resike]

Originally Posted by Mayron That's where I got confused as according to the code from lua.org, it shows that the ipairs function (not entirely sure about the pairs function) actually does try to index sometable[n] where n is an index number so I thought that would trigger __Index if index n doesn't exist in the table with nothing in it, but strangely it doesn't. But I do understand how the __Index method works and how it looks at the original table before looking at the "inherited" one I just wish that for loop iterator functions would trigger the __index method when it tries to index something that does not exist in the original table but from reading the link Resike sent I just have to assume its impossible.

How about if you create your own pairs function?

[Mayron]

Originally Posted by Resike How about if you create your own pairs function?

I would have done this but I thought that might make my AddOn very confusing for others to use in case they wanted to build onto it but that's unlikely. I've made it possible for others to use my API library functions just in case but its inside this metatable madness lol

But thanks, I may just do that

I'm basically rewriting my entire UI from scratch with a better framework that supports functions to be used for my UI in a much nicer, organised structure. But I thought it would be nice for me and others if pairs and ipairs could traverse all the different functions etc.. that are stored in my library/API tables. I've documented it all nicely so the only way to notice that they actually exist is to read the .lua files which I guess will have to do

But making my own pairs function will help me program much easier so that's one problem solved!

[SDPhantom]

pairs() and ipairs() functions don't perform any Lua indexing operations. They're C-side code that iterate through the indexed and hash arrays that make up any table within Lua's C code. If there's nothing in those C arrays, nothing happens. Metatables are something else entirely and don't touch the C arrays of the tables they're assigned to.

In essence, if you make a blank table and iterate though it, you get nothing. It doesn't matter if there's a metatable assigned or not.



If you really need to iterate though the metatable, you can do so, but it's considered its own table.

for k,v in pairs(getmetatable(t).__index) do
-- Do stuff
end

It'll be better if you had access to the table you assigned to __index and iterate through that directly.



Though, if your intent were to expose the metatable functions as an API, it's best to make the functions as API first, then link them into a metatable rather than the other way around.

function DoSomething(t,val)
-- Do stuff
end

-- Link function to the metatable of NewTable
local NewTable=setmetatable({},{__index={DoSomething=DoSomething}});

-- Now these two calls will be equivalent
DoSomething(NewTable,"SomeValue");
NewTable:DoSomething("SomeValue");

[Mayron]

Thanks a lot! That Lua.org page just confused me then showing the pairs and ipairs functions as Lua code. I may try the approach you recommended. It sounds much easier