tables, table.unpack and nil values

[Ketho]

• I'm confused why the first example does print the nil values, but the latter does not.
  Aren't these tables the "same"?

  1. local t = {nil, nil, 3, 7, nil, 4} 
      
     print(#t) -- 6
     print(unpack(t)) -- nil, nil, 3, 7, nil, 4

  2. local t = {}
     t[3] = 3
     t[4] = 7
     t[6] = 4
      
     print(#t) -- 0
     print(unpack(t)) -- nil

  I was always thinking that table.unpack can't iterate over "holes" in the same way as an ipairs-loop
  (I'm not a programmer, and I don't know what's the proper terminology)
  
  
• On another note, I'm also wondering about this (4 months old) post from Lur,
  concerning tables with a nil value at the end of a table

  Originally Posted by Lur Thanks a lot. But why local t1 = {1, 2, nil, "wwew", nil, 3} -- #t1 = 6 local t2 = {1, 2, nil, "wwew", 3, nil} -- #t2 = 2 What's wrong with all of that? Why tables with a nil as the last value are truncated to the first nil occurrence while with any non-nil end are counted correct?

[Foxlit]

unpack(t, i, j) returns t[i], t[i+1], ..., t[j]; if you do not specify i or j, they're defaulted to 1 and #t respectively -- so whether unpack will return values from an array with holes in it depends on the behavior of #t.

The Length Operator The length of a table t is defined to be any integer index n such that t[n] is not nil and t[n+1] is nil; moreover, if t[1] is nil, n can be zero. For a regular array, with non-nil values from 1 to a given n, its length is exactly that n, the index of its last value. If the array has "holes" (that is, nil values between other non-nil values), then #t can be any of the indices that directly precedes a nil value (that is, it may consider any such nil value as the end of the array).

If your array has holes in it, there are no guarantees as to which value #t (of those described in the quote above) will return, as illustrated by your examples. There's nothing wrong with the code you've provided as far as the language is concerned. You simply should not expect or rely on #t to return specific values for such arrays.


On a tangential note, ipairs explicitly iterates over t[1], t[2], t[3], ... stopping at the first nil value rather than using the length operator, so it'll never make it past the first nil in the array.

[SDPhantom]

There are quirks in how Lua stores tables internally. Every table is created with 2 data arrays, one for sequential data and another for associative data. Sequential data is stored when you assign a non-nil value to the next index in a table, this index is equal to #t+1. If you assign data to a key that is greater than the next index or is a non-number, the data is stored in the associative array. Both the length operator and ipairs() operate exclusively off the sequential array.

The table constructor { } breaks this rule in its own way in which undefined keys are written directly into the sequential array and may include embedded nils. The table constructor also trims any nils from the right side of the list.

There is another quirk in Lua that allows you to manually force nils to be embedded into the sequential array of a table. This is done by inserting two non-nil values then overwriting the first with nil.

[d87]

http://www.lua.org/source/5.2/ltable.c.html#luaH_getn

{x,y,z} seems to be an "array" notation, which makes sense to have (for example to store function arguments with holes in a table).
When you construct table that way, sizearray variable is filled without checking about holes. Later, when you request table length, it just checks if last element according to sizearray is nil, and if it is, tries to find boundary(non-nil value followed by nil value) manually.
Like in t2 example, it might not be the the boundary you want, so you better strip nils yourself in this case.

local t3 = {nil, nil, 4, 7, nil, 4}
t3[5]= 2   -- #t3 = 6
t3[7]= 2   -- #t3 = 0

Also, if you try to expand your array, sizearray is recalculated to first boundary it founds.

Edit: i'm late

[SDPhantom]

Originally Posted by d87 local t3 = {nil, nil, 4, 7, nil, 4} t3[5]= 2 -- #t3 = 6 t3[7]= 2 -- #t3 = 0 Also, if you try to expand your array, sizearray is recalculated to first boundary it founds.

I've tested this with the following function, sizearray doesn't ever seem to decrease like said.

function tAppend(tbl,...)
--  This should allow unpack() to work with embedded nils as if {...} was used
--  We need to keep the array allocation intact, Lua ignores setting the next index to nil
--  and creates gaps, pouring the next values into the non-array section of the table
    local tlen,previsnil=#tbl,false;
    for i=1,select("#",...) do
        local id,val=tlen+i,select(i,...);
        tbl[id]=val or false;-- Force to an actual value
        if previsnil then tbl[id-1]=nil; end
        previsnil=(val==nil);
    end
    if previsnil then tbl[#tbl]=nil; end
    return tbl;
end

Running the following code allows it to resize the table a few times to give it a chance to recalculate the sizearray value, but the embedded nils are still counting towards the size.

local t=tAppend({},1,nil,nil,4,nil,6,7,8);
print(#t);	-- Prints 8

To make the process better understood, this is the same as the following:

local t={};

t[1]=1;
t[2]=false;
t[3]=false;	t[2]=nil;
t[4]=4;		t[3]=nil;
t[5]=false;
t[6]=6;		t[5]=nil;
t[7]=7;
t[8]=8;

print(#t);	-- Prints 8

[d87]

local t=tAppend({},1,nil,nil,4,nil,6,7,8);
print(#t);	-- Prints 8

It's 1 on 5.1. But on 5.2 it is indeed 8...

[SDPhantom]

Originally Posted by d87 local t=tAppend({},1,nil,nil,4,nil,6,7,8); print(#t); -- Prints 8 It's 1 on 5.1. But on 5.2 it is indeed 8...

WoW has been running 5.1 since BC, they have not updated to 5.2 as of now.
All of my observations have been tested on the current implementation of Lua running on WoW.

[Ketho]

Well this answers a lot for me. Thank you Foxlit, SDPhantom and d87 for the, very clear explanation

Although I had some difficulty trying to understand about the length operator and the "embedded nils" in the table constructor, and the difference with ipairs

But especially Lur's example was one of the weirdest things about Lua that prompted me to ask this question

[Billtopia]

something on unpack that I didn't see above is for nils is in order for it to properly unpack the table all the values should be assigned at one time like the following

local temp = { nil, 1, nil, nil, nil, "blah", whatever }

if they are assigned separately it would not unpack right but as a single declaration the following would work fine

local var1, var2, var3, var4, var5, var6, var7 = unpack( temp )

[SDPhantom]

Originally Posted by Billtopia something on unpack that I didn't see above is for nils is in order for it to properly unpack the table all the values should be assigned at one time like the following local temp = { nil, 1, nil, nil, nil, "blah", whatever } if they are assigned separately it would not unpack right but as a single declaration the following would work fine local var1, var2, var3, var4, var5, var6, var7 = unpack( temp )

See my post above showing code for a custom function I made called tAppend().
The second and third paragraphs of my post preceding that also describes exactly what you said and a workaround for it.

Note calling tAppend(table,nil) will produce an undefined result, the list of values needs to end with a non-nil value.

[Foxlit]

Originally Posted by SDPhantom See my post above showing code for a custom function I made called tAppend().

The demo code for your function prints 1 for me on live realms, rather than the 8 you advertise. Perhaps trying to build functions around unspecified behavior is a bad idea after all?

[SDPhantom]

Originally Posted by Foxlit The demo code for your function prints 1 for me on live realms, rather than the 8 you advertise. ...

It's worked reliably on my system through several tests and retests across several sessions using the most current version of WoW. Since this is the second time this code has been questioned, I'll have to see if I can get it to glitch on another system.

Originally Posted by Foxlit ... Perhaps trying to build functions around unspecified behavior is a bad idea after all?

Discovering undocumented effects is a way to open a path to new features and ideas. For example, the fact that the metatables for UI objects are all shared among objects of the same type. Blizzard hasn't released any info that states this behavior, yet it has opened up the possibility to hook a single function to be triggered for all frames of that specific type.

[Billtopia]

try this to append items to your table so it will be "unpack"able

so put your table as 1st arg and values to add as 2nd, 3rd, ...

 tAppend = function( myTable, ... )
     myTable = { unpack( myTable ), ... }
 end

and this for the size if # don't work

 tSize = function( myTable )
    local temp = { unpack ( myTable ), "EndOfMyTable" }
    return #temp - 1
 end

or if that doesn't work then

 tSize = function( myTable )
    local temp = { unpack ( myTable ), "EndOfMyTable" }
    local x = 1
    while true do
         if type(temp[x]) == "string" and temp[x] == "EndOfMyTable" then
             return x - 1
         end
         x = x + 1
     end
 end

[SDPhantom]

The original idea of the function was to preserve the existing contents of the table and add new values to the end, hence the name tAppend(). Though it would be easy to wrap the table constructor in a function, there is nothing to gain from doing so. Any other possible ways to achieve the original design would risk a stack overflow and/or cause huge spikes in memory usage. No other method I could think of would only work on modifying the original table as intended.

[Billtopia]

for everything I have read on unpack it appears the method of recreating the table as I did above ( with or without the function wrapper ) is the only easy way to guarantee a proper unpack when you have nils within your table.

http://www.wowwiki.com/API_unpack gives a few ideas but nothing that would really allow you to add on easily to the table on the fly

[SDPhantom]

Originally Posted by Billtopia tAppend = function( myTable, ... ) myTable = { unpack( myTable ), ... } end

Reading this more, I discovered a couple fatal flaws. When you're referencing to unpack(myTable), since it isn't the last value in the list for the table constructor, it's only using the first value returned and discarding the rest of the table. Secondly, you're only reassigning the local variable of the function to the new pointer, this won't affect the table passed in any way.

[Billtopia]

I believe the thing to do then would be the following to get all of the values from unpack

myTable = { {unpack( myTable )}, ... }


and as for only changing the local pointer... I myself normally use one table with all my addon values within it so I can access it using something like:
myTable["Key"]
I pass whatever function needs it as the table and the key as a string

that allows me to access the table in functions and not a local copy... you could also do close to the same if the table is in the global namespace:
_G["MyVariable"]
just pass the table as a string

[SDPhantom]

Originally Posted by Billtopia myTable = { {unpack( myTable )}, ... }

This not only creates 2 new tables, but also changes the table structure.

Due to the design specs of the library my function was taken from, creating new tables is not allowed and completely contradicts the purpose of the library in the first place. The library I had this code in is a table recycler for use in code that calls frequently, perhaps multiple times each rendering pass. The idea behind it is to mimic the table constructor accurately on an existing table. Using an existing table means there's no new tables building up in memory waiting for a garbage collection pass. An ability of this library is that deallocated tables are stored as weak values, meaning when the garbage collector does run, they are released to it if they haven't been reallocated.

Note mimicking the table constructor accurately means including this undocumented effect.

[Phanx]

Originally Posted by Billtopia myTable = { {unpack( myTable )}, ... }

If your original table looks like this:

myTable = { "one", "two", "three" }

#myTable == 3

And you add the values "four" and "five" on the end:

tAppend(myTable, "four", "five")

Then your code is actually doing this:

myTable = { { unpack( { "one", "two", "three" } ) }, "four", "five" }

which simplifies to this:

myTable = { { "one", "two", "three" }, "four", "five" }

#myTable == 3

You've just created a new table that still only has 3 values, and another new table containing the original three values. Now imagine what happens after you run that a couple of times:

myTable = {
	{
		{
			{
				{
					{
						"one",
						"two",
						"three",
					},
					"four",
					"five",
				},
				"six",
			},
			"seven",
			"eight",
			"nine",
		},
		"ten",
		"eleven",
	},
	"twelve",
}

Is that really a table structure you'd want to try to work with?

Plus, if you were to do anything like this, you would never be able to use local references to your table. Add together the ridiculously convoluted structure you see above, the unnecessary overhead of performing a global lookup every time you access your table, and the significant memory waste of creating two new tables and discarding one table every time you append anything, and this is just horribly inefficient and unusable code that you should never even consider using.

[Billtopia]

Nah... that was a thought that didn't work out lol

only way I could see it would have a to unpack the table,
have a loop create a string to recreate the table values,
append the added values to the string,
and then loadstring it...

only problems I see is then you are throwing out functions rapidly instead of tables, and wasting more cpu cycles and memory

the only real question is how huge are these tables that you have to worry about the garbage collection? or is there another way to write the code so that you don't need to use so many tables?

Oh well... I guess lua doesn't enjoy the nils as much as blizzard enjoys supplying them as returns.
(although the logical sweetness of nil == false and not( nil or false) == true almost gives a 3rd logic state )