Thread Tools Display Modes
05-07-12, 05:56 PM   #1
Ketho
A Pyroguard Emberseer
 
Ketho's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2010
Posts: 1,026
tables, table.unpack and nil values

  • I'm confused why the first example does print the nil values, but the latter does not.
    Aren't these tables the "same"?
    1. Lua Code:
      1. local t = {nil, nil, 3, 7, nil, 4}
      2.  
      3. print(#t) -- 6
      4. print(unpack(t)) -- nil, nil, 3, 7, nil, 4
    2. Lua Code:
      1. local t = {}
      2. t[3] = 3
      3. t[4] = 7
      4. t[6] = 4
      5.  
      6. print(#t) -- 0
      7. print(unpack(t)) -- nil
    I was always thinking that table.unpack can't iterate over "holes" in the same way as an ipairs-loop
    (I'm not a programmer, and I don't know what's the proper terminology)

  • On another note, I'm also wondering about this (4 months old) post from Lur,
    concerning tables with a nil value at the end of a table
    Originally Posted by Lur View Post
    Thanks a lot. But why
    Lua Code:
    1. local t1 = {1, 2, nil, "wwew", nil, 3} -- #t1 = 6
    2. local t2 = {1, 2, nil, "wwew", 3, nil} -- #t2 = 2

    What's wrong with all of that? Why tables with a nil as the last value are truncated to the first nil occurrence while with any non-nil end are counted correct?
  Reply With Quote
05-07-12, 07:05 PM   #2
Foxlit
A Warpwood Thunder Caller
AddOn Author - Click to view addons
Join Date: Nov 2006
Posts: 91
unpack(t, i, j) returns t[i], t[i+1], ..., t[j]; if you do not specify i or j, they're defaulted to 1 and #t respectively -- so whether unpack will return values from an array with holes in it depends on the behavior of #t.

The Length Operator

The length of a table t is defined to be any integer index n such that t[n] is not nil and t[n+1] is nil; moreover, if t[1] is nil, n can be zero. For a regular array, with non-nil values from 1 to a given n, its length is exactly that n, the index of its last value. If the array has "holes" (that is, nil values between other non-nil values), then #t can be any of the indices that directly precedes a nil value (that is, it may consider any such nil value as the end of the array).
If your array has holes in it, there are no guarantees as to which value #t (of those described in the quote above) will return, as illustrated by your examples. There's nothing wrong with the code you've provided as far as the language is concerned. You simply should not expect or rely on #t to return specific values for such arrays.


On a tangential note, ipairs explicitly iterates over t[1], t[2], t[3], ... stopping at the first nil value rather than using the length operator, so it'll never make it past the first nil in the array.
__________________
... and you do get used to it, after a while.
  Reply With Quote
05-07-12, 07:25 PM   #3
SDPhantom
A Pyroguard Emberseer
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Jul 2006
Posts: 2,313
There are quirks in how Lua stores tables internally. Every table is created with 2 data arrays, one for sequential data and another for associative data. Sequential data is stored when you assign a non-nil value to the next index in a table, this index is equal to #t+1. If you assign data to a key that is greater than the next index or is a non-number, the data is stored in the associative array. Both the length operator and ipairs() operate exclusively off the sequential array.

The table constructor { } breaks this rule in its own way in which undefined keys are written directly into the sequential array and may include embedded nils. The table constructor also trims any nils from the right side of the list.

There is another quirk in Lua that allows you to manually force nils to be embedded into the sequential array of a table. This is done by inserting two non-nil values then overwriting the first with nil.
__________________
WoWInterface AddOns
"All I want is a pretty girl, a decent meal, and the right to shoot lightning at fools."
-Anders (Dragon Age: Origins - Awakening)

Last edited by SDPhantom : 05-07-12 at 07:49 PM.
  Reply With Quote
05-07-12, 07:50 PM   #4
d87
A Chromatic Dragonspawn
 
d87's Avatar
AddOn Author - Click to view addons
Join Date: Jan 2006
Posts: 163
http://www.lua.org/source/5.2/ltable.c.html#luaH_getn

{x,y,z} seems to be an "array" notation, which makes sense to have (for example to store function arguments with holes in a table).
When you construct table that way, sizearray variable is filled without checking about holes. Later, when you request table length, it just checks if last element according to sizearray is nil, and if it is, tries to find boundary(non-nil value followed by nil value) manually.
Like in t2 example, it might not be the the boundary you want, so you better strip nils yourself in this case.

Code:
local t3 = {nil, nil, 4, 7, nil, 4}
t3[5]= 2   -- #t3 = 6
t3[7]= 2   -- #t3 = 0
Also, if you try to expand your array, sizearray is recalculated to first boundary it founds.

Edit: i'm late

Last edited by d87 : 05-07-12 at 08:07 PM.
  Reply With Quote
05-08-12, 12:13 AM   #5
SDPhantom
A Pyroguard Emberseer
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Jul 2006
Posts: 2,313
Originally Posted by d87 View Post
Code:
local t3 = {nil, nil, 4, 7, nil, 4}
t3[5]= 2   -- #t3 = 6
t3[7]= 2   -- #t3 = 0
Also, if you try to expand your array, sizearray is recalculated to first boundary it founds.
I've tested this with the following function, sizearray doesn't ever seem to decrease like said.

Lua Code:
  1. function tAppend(tbl,...)
  2. --  This should allow unpack() to work with embedded nils as if {...} was used
  3. --  We need to keep the array allocation intact, Lua ignores setting the next index to nil
  4. --  and creates gaps, pouring the next values into the non-array section of the table
  5.     local tlen,previsnil=#tbl,false;
  6.     for i=1,select("#",...) do
  7.         local id,val=tlen+i,select(i,...);
  8.         tbl[id]=val or false;-- Force to an actual value
  9.         if previsnil then tbl[id-1]=nil; end
  10.         previsnil=(val==nil);
  11.     end
  12.     if previsnil then tbl[#tbl]=nil; end
  13.     return tbl;
  14. end

Running the following code allows it to resize the table a few times to give it a chance to recalculate the sizearray value, but the embedded nils are still counting towards the size.

Code:
local t=tAppend({},1,nil,nil,4,nil,6,7,8);
print(#t);	-- Prints 8




To make the process better understood, this is the same as the following:
Code:
local t={};

t[1]=1;
t[2]=false;
t[3]=false;	t[2]=nil;
t[4]=4;		t[3]=nil;
t[5]=false;
t[6]=6;		t[5]=nil;
t[7]=7;
t[8]=8;

print(#t);	-- Prints 8
__________________
WoWInterface AddOns
"All I want is a pretty girl, a decent meal, and the right to shoot lightning at fools."
-Anders (Dragon Age: Origins - Awakening)
  Reply With Quote
05-08-12, 01:44 AM   #6
d87
A Chromatic Dragonspawn
 
d87's Avatar
AddOn Author - Click to view addons
Join Date: Jan 2006
Posts: 163
Code:
local t=tAppend({},1,nil,nil,4,nil,6,7,8);
print(#t);	-- Prints 8
It's 1 on 5.1. But on 5.2 it is indeed 8...
  Reply With Quote
05-08-12, 03:14 PM   #7
SDPhantom
A Pyroguard Emberseer
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Jul 2006
Posts: 2,313
Originally Posted by d87 View Post
Code:
local t=tAppend({},1,nil,nil,4,nil,6,7,8);
print(#t);	-- Prints 8
It's 1 on 5.1. But on 5.2 it is indeed 8...
WoW has been running 5.1 since BC, they have not updated to 5.2 as of now.
All of my observations have been tested on the current implementation of Lua running on WoW.
__________________
WoWInterface AddOns
"All I want is a pretty girl, a decent meal, and the right to shoot lightning at fools."
-Anders (Dragon Age: Origins - Awakening)

Last edited by SDPhantom : 05-08-12 at 03:19 PM.
  Reply With Quote
05-08-12, 06:22 PM   #8
Ketho
A Pyroguard Emberseer
 
Ketho's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2010
Posts: 1,026
Smile

Well this answers a lot for me. Thank you Foxlit, SDPhantom and d87 for the, very clear explanation

Although I had some difficulty trying to understand about the length operator and the "embedded nils" in the table constructor, and the difference with ipairs

But especially Lur's example was one of the weirdest things about Lua that prompted me to ask this question
  Reply With Quote
06-18-12, 04:00 PM   #9
Billtopia
A Flamescale Wyrmkin
AddOn Author - Click to view addons
Join Date: Apr 2009
Posts: 110
something on unpack that I didn't see above is for nils is in order for it to properly unpack the table all the values should be assigned at one time like the following

local temp = { nil, 1, nil, nil, nil, "blah", whatever }

if they are assigned separately it would not unpack right but as a single declaration the following would work fine

local var1, var2, var3, var4, var5, var6, var7 = unpack( temp )
  Reply With Quote
06-18-12, 07:03 PM   #10
SDPhantom
A Pyroguard Emberseer
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Jul 2006
Posts: 2,313
Originally Posted by Billtopia View Post
something on unpack that I didn't see above is for nils is in order for it to properly unpack the table all the values should be assigned at one time like the following

local temp = { nil, 1, nil, nil, nil, "blah", whatever }

if they are assigned separately it would not unpack right but as a single declaration the following would work fine

local var1, var2, var3, var4, var5, var6, var7 = unpack( temp )
See my post above showing code for a custom function I made called tAppend().
The second and third paragraphs of my post preceding that also describes exactly what you said and a workaround for it.

Note calling tAppend(table,nil) will produce an undefined result, the list of values needs to end with a non-nil value.
__________________
WoWInterface AddOns
"All I want is a pretty girl, a decent meal, and the right to shoot lightning at fools."
-Anders (Dragon Age: Origins - Awakening)

Last edited by SDPhantom : 06-18-12 at 07:11 PM.
  Reply With Quote
06-19-12, 08:02 AM   #11
Foxlit
A Warpwood Thunder Caller
AddOn Author - Click to view addons
Join Date: Nov 2006
Posts: 91
Originally Posted by SDPhantom View Post
See my post above showing code for a custom function I made called tAppend().
The demo code for your function prints 1 for me on live realms, rather than the 8 you advertise. Perhaps trying to build functions around unspecified behavior is a bad idea after all?
__________________
... and you do get used to it, after a while.
  Reply With Quote
06-19-12, 11:00 AM   #12
SDPhantom
A Pyroguard Emberseer
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Jul 2006
Posts: 2,313
Originally Posted by Foxlit View Post
The demo code for your function prints 1 for me on live realms, rather than the 8 you advertise. ...
It's worked reliably on my system through several tests and retests across several sessions using the most current version of WoW. Since this is the second time this code has been questioned, I'll have to see if I can get it to glitch on another system.



Originally Posted by Foxlit View Post
... Perhaps trying to build functions around unspecified behavior is a bad idea after all?
Discovering undocumented effects is a way to open a path to new features and ideas. For example, the fact that the metatables for UI objects are all shared among objects of the same type. Blizzard hasn't released any info that states this behavior, yet it has opened up the possibility to hook a single function to be triggered for all frames of that specific type.
__________________
WoWInterface AddOns
"All I want is a pretty girl, a decent meal, and the right to shoot lightning at fools."
-Anders (Dragon Age: Origins - Awakening)
  Reply With Quote
06-19-12, 11:05 AM   #13
Billtopia
A Flamescale Wyrmkin
AddOn Author - Click to view addons
Join Date: Apr 2009
Posts: 110
try this to append items to your table so it will be "unpack"able

so put your table as 1st arg and values to add as 2nd, 3rd, ...

Code:
 tAppend = function( myTable, ... )
     myTable = { unpack( myTable ), ... }
 end
and this for the size if # don't work

Code:
 tSize = function( myTable )
    local temp = { unpack ( myTable ), "EndOfMyTable" }
    return #temp - 1
 end
or if that doesn't work then
Code:
 tSize = function( myTable )
    local temp = { unpack ( myTable ), "EndOfMyTable" }
    local x = 1
    while true do
         if type(temp[x]) == "string" and temp[x] == "EndOfMyTable" then
             return x - 1
         end
         x = x + 1
     end
 end
  Reply With Quote
06-19-12, 11:53 AM   #14
SDPhantom
A Pyroguard Emberseer
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Jul 2006
Posts: 2,313
The original idea of the function was to preserve the existing contents of the table and add new values to the end, hence the name tAppend(). Though it would be easy to wrap the table constructor in a function, there is nothing to gain from doing so. Any other possible ways to achieve the original design would risk a stack overflow and/or cause huge spikes in memory usage. No other method I could think of would only work on modifying the original table as intended.
__________________
WoWInterface AddOns
"All I want is a pretty girl, a decent meal, and the right to shoot lightning at fools."
-Anders (Dragon Age: Origins - Awakening)

Last edited by SDPhantom : 06-19-12 at 12:00 PM.
  Reply With Quote
06-19-12, 12:59 PM   #15
Billtopia
A Flamescale Wyrmkin
AddOn Author - Click to view addons
Join Date: Apr 2009
Posts: 110
for everything I have read on unpack it appears the method of recreating the table as I did above ( with or without the function wrapper ) is the only easy way to guarantee a proper unpack when you have nils within your table.

http://www.wowwiki.com/API_unpack gives a few ideas but nothing that would really allow you to add on easily to the table on the fly
  Reply With Quote
06-20-12, 01:20 AM   #16
SDPhantom
A Pyroguard Emberseer
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Jul 2006
Posts: 2,313
Originally Posted by Billtopia View Post
Code:
 tAppend = function( myTable, ... )
     myTable = { unpack( myTable ), ... }
 end
Reading this more, I discovered a couple fatal flaws. When you're referencing to unpack(myTable), since it isn't the last value in the list for the table constructor, it's only using the first value returned and discarding the rest of the table. Secondly, you're only reassigning the local variable of the function to the new pointer, this won't affect the table passed in any way.
__________________
WoWInterface AddOns
"All I want is a pretty girl, a decent meal, and the right to shoot lightning at fools."
-Anders (Dragon Age: Origins - Awakening)

Last edited by SDPhantom : 06-20-12 at 01:24 AM.
  Reply With Quote
06-20-12, 07:16 AM   #17
Billtopia
A Flamescale Wyrmkin
AddOn Author - Click to view addons
Join Date: Apr 2009
Posts: 110
I believe the thing to do then would be the following to get all of the values from unpack

myTable = { {unpack( myTable )}, ... }


and as for only changing the local pointer... I myself normally use one table with all my addon values within it so I can access it using something like:
myTable["Key"]
I pass whatever function needs it as the table and the key as a string

that allows me to access the table in functions and not a local copy... you could also do close to the same if the table is in the global namespace:
_G["MyVariable"]
just pass the table as a string
  Reply With Quote
06-20-12, 11:13 AM   #18
SDPhantom
A Pyroguard Emberseer
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Jul 2006
Posts: 2,313
Originally Posted by Billtopia View Post
Code:
myTable = { {unpack( myTable )}, ... }
This not only creates 2 new tables, but also changes the table structure.

Due to the design specs of the library my function was taken from, creating new tables is not allowed and completely contradicts the purpose of the library in the first place. The library I had this code in is a table recycler for use in code that calls frequently, perhaps multiple times each rendering pass. The idea behind it is to mimic the table constructor accurately on an existing table. Using an existing table means there's no new tables building up in memory waiting for a garbage collection pass. An ability of this library is that deallocated tables are stored as weak values, meaning when the garbage collector does run, they are released to it if they haven't been reallocated.

Note mimicking the table constructor accurately means including this undocumented effect.
__________________
WoWInterface AddOns
"All I want is a pretty girl, a decent meal, and the right to shoot lightning at fools."
-Anders (Dragon Age: Origins - Awakening)
  Reply With Quote
06-20-12, 05:08 PM   #19
Phanx
Cat.
 
Phanx's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2006
Posts: 5,617
Originally Posted by Billtopia View Post
myTable = { {unpack( myTable )}, ... }
If your original table looks like this:
Code:
myTable = { "one", "two", "three" }
#myTable == 3

And you add the values "four" and "five" on the end:
Code:
tAppend(myTable, "four", "five")
Then your code is actually doing this:
Code:
myTable = { { unpack( { "one", "two", "three" } ) }, "four", "five" }
which simplifies to this:
Code:
myTable = { { "one", "two", "three" }, "four", "five" }
#myTable == 3

You've just created a new table that still only has 3 values, and another new table containing the original three values. Now imagine what happens after you run that a couple of times:

Code:
myTable = {
	{
		{
			{
				{
					{
						"one",
						"two",
						"three",
					},
					"four",
					"five",
				},
				"six",
			},
			"seven",
			"eight",
			"nine",
		},
		"ten",
		"eleven",
	},
	"twelve",
}
Is that really a table structure you'd want to try to work with?

Plus, if you were to do anything like this, you would never be able to use local references to your table. Add together the ridiculously convoluted structure you see above, the unnecessary overhead of performing a global lookup every time you access your table, and the significant memory waste of creating two new tables and discarding one table every time you append anything, and this is just horribly inefficient and unusable code that you should never even consider using.
__________________
Retired author of too many addons.
Message me if you're interested in taking over one of my addons.
Don’t message me about addon bugs or programming questions.
  Reply With Quote
06-20-12, 07:54 PM   #20
Billtopia
A Flamescale Wyrmkin
AddOn Author - Click to view addons
Join Date: Apr 2009
Posts: 110
Nah... that was a thought that didn't work out lol

only way I could see it would have a to unpack the table,
have a loop create a string to recreate the table values,
append the added values to the string,
and then loadstring it...

only problems I see is then you are throwing out functions rapidly instead of tables, and wasting more cpu cycles and memory

the only real question is how huge are these tables that you have to worry about the garbage collection? or is there another way to write the code so that you don't need to use so many tables?

Oh well... I guess lua doesn't enjoy the nils as much as blizzard enjoys supplying them as returns.
(although the logical sweetness of nil == false and not( nil or false) == true almost gives a 3rd logic state )
  Reply With Quote

WoWInterface » Developer Discussions » Lua/XML Help » tables, table.unpack and nil values

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off