It is normal for nUI to tick off about 2-300k every couple of seconds since it is performing an enormous amount of work at anywhere between 30 and 90 times a second. However, most of the memory it is ticking off was temporary dynamic memory (aka garbage) and it should be getting released by the garbage collector periodically.
As a rule, nUI tends to use around 11meg when solo and that will tend to tick up until you hit around 13 to 15meg and then reset. If you are in a 40 man raid, that number tends to be more like 35meg and ticking up to about 50meg or so before it resets.
nUI has *no* control over the garbage collector and when it runs. If your system is ticking up memory until it runs out and then throwing an error, then something in your WoW installation is *very* broken. WoW's garbage collector is known to have issues since the 3.0 patch, but the problem you're describing is the most sever I've ever heard and, frankly,
I don't know what to tell you... have you tried the clean install for nUI? If not, follow the directions here
http://www.wowinterface.com/forums/s...ad.php?t=21716 and see what happens. I have to admit though that I'm not hopeful its going to fix it in your case. I have a suspicion that your WoW install itself is corrupt. You could try running the repair utility though I have personally had little luck with that fixing problems like this. Alternatively, as painful as it may be, you may need to reinstall WoW from scratch.
If it matters any, I am busy incrementally re-writing the core of nUI's event engine to eliminate as much garbage production as I possibly can in order to reduce the impact of the garbage collector problem introduced in patch 3.0, but I do think your problem runs deeper than just that.