Lua/Renoise Scripting Performance Tips & Questions

My contributions will probably be mostly supposedly bright ideas that will turn out to be horribly wrong, but it’s all for the sake of learning right? So share away, we can sort it out later.

The only thing I have to offer for now is this:

Make main lua contain nothing but the menu entry, which only when it is invoked reqires the file where all the actual functions are, and calls the main function.

I think if you have a hundred scripts, it would really help if those were all small, since they all get loaded at startup… !

But the real reason I like this and will try to stick to it from now on, is that editing files that are all called main.lua is confusing and painful, I’d rather have descriptive names

renoise.tool():add_menu_entry {  
 name = "Sample Editor:Process:Rubberband...",  
 invoke = function()  
 require("rubberbandremix")  
 show_rubberband_dialog()  
 end  
}  

168 bytes! Sure, most tools have more menu entries (and hotkeys), but the point is the same - everything that you do not have to do at startup, move to an external file which only gets loaded once the tool is actually activated. Keep your main.lua as slim as possible. That will not so much improve the performance of your own script, but together with all other scripts it can help keep Renoise startup times as short as they are.

I just think once we have hundreds of scripts, you would notice a MASSIVE difference between every main.lua just being a short stub, and between main.lua containing the main routines, or even all of the code. I might be wrong though, so maybe people with more clue can confirm or correct it.

But if it is a good idea, I think it’s better to pick up such habits earlier than later, especially since just like with pollution and global warming, each individual doesn’t really notice anything, but suddenly, Renoise takes longer and longer to load… wakes up in cold sweat

Since your script get’s executed by Renoise on load it should also load the required includes at this point, so you’re basically adding a call to open an extra file instead of saving performance, but to be honest… on modern computers you’re probably not notice the difference anyway. Also be careful adding a require directly into a menu function. If you have only one menu entry there should be no harm, but imagine now you have 3 entries and each includes the same file into the menu function you’ve sort of tripled the size of the document which the interpreter has to parse, so it’s much better to place that require at the top in main.lua. Maybe LUA is smart enough to optimize this away, but i wouldn’t count on this. You can sort of imagine a require command like a text replace action. The require get’s replaced by the file contents you specify there.

Regarding performance tips: If you have a function which does lot’s of value comparisons, try to specify these comparisons first which are most likely to be true more often. For small scripts this is not much of an impact on performance, but for bigger projects it can add up. Even though as usual, on modern computers saving a few milliseconds is hardly noticable at all, so i wouldn’t worry too much about that.

More tips are here: http://lua-users.org/wiki/OptimisationTips

Important note: you do have to define your preferences at startup by the way, so that goes into main.lua too (Just spent 10 minutes being confused by this ^^)

But the script (quite often) doesn’t get executed, only init stuff does, menu entries, preferences etc.

So it’s not even known yet if all the other stuff ever will be required. Most scripts just sit there, after all, getting used every now and then, but not every Renoise session.

That’s the thing: on modern computers, you won’t notice the difference between including one file and including two, when calling the script… however, at startup main.lua’s of all scripts get loaded, one after the other, as fast as possible, so there it can really add up to a point where you do notice it on any machine… When you invoke a tool by menu entry or shortcut, that’s the only thing going on, but startup is very different.

That’s why I bring this up, since individual plugin authors might not notice a difference, but it accumulates after a while. And since many people will start out with cloning existing scripts, we better set an example while we still can Because I WANT to have 500 scripts installed. That’s the very least! And that will never be instantly loaded no matter what. But that I’m afraid that might really suck if everything gets put into main.lua.

as opposed to dofile and loadfile, require only loads each file once, so that’s a non-issue fortunately

“The First Rule of Program Optimization: Don’t do it. The Second Rule of Program Optimization (for experts only!): Don’t do it yet.”

Third Rule that applies here: If its really getting a problem at some time: Let ME take care of this “internally” in Renoise. This is nothing you guys should bother with. Its not about optimizing DSP code or something…

I wouldn’t call that optimization, I’d call it separating the stuff you need for registering the pluging from the stuff you need when actually doing something. I do it mostly because it’s cleaner actually.

The way I hack about with LUA it would be really silly to talk about performance anyway. But still, why throw startup time away for no benefit, just so you can “take care of it”? What is there you could do about it internally? A file has to be loaded before it can be parsed, and if you’d rather preparse them than people using some common sense, be my guest.

I mean, you don’t have to give us tips, but to say we shouldn’t do this because we’re small children and this isn’t big boy stuff and hence not worth the effort is just silly sorry, and the tone on these forums is very different from the fullmouthed “everybody is welcome to join the hacking” announcement. And I don’t mean that people are busy, but this attitude of “let the pros handle that” is annoying. If you can’t or don’t wanna help fine, but why actively discourage people from being curious like that? RTFM, and don’t you dare complain if that gives you eye cancer… bleh to that.

I don’t care, for me it’s a challenge, I derive the joy from trying, not from having it handed to me on a silver platter. BUT I also am a user and am looking forward to using lots and lots of scripts… so again, what’s the harm in keeping main.lua slim? It’s good practice in two ways: it forces you to think about what is actually going on and when, and it makes sense, especially since startup means doing the same stuff over and over again. Every second shaved off startup is 1 minute shaved off the week. Sure, that’s not “important”, but then again what is?

If I could think of a good way of automating it, I’d try creating a hundred plugins with a stub and a main file, and a hundred plugins where all is one file, just to see what happens. But I’m pretty sure there will be a difference, and unless you precompile all the scripts there is nothing you can do to simply “handle that”, that’s just arrogant, and even if you could that would be a suboptimal solution to simply not loading stuff when it’s not actually needed.

I mean, geez, this is basically all I suggested: don’t load stuff you don’t need. AKA no-brainer. But that, according to you, is for experts (just like modal dialogs and all that other quantum mechanics hi tech shit), and IF it becomes a problem, then the right solution is not that we all become experts in the problem we are causing by not being experts, but that papa taktik handles it? LOL, really. Thank fuck there are LUA forums, there’s just no point in discussing this here.

This is advice that comes from decades of general computing experience, yet you are taking it personally and starting yet another troll tread.

Google “Premature optimization”… Geez. Why do you take everything personally?

The reason why you should not optimize is that its in most cases not even worth the time thinking about it. In contrary, if you optimize stuff that doesn’t need to be optimized, you just make everything more complicated than it has to be. Then, if you really want to optimize, you have to know what to optimize, what exactly the problem/bottleneck is, and not what maybe the problem could be - in theory. First measure the problem, then solve it. -> not thinking about stuff when not actually needed.

The reason why it has to be me who does this, is not because I’m the best Lua scripting dude in the world and you all are children, but because I think that IF we want/need to take care of this, then this should be done internally in Renoise - for every single script that runs in Renoise, and not just a few ones.

But how would you do that? Not being cocky, but curious. Preparsing them I guess, and caching just the stuff that is needed at startup?

To me it kinda did seem like a “problem” when I realized that I had a main.lua of 20KB which actually only needs to be 1kb. Not a performance problem, more like a “this annoys me because it’s unelegant and unnecessary” problem.

I really think this actually more an exercise for newbies than hardcore optimization. As I said, it forces you to think more, about program path and variable scopes. Dunno, I actually liked doing it, that’s the main reason I shared it (even though I probably made it sound like it’s a huuuge performance issue). I also think it’s nicer to have the main files called “scriptname.lua” instead of “main.lua” (in text editors that don’t show the full path for example), but that’s just a bonus.

Maybe in an attempt to save face I’ll still come up with something to automate creating a LOT of scripts (shouldn’t be hard with LUA), and then see what happens.

Dunno. Following the rule above, I will, we will see !when! it gets a problem

LOL? I know what premature optimization is, and don’t think this is. It’s just good style.

And what I did I mistakenly take personally? You make it sound like taktik didn’t directly address what I said, but just said something I about optimization which I then misunderstood to apply to what I said. bleh.

or do you mean the general venting? what in there makes you think I’m personally butthurt? The condescending attitude doesn’t discourage ME, but it may discourage others, and it’s just daft when it’s not newbies bugging the devs, but newbies chattering amongst each other. Since last monday there have been a few moments where I thought, WTF, why so rude and unhelpful, if you don’t wanna reply then just don’t so someone else will, instead of giving an unhelpful, superficial reply that just shuts any further discussion down… and this was the occasion where I let it all out. Big deal, hmm? Got a problem with anything I said, address it directly. (who are you calling troll?)

And starting yet another troll thread? WTF. Why is this a troll thread, and could you name at least one other I started recently? Haha…

Worrying about bytes of text is a non-issue. If you are really interested in all this, start here:

Specifically the concept of a Virtual Machine.

I don’t want to be an asshole, but I did two semesters at university about “compilers” and barely passed the courses. You have a lot of reading to do. Scripting is meant to be readable, worrying about stuff that isn’t algorithmic is not very useful. The concept of main.lua comes from C, thus the idea of conventions.

My philosophy, learn by doing, communicate without being aggressive because the learning never ends, and you look like a knob when you finally get it. Been “scripting” for over 10 years, I’ve barely scratched the surface, eureka moments every other day.

Bonus reading:

http://c2.com/cgi/wiki?PrematureOptimization

The whole site above is a mini bible, but a few clicks away you get:

"You are unlikely to know up front whether an optimisation will be of any real benefit. Just write the code the simplest way. If, eventually after profiling you discover a bottleneck optimise that. "

Is Taktik a genius or someone who knows how to cut and paste from advice that’s been around for decades?

Good times.

thanks for the internals link, but otherwise…

haha… look, if you read and interpret 100 2kb files, or read and interpret a 100 20kb files, that will be a difference, no compiler course required to now that. Yeah, harddrives read in chunks so maybe it’s more like 8kb stubs versus 100kb blobs… still, at some point it makes a difference, and for notebooks where every kilobyte of memory counts, it’s nice to be efficient, too. Because there is no good reason not to be.

And regarding speed, sure it’s not much but it’s at startup, all at once, if I can have cleaner code AND avoid loading stuff that isn’t necessary, saving memory and maybe even startup time – fuck yeah I’m all for it, sign me up. Oh, it’s simple and makes keeping track of the projects easier, too, well, wow! Sounds like having the cake and eating it?

What exactly is unreadable about separating startup from main… now you’re just talking about “optimization” on your terms, or what? What does this have to do with anything here? You see, when you say “optimization” you mean doing something in a LESS readable or intuitive or comfortable way, for the sake of performance, right?

But when you get MORE logical code AND more performance, without any drawbacks, it’s not premature optimization, it’s just prettier and very slightly more efficient – which in turn can result in a measurable startup speed gain when hundreds of scripts do it, and not just one. That’s not rocket science.

No drawback, slight garuanteed benefit, potentially bigger benefit over time.

I keep asking, what’s the harm in doing that. And you just give me general computer theory bullshit. I ask, what is in this case, the harm of doing this specific thing? Especially since I didn’t command everybody do it, I just said I will and I think it’s a good idea – which still stands uncorrected.

Look, Smartypants, how am I going to optimise when the bottleneck is not in my program, but in the the sum of all the files loaded at startup? By many authors who start out by copying the structure of the provided examples and early scripts? So yeah, study less, think more…

Nothing, I do this all the time. I agree.

No drawback guaranteed? You have a benchmark? Cool. Show me the numbers. I’m ready to believe.

But, what if it turns out the opposite of what you are saying is true? What if there’s some “smart caching” set up and you work against it. Or that Lua is good at compiling everything at once, but bad doing it dynamically on the fly? Biggest bottleneck in most apps is file system I/O, increasing them via Lua require (this is a “high level” I/O interface) could be worse. You are working from assumptions, not facts. Not saying the assumptions aren’t reasonable, just saying they aren’t facts and dwelling on them before they are a problem is the textbook definition of “premature optimization”.

But at this point, I think this is a misunderstanding. Your thread is titled “Lua/Renoise Scripting Performance Tips & Questions” and I’m focused on “performance” or, more to the point, the need not to care. Whereas you you are talking style and best practices, which is another thing all together and I will duck out now.

It makes your code readable for the rest of the world if you split your stuff properly. I doubt if it will make things speed-optimal in every case though, but you would help out beginners to make things easier to understand. (yet if you structurize it properly)
Adding a require in a menu-definition is dangerous, specially if you need globals to share between everything and these globals should always be required in the top of your main.lua outside menu definitions.
I guess that is a part of what Taktik also meanted by don’t do it, the other part is, it probably doesn’t offer much use, speed-wise and the third part of the meaning is :We have a tool browser, we can disable unused tools (meaning they shall not be loaded) so dimminish loading time of tools.
But that’s just my simple 2 cents.

Nothing wrong about a clever separation of files in a lua program, but actually asking the lua interpret to re-evaluate a file everytime i click the menu item seems not a good idea, cause probably you are speeding up the initialization, but slowing a bit the action when it’s triggered.

Anyway Conner is right, we need numbers to prove the facts (and imho with a modern cpu you wouldn’t notice a real difference really)…

I can’t promise it soon but yeah, I already decided I’ll write a short script cranks out hundreds of scipts, because now I really wanna know.

Well yeah, that’s generally often true “these days”, good point.

I already mentioned earlier that I’d like to test it, so it’s not like I care more about my noob theory than about the facts.

I actually initially said “I might be shown incorrect”. But that would kinda mean a test as well, and not just telling me I haven’t proven it, that I know myself.

Well, that’s a bit cheeky of me then. I claimed it would be faster, and that’s separate from it being a good idea that “can’t hurt” (which seems to be agreed upon). So yeah, I’ll test it or rather write something to generate scripts so everybody can test it, but I can’t promise when.

But is that really what is happening?

I’m not 100% sure what to make of this:

require()

dofile():

loadfile()

Maybe it’s just bad writing, because it wouldn’t make sense if require was “doing the same job as dofile” in that it recompiles the code everytime (which the description of loadfile() seems to imply), or would it?

Because then yeah, it would be better to use loadfile and keep track manually to load each file only once, and reuse the compiled chunks, but I kinda assumed require does ALL that, otherwise it would just be silly… cache the file and load it only once, but each time it gets requested, recompile it??? nah…

That depends on the scope where this one is called…

Just for fun a test…:

  
require "mylocals"  
  
function myfunction()  
 require "myglobals"  
  
 if initial_global_variable=true then  
 require "myueberglobals"  
 end  
  
 --Try here accessing a variable or constant defined in myueberglobals.lua  
  
 myfunction2()  
end  
  
function myfunction2()  
 --Try accessing a variable or constant defined in myueberglobals.lua  
 --Try accessing a variable or constant defined in myglobals.lua  
  
end  
  

I know vv, but it’s not a problem when the require and the function call are in the same scope? However…

Well, I actually that and kraken & beatslaughter are right, and even though the file gets loaded and compiled only once, I think its contents get instantiated every time so to speak, each local variable in it the required file exists as often as the file gets required for example. Which is of course useless for big included files and lots of entry points that can be called, like in the example below… I have to test this more, but I yeah, doesn’t look good for my pretty little performance tip haha…

  
renoise.tool():add_menu_entry {  
 name = "Sample Editor:Process:RunCmdsOnSelection...",  
 invoke = function()  
 require "runcommandsonselection"  
 show_runonselection_dialog()  
 end  
}  
  
require "config"  
  
for preset_id, preset in pairs(t_gui_presets) do  
 if preset.menu_title ~= "" then  
 renoise.tool():add_menu_entry {  
 name = "Sample Editor:Process:"..preset.menu_title,  
 invoke = function()  
 require "runcommandsonselection"  
 launch_gui_preset(preset.name)  
 end  
 }  
 end  
 if preset.shortcut_title ~= "" then  
 renoise.tool():add_keybinding {  
 name = "Sample Editor:Process:"..preset.shortcut_title,  
 invoke = function()  
 require "runcommandsonselection"  
 launch_gui_preset(preset.name)  
 end  
 }  
 end  
end  
  

^ prolly not such a good idea after all

no obviously not recompile it, but instead re evaluate it in the function scope (which is different from just executing a ready to run function)… what about the global scope ?

i’m not against this, but i would rather keep myself away from require something in a limited scope unless there is a reason to do so (security mainly). this also makes reaching global variables a bit more obscure, cause you need to know which scope you are in.