How do i optimize a tool, or find out how long it takes for it to reload?

[esaruoho]

how do i figure out how long it takes for a tool to load all it’s parts? does having 35 separate .lua files loaded from main.lua mean it’s heavier, than when having all the functions + menu entries + midi mappings + shortcuts in one .lua file?
any ideas about optimization would be appreciated.

side question: let’s say i have this many lua scripts being loaded on main.lua.

would this result in slowdowns, because it feels like it takes about 3-5 seconds for the tool to reload. if i combined these into one main.lua, would they take less space?

[joule]

You know about the os.clock() function, and doing rudimentary benchmarks with it?

I assume you have a lot of requries in your tool that you want to measure, so you could put the init line at the first line in main.lua and then printing the duration at the very end of main.lua.

local init_time = os.clock()

-- anything between here

print(os.clock()-init_time)

[esaruoho]

hmm does that mean i could use this with a require “file” sandwiches inbetween?
and that way result in some info on what takes longer to load? i just edited my original a bit ^^

[joule]

I mean, everything in lua is sequential (as per turing machines, eh?) so you should be able to use those two lines to check the duration of what happens inbetween.

If you are running into the issue of tools taking long time to load, due to bytecoding all functions upon init, I would look into calling require ‘file’ only upon demand if possible. I think you can call require at any place and time, so potentially I THINK you can have a long main.lua with all your shortcut definitions and only calling “require” inside the actual shortcut def code. It should be possible, and would save some memory and start-up time for anyone who installed your super-tool.

I haven’t confirmed this use of require but see no reason why it shouldn’t work. Lemme know…

[esaruoho]

is this a lot?

rx
0
0
Loaded 0g01
0
0
Loaded PakettiKeyBindings
0
0
Loaded Customization
0
0
Theme Selector
0
0
Skipped if not v2.8
0
0
eSpeak Loaded
0
0
PlayerProSuite
0.01600000000326
0.01600000000326
base64float
0.01600000000326
0.01600000000326
Automation
0.01600000000326
Audio Processing
0.01600000000326
BeatDetect
0.01600000000326
PakettiControls
0.01600000000326
DeviceChains
0.01600000000326
PakettiDynamicViews
0.01600000000326
Gater
0.01600000000326
0.032000000006519
ImpulseTracker
0.032000000006519
InstrumentBox
0.032000000006519
PakettiLoaders
0.032000000006519
LoadDevices
0.032000000006519
LoadPlugins
0.032000000006519
MainMenuEntries
0.048000000009779
Midi
0.048000000009779
MidiPopulator
0.064000000013039
PatternEditor
0.064000000013039
CheatSheet
0.064000000013039
PatternMatrix
0.064000000013039
PatternSequencer
0.064000000013039
Phrase Editor
0.064000000013039
OctaMED
0.064000000013039
SampleLoader
0.064000000013039
Samples
0.080000000016298
Stacker
0.096000000019558
TKNA
Recorder
0.096000000019558
Coluga
0.096000000019558
Requests
0.22400000001653
0.22400000001653
PakettiEightOneTwenty
0.24000000001979
Experimental
0.24000000001979
PakettiSandbox
0.24000000001979
Stretch
0.24000000001979
Tuplet Generator
0.24000000001979
PakettiWavetabler
0.24000000001979
Finished

EDIT: feels like i need to organize this better in order to be able to figure out what the timesinks are.
and what is 0.24 in this case? 24 seconds? 2.4 seconds?

[joule]

If that is truly from the start of main.lua to the end of main.lua, then your tool takes 0.016s to load and is of no matter. I guess the lua bytecoding is super fast. I don’t think your tool is the issue here (?).

[esaruoho]

but… what is 0.24?

[joule]

os.clock() returns seconds. If it returned 0.24 it would be 0.24 seconds.

Do you run a single benchmark here? 0.24 is almost nothing, but if you have many of those it would add up.

1/4 of a second for loading a tool we could live with, I think.

[esaruoho]

it’s a codeblock and it requires scrolling - which seems to be invisible on this forum.
there are for loops that create shortcuts for 512 things at a time (if there’s a need for 512 shortcuts for a specific thing).
i’m gonna try and organize these so that the shorter ones are first and the large chunks that take more of the 0.24seconds, are last, and then start looking at whether it takes less seconds to splice them into smaller pieces, or what to do.
or maybe turn some of the 512 shortcut generators off and see what happens.
all i know is it takes about 2-4 seconds to boot up&reload the tool after every save, and people are complaining that the tool, in non-running-mode, is cpu-intensive and makes renoise laggy.
and no, i haven’t gone wild with notifiers. i hardly have any.

[joule]

Ah interesting…

I have some vague memories of similar experience. Perhaps it happened with gui intensive stuff, meaning viewbuilder bytecoding creates some of this overhead. Just a very wild guess.

See if you can implement require on demand. But if your benchmark says 0.24s, it should be the truth.

However… if you reload the tool, maybe there is a lot of garbage collection going on before it actually reloads. That could be an alternative culprit, if being speculative. Not being the loading that consumes time, but the “unloading” of the previous state? I have some old hunches that viewbuilder garbage collection is so-so… No idea, but I think these kind of terms are worth investigating.

[esaruoho]

is it possible that it’s the 80kb of preferences.xml that is the bottleneck here?

rx, 2318 lines, 1.75 ms
Paketti0G01_Loader, 857 lines, 2.00 ms
PakettiKeyBindings, 1443 lines, 1.38 ms
PakettiCustomization, 61 lines, 0.25 ms
PakettiThemeSelector, 516 lines, 2.12 ms
PakettieSpeak, 930 lines, 2.63 ms
PakettiPlayerProSuite, 852 lines, 2.37 ms
base64float, 203 lines, 0.25 ms
PakettiDeviceChains, 85 lines, 0.25 ms
PakettiLoadDevices, 496 lines, 0.50 ms
PakettiLoadPlugins, 534 lines, 0.50 ms
PakettiPatternSequencer, 47 lines, 0.50 ms
PakettiPatternMatrix, 176 lines, 0.37 ms
PakettiSampleLoader, 0 lines, 0.00 ms
PakettiSandbox, 352 lines, 0.38 ms
PakettiTupletGenerator, 386 lines, 0.50 ms
PakettiAudioProcessing, 1538 lines, 1.25 ms
PakettiBeatDetect, 396 lines, 0.88 ms
PakettiControls, 544 lines, 1.25 ms
PakettiInstrumentBox, 280 lines, 0.63 ms
PakettiPhraseEditor, 461 lines, 1.13 ms
PakettiOctaMEDSuite, 601 lines, 0.87 ms
PakettiStacker, 518 lines, 0.75 ms
PakettiRecorder, 403 lines, 0.63 ms
PakettiColuga, 854 lines, 0.87 ms
PakettiStretch, 925 lines, 0.87 ms
PakettiWavetabler, 223 lines, 0.50 ms
PakettiAutomation, 2776 lines, 2.50 ms
PakettiDynamicViews, 703 lines, 4.75 ms
PakettiGater, 1233 lines, 2.75 ms
PakettiImpulseTracker, 2112 lines, 2.12 ms
PakettiMainMenuEntries, 383 lines, 6.62 ms
PakettiMidi, 1692 lines, 5.12 ms
PakettiMidiPopulator, 531 lines, 0.50 ms
PakettiLoaders, 3137 lines, 6.62 ms
PakettiPatternEditor, 4583 lines, 9.50 ms
PakettiPatternEditorCheatSheet, 953 lines, 1.13 ms
PakettiSamples, 4249 lines, 6.50 ms
PakettiTkna, 1495 lines, 19.37 ms
PakettiEightOneTwenty, 1457 lines, 5.75 ms
PakettiExperimental_Verify, 4543 lines, 6.50 ms
PakettiRequests, 9168 lines, 117.75 ms
Total load time: 0.224 seconds

organized the long ones to the end, etc.

[joule]

Having the API (or c backend?) parse 80kb of preferences Document really sounds as if it could be heavy, yes. I assume quite a lot of it is parsed on the lua side even (if it’s using Document API), which probably entails quite a bit of object creation and string manipulation.

It might be a good idea looking into refactoring all of your structure, to load everything on a need-to-be-loaded basis. I was thinking about this a long time ago for your tool. Compartmentalize into classes that only load prefs and ‘requires’ when needed. Didn’t know it would actually become an issue though…

Disable all prefs loading possible to see what it does in terms of reload speed? Pinpoint. If it breaks everything, then load only the prefs into a fresh tool and benchmark that loading with os.clock.

PS. Just saying how I would go about pinpointing the issue. I don’t claim absolute expertise.