Using External Lua Modules

Renoise Tool Development
#1

Hi there,

Not sure if i’m trying to do something beyond the scope of the lua implementation.

I have downloaded luasql - a DBMS interface module (http://www.keplerproject.org/luasql/) - the binary dll version.

I extracted the dll into Program Files\Renoise\luasql. When I try to ‘require’ the module I got an error “the specified module can not be found” - I found that the module was trying to access the Lua dll by a specific name (lua5.1.dll). I tried downloading some ‘forwarding’ modules which as I understand it are supposed to forward any requests for lua5.1 through to lua.dll. After installing at Program Files\Renoise the error message changed to “the specified procedure can not be found”.

I tried copying the full lua5.1.dll module from a full version of Lua. This seemed to work much better. I can connect to the database and create a cursor. However as soon as I attempt to fetch cursor results Renoise crashes ;)

Just wondering whether we should be able to use third party library/modules from within the Renoise lua implementation?

Thanks,

Jonathan

1 Like
Usage of sqlite in a Renoise tool?
#2

Renoises Lua interpreter does not look for system Lua library paths. The easiest and for Renoise tools most compatible way to use libraries, is to put them into the XRNX bundle you are developing. The XRNX bundle always is the current working directory (".") for xrnx scripts. Then you can also be sure that your tool is self-contained, does run on other computers which have no Lua and no external Lua libraries installed.

If you don’t care about making your tool usable for others, then you can also hack Luas module search path to look for libraries at any places:
package.path = package.path … “;SOME_PATH/?.lua”
package.cpath = package.cpath … “;SOME_PATH/?.dll” – or ?.so for linux, ?.dylib for OSX

See also http://pgl.yoyo.org/luai/i/package.path

1 Like
#3

Ten years, and still relevant. :slight_smile:

Hi taktik, is this advice of bundling extension .dll files with the tool still holding in today’s v3.2.1? I didn’t hack any search paths, but I’m seeing a strange behavior where I have to move my .dll out of the tool folder in order for it to require() successfully. Details here.

#4

If you don’t want to share the dll with other tools, it should be possible to simply copy the SomeLib.dll file next to the XRNX tool’s main.lua and then do a require(SomeLib). This will look for SomeLib.lua and SomeLib.dll in the bundle path.

2 Likes
#5

Thanks taktik, I did get require() for the .dll working by putting the bundle directory explicitly at the start of package.cpath (details in the other thread). Leaving ./?.dll as the first entry in cpath was causing problems.

Now .dll interop works, mostly, but I have some Lua compatibility issues causing crashes – I guess because my version of Lua is not exactly the same as what’s built into Renoise. Apparently, Renoise has Lua statically linked into the .exe so a native Lua module developer isn’t able to target a specific version of, say, lua-5.1.5.dll used by Renoise. I’ll find a workaround eventually, but it would be ideal to run the same exact C Lua functions that Renoise is using to modify a LuaState object. Deploying the companion .lib and .h files would make Lua native extensions easier to develop, but I haven’t seen anything like this yet.

#6

Yes, you’re right. To properly share the interpreter code between the binary Lua extension and the host (Renoise), Renoise would need to publish and link all core Lua interpreter functions in a shared DLL, which then can be linked by the Lua extension DLLs.

As you see, we never really bothered with this as no one really asked for it and we didn’t saw the need of it. What kind of extension would you like to build this way which isn’t possible in plain Lua?

#7

The Akai Fire has a 128x64 LED display that requires a lot of strange bit twiddling. The rendering and sysex generation code in Lua is unusably slow even after doing some hand optimization, but with my Rust module, the same is literally instantaneous. I know Lua is supposed to be fast, but native Rust (or C/C++) is so much faster that new kinds of things become possible.

Right now I have to accept one of these alternatives:

  • Fast updates in Rust, with crashes – unusable.
  • Slow updates in Lua, without crashes – unusable.
  • No LED at all – usable, but flying blind.

But with a broken-out Lua library, it becomes:

  • Fast updates in Rust/C/C++, totally usable and the possibilities are inspiring! :slight_smile:

I hope I’m just the first of many who would develop powerful native extensions for Renoise. It may be the rare script user that wants .dll modules, but some developers could potentially offer extraordinary value with native modules. For example, random access to Renoise song data could be used to quickly synthesize samples that do tweening and complex calculations based on the whole rather than “now” as typical DSP chains do. With native opened up, there are lots of interop possibilities, and workflows can be made fast and seamless.

I am a big fan of static linking, personally, but I do appreciate you taking a moment to consider these little projects. Renoise is awesome and the main reason I’m here is because Bitwig’s API is restrictive by comparison.

#8

Uh huh. Naturally this depends on how ‘open’ taktik makes interfacing C extensions to the internal Renoise framework/objects. Maybe developers (how many?) would want some C level access to the audio engine and graphics. Good documentation etc. A lot of work for taktik(?)

#9

A full native interface directly into Renoise internals would be cool, and not too much burden as long as no maintenance/compatibility promises are made – but yeah that would be quite some work.

What we’re talking about here isn’t a native Renoise API, though. Just using the existing Lua API, but making it possible to have native Lua modules. It’s a one-time process of breaking Lua out to .dll + .h + .lib files so that devs can do native/Lua interop.

#10

If you could post the related code here we maybe can help optimizing this a bit. The non standard bit library in Renoise (see https://files.renoise.com/xrnx/documentation/Lua.Standard.API.lua.html#h2_27) may also be helpful here.

#11

It’s a kind offer, thank you, but I don’t want to waste anyone’s time. Digging into my code would get messy, and any improvements would be marginal compared to what I already have working in Rust. I know Lua is supposed to be fast, and by scripting standards, sure, but hash table lookups in loops are never going to compete with bare metal register operations and machine-optimized zero cost abstraction. Inlining, branch elimination, etc. might help the Lua code but it’s still going to be orders of magnitude slower, which will affect the responsiveness of the tool.

I’m just kind of cheating with the native module because my linked Lua implementation is close enough to what Renoise runs – but it leaks 104 GC bytes on every interop call and might have other undefined behavior, too. :slight_smile: It’s okay. If tool devs can’t target a specific Lua lib, I’ll find another way eventually. Maybe run a service process with socket protocol or something. I was just hoping to stay close to Renoise without too much glue code – but I didn’t expect Renoise-side changes, really. Only brought up the .lib, .h, .dll thing because it seemed like an easy win all around – but I don’t know, you may well have great reasons to keep it all statically linked.

#12

As I investigate, I wonder if it really is even necessary to break Lua out to .dll in order to get this working properly. It may be easier than first thought, simply by statically linking an identical version of Lua into my own native module. I have the state pointers, and as long as the data, stack, etc. is changed exactly as it should be – then nothing should leak or blow up. I mention this because I thought Renoise was using Lua 5.1 but some aspects seem to be LuaJIT, and now I’m not sure. I can source it, if I know exactly what version to look for and the implementation hasn’t been changed for Renoise in a way that breaks binary interface.

#13

Just curious voidshine. (I don’t own an Akai Fire or have sent sysex data via lua, so I’m a bit stupid here.) I take it you transfer data to/from via sysex message(s). You say it has a 128x64 screen and it requires some ‘bit twiddling’. How much data is sent via the sysex message(s) (what sort of packet size roughly in bytes)? How often is the sysex transmitted? Is that the bottleneck in lua? Or is it the bit twiddling part? (Or is it both?)

#14

@4tey yes, IIRC it’s about 1183 sysex bytes for a full screen update. 8192 bits for all 128x64 pixels; 7 bits can be packed (in an unusual order) per sysex byte; plus some header/footer data. If I want to scroll a sample or do FL Studio alike visualizations, the update frequency should be at least 20Hz. The transmission takes several milliseconds, but the rendering and image conversion was the main time sink…dozens of milliseconds at least. I know I could render pixels directly into a sysex-ready buffer by calculation, inline a bunch of calls to help Lua perform, etc. but I’ve already burned so many nights and weekends… even killed the whole concept of vector math in my Lua codebase for performance. I don’t want to bend too much further for Lua because frankly if I hate coding the project then it isn’t going to happen. Fortunately in Rust, everything is beautiful: I use vectors everywhere for free, render with standard image buffers, and convert to an optimally sized sysex buffer in … zero milliseconds!

And the native/Lua interop is so close to being usable… :slight_smile:

#15

Okay, I think I see (I was just curious). So to get 20 frames per second update you got to send about 23-24 kilobytes of sysex data per second to the controller. You say that that isn’t really the main bottleneck (per se), but the ‘bit twiddling’ image conversion is, taking dozens of milliseconds. Well a ‘dozen’ is 12, plural, so (I assume) upward of 50+ milliseconds to do the image conversion in lua(?), in which case you don’t have the bandwidth to update the AKAI screen at a constant 20 FPS.

Luckily if I owned a Akai Fire myself I wouldn’t be updating the screen at 20 FPS to do a smooth sample scrolling animation or FL studio type visualization on the controller screen. (But I can understand say demo sceners if they want to rotate a 3D 1-bitplane wireframe cube on the display…because it’s there :slight_smile: ) Good luck in your external C library for better integration with lua request to taktik.

#16

Good news – for me, @taktik , and I guess any other developer of high performance Renoise extensions: it is indeed possible to get native modules working if you static link a Lua implementation that’s “close enough” to what’s running within Renoise. Got mine running, solid as a rock so long as I steer clear of garbage collection and table allocations. It’s Lua v5.1.5 with minor hacks, in case anyone else is interested. :slight_smile:

#17

Pardon me, I still don’t fully understand this. I would like to use sqlite within a Renoise tool, but the usual approaches in LUA using sqlite require a compiled dynlib/dll inside the tool.

So first I would compile such a luasqlite dynlib for all the three platforms, then put those 3(?) dynlib/dll/so into my tool dir, and include that using a require?

Also I am quite a bit lost which lib would be the best choice. There seem to be a lot of very old sqlite-lua libs… Any tips for me?

Thanks for help :slightly_smiling_face:

P.S: SQL inside a tool would make a lot of neat tools possible, which would be a lot of more work without SQL. e.g. a plugin manager with tags, categories, ratings etc. There also is such stuff like “LuaRocks”, which makes it possible to install additional funcitonality to lua? Could luarocks be intergrated instead? LuaSQL: Database connectivity for the Lua programming language I guess that would also ensure system stability…

Or ulua? As it was suggested in my other thread by lowph, this indeed seems to be the most recent approach: GitHub - stepelu/lua-ljsqlite3: LJSQLite3: SQlite3 Interface

There also are LUA ORMs, like this one: GitHub - wesen1/LuaORM: An Object-Relational Mapping (ORM) for Lua based on 4DaysORM
It seems to be also based on LuaSQL, like the most recent other projects, too. EDIT: Ah, damn LuaORM does not seem to support sqlite yet. But it looks tasty.