R.3.5.3 How to keep a variable that outlives _AUTO_RELOAD_DEBUG

Renoise Tool Development
1

The next documentation describes the operation of _AUTO_RELOAD_DEBUG:

tool: globals

When working with Renoise’s Scripting Editor, saving a script will automatically reload the tool that belongs to the file. This way you can simply change your files and immediately see/test the changes within Renoise.

When working with an external text editor, you can enable the following debug
option somewhere in the tool’s main.lua file:

_AUTO_RELOAD_DEBUG = function()
  -- do tests like showing a dialog, prompts whatever, or simply do nothing
end

Now, as soon as you save your script outside of Renoise, and then focus Renoise
again (alt-tab to Renoise, for example), your script will instantly get reloaded
and the notifier is called.

If you don’t need a notifier function to be called each time the script reloads,
you can also simply set _AUTO_RELOAD_DEBUG = true to enable the automatic
reloading of your tool.

@type (true|fun())?

_AUTO_RELOAD_DEBUG = nil;

This means that all local or global in memory is destroyed, and the entire tool is subsequently reread.

  1. How would it be possible to preserve a variable (or table) that survives _AUTO_RELOAD_DEBUG and therefore we can access that value later?

  2. On the other hand, is there a function that allows us to completely restart a tool whenever we want, allowing us to preserve whatever we want somewhere (ideally temporarily in memory), be it a variable or a table?

The idea would be to be able to save a variable (or table) without having to create a file outside of the tool’s root folder.

For example, here’s a tricky thing I came up with: Song comments are permanent and can be manipulated from the API, without affecting song reloads of the tools (it is a data saved inside the song). I think it would be very useful to have a safe place where we could store permanent external data, outside of the tool’s root folder, to later retrieve those desired states. At least let them survive the open Renoise session.

1 Like
2

I’m still pressing this point and want to explain it better.

I suppose I’m asking for the API to give the programmer the ability to completely reset a window tool, but to allow us to preserve variables somewhere safe and untouchable (the variables that we can choose).

_AUTO_RELOAD_DEBUG resets the entire window tool, but it breaks all variables in the process (_G is destroyed, and the data in memory is destroyed).

This is especially useful if we want to change the tool’s GUI, for example, when we want to change the states of many objects by changing a single variable, in very complex windows. The simplest solution is to reset the window tool. Something like this would also be useful in other scenarios.

How do we do this? By enabling special access from the API that doesn’t affect the tool’s complete reload and is available from the start; the tool only needs to be loaded for this to work.

A hypothetical example:

renoise.tool().secure_data. Read and write access to variables or tables. As long as it cannot be altered and survives while a Renoise session remains open, that would be sufficient. I suppose this would work with the tool’s “id”.

3

Not sure I get what you want but the preferences for tools allow you to save state across different loads of your tool. As long as your state can be serialized using the Document API it can be saved.

4

I know, I often use the preferences.xml file in all my tools.

But this file can also be manually modified, or even accidentally deleted. This can cause you to lose all your settings.

I’m simply suggesting a secure memory access point for storing tables or variables that can’t be manipulated if auto_reload_debug is executed, or perhaps another future native feature that would allow for a manual (forced) complete tool restart (the same thing auto_reload_debug does automatically). Currently, we don’t have any way to restart a tool, as far as I know.

5

In what case would you want to restart a tool when neither “Reload All Tools”, nor rewriting the tool file is sufficient?

For the secure storage part, I don’t see why the preferences file is not a good solution. Being able to access the preferences.xml is an advantage I think, makes development easier and it means the user can even backup their preferences if they want to.

I assume you meant to use this for implementing some sort of DRM?

6

The preferences.xml file isn’t a safe place to store a table or a boolean because it can be manually edited or deleted, and when you run something like auto_reload_debug, everything is destroyed and reloaded, including the preferences.xml file if it’s deleted (I believe this didn’t work this way in previous versions). Restarting “all” the tools isn’t practical either, because it restarts them all.

The problem is that we don’t have a method to restart an individual tool while simultaneously maintaining a safe place in memory to store a boolean or a table, for example, with window sizes or something similar.

It’s a way to maintain control in case of modification. For example, you manually add Lua or XML files that are part of profiles or extra configurations offered by your tool. For this to work, the tool needs to be restarted somehow. For this to be secure, we need the option to securely preserve the “state of something” so that the user can’t modify anything that disrupts its proper functioning, at least during an open Renoise session.

If there were at least some way to detect which files are altered or deleted before running auto_reload_debug (or a future function that does the same thing when invoked manually), we could then decide what to do. An idle_observable could be used for this, but even then, we still need to be able to preserve the state of something (a boolean, or ideally a table) for later use.

I’m just pointing out that, to my knowledge, there’s no place in the API to preserve this kind of “secure” information for these cases, even to protect your tool, as you mentioned.

When we create simple tools, these issues don’t matter. But when they’re complex, especially multi-state window tools, it would be good to have these two things available:

  1. A function to restart the tool based on its ID (destroying everything, just like auto_reload_debug does).

  2. At the same time, a secure place in the API to safely store data (a table, or boolean variables, numbers, or strings), ideally in memory, readable and writable, that persists at least for the duration of the open Renoise session. When Renoise closes, this data is destroyed from memory.

In fact, I thought the API had something like tool_data where data could be safely stored. But I can’t get this to work.

Inject/fetch custom XRNX scripting tool data into the song. Can only be called
from scripts that are running in Renoise scripting tool bundles; attempts to
access the data from e.g. the scripting terminal will result in an error.
Returns nil when no data is present.

Each tool gets it’s own data slot in the song, which is resolved by the tool’s
bundle id, so this data is unique for every tool and persistent across tools
with the same bundle id (but possibly different versions).
If you want to store renoise.Document data in here, you can use the
renoise.Document’s ‘to_string’ and ‘from_string’ functions to serialize the data.
Alternatively, write your own serializers for your custom data.
@field tool_data string?

Could this be a “safe space”? I’ve never used it, but it seems you can save tool data in the xrns song file, which would be a great place for each individual tool.

Are there any concrete examples of how to work with this?

renoise.song().tool_data = nil in a new song. I suppose you have to start it somehow and then save the data to it. I don’t intend to store a lot of data there (like the entire preferences.xml file), but just a few pieces of data, strings, or tables…

7

The Document API is a bit confusing but you can use it to write and read a DocumentNode object to the Song.tool_data like this (might be a cleaner way but this works).

-- define a class inheriting from DocumentNode with the needed properties
MyToolData = {}
class "MyToolData" (renoise.Document.DocumentNode)
function MyToolData:__init()
  renoise.Document.DocumentNode.__init(self)
  self:add_properties {
    flag = false,
    int = 0,
  }
end

-- get the tool data from the song
local stringified = renoise.song().tool_data

if stringified == nil then
  print("the song's tool_data is empty")
else
  -- create an instance of our data class
  local data = MyToolData()
  -- parse the existing string we got using it
  local success, error = data:from_string(stringified)
  if success then
    -- we can now access all properties that we defined
    print("int: " .. data.int.value)
    print("flag: " .. tostring(data.flag.value))
  else
    -- the string couldn't be parsed as a valid MyToolData
    print(error)
  end
end

-- create our data document and change some fields
local new_data = MyToolData()
new_data.flag.value = true
new_data.int.value = 42

-- serialize it as a string for the tool data
renoise.song().tool_data = new_data:to_string()

-- -- set the tool_data to nil to clear it
-- renoise.song().tool_data = nil

Try putting this into a tool and reload it twice in the same song, at first load the tool_data should be empty, but later reloads should get you the data serialized at the end. Once loaded, handling the Document should be identical to using the tool preferences.

8

I’m keeping this documentation here in case anyone wants to use it. I think the cleanest approach is to define my_data directly:

local song = renoise.song()

-- Check initial state
print(song.tool_data) --> nil

-- To start and save data (use the memory)
if not song.tool_data then
  local my_data = renoise.Document.create("MyData") {
    count = 0,
    last_note = "",
    history = {}
  }
  -- Serialice a string before to assign (save the data)
  song.tool_data = my_data:to_string()
end

-- To read
-- Restore the data from song.tool_data
local restored_data = renoise.Document.from_string(song.tool_data)

-- Access to data
print(restored_data.count)      --> 0
print(restored_data.last_note)  --> ""

-- Modify existen data
restored_data.count = restored_data.count + 1
-- Save this modification serialized in tool_data
song.tool_data = restored_data:to_string()

-- Destroi all
song.tool_data = nil

All these procedures use memory until the song is saved. When the song is saved, this data is also saved in the xrns file, but during its creation, it remains in memory and can be changed or read (with fast access, no disk read/write).

It’s possible to destroy them before saving the song using an observable. However, it’s not possible to destroy them if the user disables the tool (this is a minor drawback).

At least this is the closest thing I was looking for. As far as I know, there isn’t an observable that fires right before a tool is disabled. If there were one, that would be great.

9

Note, your snippet will crash because there is no from_string on renoise.Document.

My example above is more verbose but it handles load and possible errors more explicitly.

10

There is, it’s renoise.tool().tool_will_unload_observable

11

Thanks @unless. All of this is a great help! I’ll review all of this and update that part of the documentation as soon as I’ve thoroughly tested it.

So it does seem like there’s a clean way to work with memory by manipulating tool_data and then cleaning it up, whether the tool is disabled or the song is saved. It seems like a somewhat hidden or convoluted trick, but if it works, nothing else is needed, at least for now.

The only thing missing would be the ability to invoke a function that resets an entire tool (in the destructive style of auto_reload_debug). That’s the only thing missing.

12

Here is the same example without using the class stuff.

-- define default fields for MyToolData DocumentNode
local data = renoise.Document.create("MyToolData") {
  flag = false,
  int = 0
}

-- get the tool data from the song
local stringified = renoise.song().tool_data

if stringified == nil then
  print("the song's tool_data is empty")
else
  -- parse the existing string we got using our data instance
  local success, error = data:from_string(stringified)
  if success then
    -- access the loaded properties
    print("int: " .. data.int.value)
    print("flag: " .. tostring(data.flag.value))
  else
    -- the string couldn't be parsed as a valid MyToolData
    -- data will have the default field values instead
    print(error)
  end
end

-- change some fields
data.flag.value = true
data.int.value = 42

-- serialize it as a string for the tool data
renoise.song().tool_data = data:to_string()

-- -- set the tool_data to nil to clear it
-- renoise.song().tool_data = nil

If you want to separate creating the data class and loading, you can use renoise.Document.instantiate("NameOfPreviouslyCreatedClass") and call from_string on that.

13

Ah, okay, that summarizes everything. For the cleaning process, we would need to add the two observables:

In case of disabling the tool: renoise.tool().tool_will_unload_observable

Invoked right before a tool gets unloaded: either because it got disabled,
reloaded or the application exists. You can cleanup resources or connections
to other devices here if necessary.

@field tool_will_unload_observable renoise.Document.Observable

In case of saving the song: renoise.tool().app_will_save_document_observable

Invoked just before the application document (song) is saved.
This is the last chance to make any changes that should be part of
the saved song. You could for example write your tool data to
`renoise.song().tool_data` here.

@field app_will_save_document_observable renoise.Document.Observable

This method would allow the memory to be used “safely” and cleanly to temporarily store (at least during an open Renoise session) unalterable variables that could be accidentally manipulated by the user (even saving that data in the song’s own xrns file).

I admit that with the recent changes to the API documentation, I’ve had to readjust to understand and, above all, find where everything is. Thanks for the help!

14

Cheers! Btw, I recommend trying out the lua language server if your editor supports it, it will warn you about things like the nonexistent Document.from_string before even having to run your code, and a lot more. It should make having to search the docs less needed in general.

15

Couldn’t you put a variable to the renoise-object simply? Like:

renoise.foreverLastingVariable = ‘verySecretTechInCamelCase’

Other tool:
print(renoise.foreverLastingVariable)

I didn’t test this BTW.

16

No, tools are isolated from each other (and their own context before reloading). The renoise object is not a shared mutable thing across tools, otherwise a tool could do stuff like reassigning renoise.song() and break basically every other tool. You can put a variable there and read it in the same tool but it will be invisible to other tools and it will be lost upon reloading the tool.

1 Like
17

Ah yes, makes sense. Thanks for the insight.

18

Can you write it out on a file? And read from the file? Thus you could also share the variable with other plug-ins or Instances?

Or maybe there is a way to use OSC services for that?

19

Yes. You can create a text file, or even a Lua file, in the main folder where all the tools are stored, for example (config_tools.lua, config_tools.txt, config_tools.xml, for example), and store variables or tables there if you want to share information.

This file could be used to store values ​​used by multiple tools, but it’s completely open and manually editable, and it wouldn’t affect tool reloading. It would be something like a common status query file.

However, we always aim for a closed environment and to isolate the tools as much as possible. _AUTO_RELOAD_DEBUG only affects the tool’s root folder and only analyzes the content of the Lua and XML files (I believe modifying a text file doesn’t affect it).

This file would also be copied if Renoise were updated, in addition to all the tools, since it is located within the general tools folder.

By the way, _AUTO_RELOAD_DEBUG is intended for the programming process and it only affects your individual tool. It’s not meant for use within an installed tool. That’s why I mentioned that tool developers don’t have any API function that can be manually triggered to restart a tool (useful for very complex window-based tools).

Something like: Okay, you’ve just changed a lot of things in the window’s GUI; it’s better to completely reset it than to make a ton of changes by calling objects by their IDs to transform it. This is just one specific case; it could apply to other things as well.

20

For this case specifically, I think it is simpler to just close the dialog and reopen it with code, or have a function that purges all views and rebuilds the contents of the root view inside the dialog.

Similarly, implementing a general reset for your tool shouldn’t be too hard if you put all your initalization code inside a single init function that you can call again when needed. The only part that typically needs some extra attention is removing/reattaching all your notifiers.