New Tool (3.0,3.1): xStream

Renoise Tools
121

What you say - could work with xstream, or should I start trying to make my own tool for this?

xStream is primarily designed for realtime use, the ability to transform/write to the pattern is done in a linear fashion, with time as the driving factor.

But from what you describe, it sounds like you’re looking for something that is more like sitting in the background, quietly monitoring any part of the pattern and acting on specific changes to the pattern data.

This kind of thing is actually not that to achieve with “plain” lua scripting, and you would probably end up with something which is more focused and practical to use.

I would advice you to check out the line notifier, that would be the key for unlocking this kind of workflow.

For example:

  1. Tool starts, it scans the pattern to find a suitable “template” track (any sequencer track, basically)
  2. If a suitable candidate is found, it somehow needs to resolve a target track (?)
  3. If both source and target can be found, it attaches line notifiers and starts to monitor changes.
  4. Repeat step 1 every time you navigate to a new pattern (don’t forget to detach existing line notifiers)
122

Ok thanks you’re right. I see now the purpose of xstream being more kind of interactive/realtime event data processor for transformations of what is currently acting, kinda like a programmable uber-mutant-arpeggiator that will also have its action recorded as its happening…

I’ll look into trying to build a groove tool to my liking. I think with the Ideas I have in mind so far it would be better to do this from ground up anyways, so automation and also phrases can be taken into account at some stage.

Yes, you’re right, aside from the groove/straight toggle and some config dialogs it should mainly sit in the background and just automatically act on the data a composer is working with…or alternatively also be a magic box you trigger onto a song with some settings, and transform the whole song’s timings at once when you hit the “run” button. The renoise observer system is of course a nice twist to building the realtime automatic processing, I’ve just read into it & just hope performance won’t kill this, because it would de facto have to permanently observe and act on all the pattern data in song.

123

programmable uber-mutant-arpeggiator

Yes, the one arpeggiator to end all arpeggiators :smiley:

Re performance, I just want to add a few words of advice: conceptually, line notifiers are straight-forward, but in the detail, they can be a bit tricky. What I mean by this is, that you will absolutely need to receive the notifications but act on them with a slight delay. Because, line notifications are fired not just when you (the user) does something like enter a note, they are triggered on any change to the pattern data. E.g. clearing a pattern will result in a line notifier for every single line === potentially 100s of notifications.

So the implementation is basically this: observe the lines, but debounce any action you take, e.g. by raising a flag and handling it a split-second later. Using the tools idle notifier is my preferred method for such scheduled updates. And even then, you probably want to make the function that handles such updates as efficient as possible. But by eliminating the most obvious bottlenecks, the performance should be pretty good.

Here’s a relevant quote about writing optimized code (from Lua programming gems, chapter 2):

Rule #1: Don’t do it.
Rule #2: Don’t do it yet. (for experts only)

And I’d like to add: do it - but don’t optimize it until you clearly understand what’s going on B)

124

I have pretty much done what Oopslfly is talking about, but a bit more elaborate - passing each line notifier event thru a sandboxed function.

Be aware that the biggest obstacle when it comes to this kind of operation (track1+track2=track3), is that chances are very low that you’ll get it working with pattern aliases. So the tool will never be perfect. This is for various design reasons (and bugs, I think) in the API. At least, that’s my experience…

125

chances are very low that you’ll get it working with pattern aliases. So the tool will never be perfect

Don’t forget thatmatrix slots can be aliased as well. It’s all very tricky in the detail, but possible.

Don’t know what bugs you are referring to?

126

There are several issues regarding pattern aliases. I’ve tried to explain it in a previous thread, but it’s a bit difficult.https://forum.renoise.com/t/nb-worked-around-slight-overhaul-of-alias-pattern-index-observable/46723

Short story:

  1. Keeping track of all aliases is generally quite tricky (but possible).

2)alias_pattern_index_observable will bang when a sequence is removed. This should be a bug, or at the very least a design flaw.

  1. alias_pattern_index_obserfvable will bang when a sequence is inserted. So, yet more work-around logic is needed to circumvent this, if it’s even possible.

Eventually, you’ll find that you have to use lots of workarounds and complicated schemes. Everything would have been solved with some simple change in the API itself. So… the motivation you’re left with after trying, is only enough for writing a crappy forum post about it :slight_smile:

127

I’ve tried to explain it in a previous thread

Hey man, thanks. That’s a constructive post you made there…good points.

You should’ve expected at least some kind of response for that. For me, it somehow slipped under the radar.

And while I’m personally not afraid to implement workarounds, I agree that anything that can’t be done cleanly/properly needs a critical examination.

128

Oh, it’s not a bug, it’s a quirk then - if with clever filtering and double checking every relevant event can be collected somehow, then things are finde with me. Ofc I can understand its problematic if you don’t know the whereabouts, and also think that a proper api that don’t need quirkyness workarounds is the best, but well, in my experience almost any complex software api has such strange behaviour somewhere in its guts.

Hm, so I thought I’d first do the processing code until it works fine. It would also set me going as I could compose a tune straight or with some native groove, and then experiment with grooves, single-run applying them on command. This would also be useful for old (quasi finished) tunes, that are to get some a posteriori ants into their pants…

Then I thought about an event catcher func that is as slim as ever possible, just filtering/collecting relevant events to some stack or so. Then a processing routine that goes in the background that will apply the action - is the idle notifier background threaded, or would I have to split the workload in packets for the whole thing not to block the engine at any point more than absolutely nessecary? Is it generally advisable to put processing out of the notifier handlers and schedule into idle routines instead?

I mean also, the nature of the task I have in mind is very simple, and with the right concept it could work out well. I don’t know how robust I would want to make this. For example the effected copy of a track is never to be touched by the user, I’d for example go for assuming the user not to touch it and periodically and by demand overwrite whatever crap happened to the track just to make sure. Maybe the processing also can be kept rather simple, by for example always processing the whole patterntrack on a change instead of trying to weed out just single notes that were changed. I will also try looking for nudge/shift tools, or similar, to lend some note overlap handling for shifting notes to (new) columns. My main concern on the live grooving approach is to keep it, for the user, as close to a normal workflow as possible, I have some Ideas going on about this. But some concepts will imply my own preffered workflow I think, or also have to add quite some visible baggage to the project in order for it to work nicely.

Maybe you want to split the discuss to another thread as its not related to your tool any more. I also have to say there will be some time passing until I can try working on this. Might have to wait for nothern hemisphere winter… I am no seasoned programmer, just an enthusiast, and I have other things to waste my time with, and want to get warm with another tool that aims accelerating sample creation/editing/processing workflow with the convolver first.

129

In this case, using an idle notifier/process slicer is not necessary if you have a fairly fast processor IMO. I’d say go for a simple approach - you can always add such things later on if really needed. It’s more sensible to first focus on optimizing any .song() access, as this often will suffice.

130

Then I thought about an event catcher func that is as slim as ever possible, just filtering/collecting relevant events to some stack or so. Then a processing routine that goes in the background that will apply the action - is the idle notifier background threaded, or would I have to split the workload in packets for the whole thing not to block the engine at any point more than absolutely nessecary? Is it generally advisable to put processing out of the notifier handlers and schedule into idle routines instead?

When you’re processing a whole song, or basically anything that takes enough time that a script timeout could happen (a dozen seconds or so), you can “slice” your process.
This is similar to a thread (co-routine), and really isn’t that hard to implement. It’s just a matter of creating a self-contained method that does the processing and periodically hands over time to the main process via a yield(). Here is a working example:
https://github.com/renoise/xrnx/tree/master/Tools/com.renoise.ExampleToolSlicedProcess.xrnx

Maybe you want to split the discuss to another thread as its not related to your tool any more. I also have to say there will be some time passing until I can try working on this. Might have to wait for nothern hemisphere winter… I am no seasoned programmer, just an enthusiast, and I have other things to waste my time with, and want to get warm with another tool that aims accelerating sample creation/editing/processing workflow with the convolver first.

You’ll probably want to create a topic once the tool takes shape, but having it here for now is fine with me :slight_smile:

131

Hello danoise and all the guys around this wonderful topic.

Many thanks for this great tool!

Recently i tried an updated version and now i can’t use it because of the error (OSX). Please see attached screenshot.

Can you help with this?

132

Hello danoise and all the guys around this wonderful topic.

Many thanks for this great tool!

Recently i tried an updated version and now i can’t use it because of the error (OSX). Please see attached screenshot.

Can you help with this?

Yeah, this bug happens for an unknown reason.

Still, there is a known fix/workaround here.

https://forum.renoise.com/t/new-tool-3-0-3-1-xstream/44430

133

I’m taking a crack at this again after I saw the updated documentation. I tried a while ago and gave up cause I couldn’t find enough documentation, so I really appreciate that update!

So, I’m trying to open my previously saved models, but I can’t.

I have a model on my desktop named “mymodel.lua” and is exactly the same as the “Untitled model.lua” model. When I try to open it using

(models toolbar >) create new model > Locate a file on disk > mymodel.lua

a “Scripting Tool Error” window pops up with the message:

‘C:\Users\Owner\AppData\Roaming\Renoise\V3.1.0\Scripts\Tools\com.renoise.xStream.xrnx\main.lua’ failed in one of its notifiers.

Please contact the author (danoise [bjorn.nesby@gmail.com]) for assistance…

.\source/xStreamModel.lua:34: Wrong type of parameter

stack traceback:

[C]: in function ‘assert’

.\source/xStreamModel.lua:34: in function <.\source/xStreamModel.lua:31>

[C]: in function ‘xStreamModel’

.\source/xStreamUIModelCreate.lua:292: in function ‘navigate_to_model’

.\source/xStreamUIModelCreate.lua:239: in function ‘show_next_page’

.\source/xStreamUIModelCreate.lua:143: in function <.\source/xStreamUIModelCreate.lua:142>

I get the same thing when I copy and paste the file’s contents using the “Paste from clipboard” option.I also tried uninstalling xStream, closing Renoise, and reinstalling.

This also happens when I try opening up one of the example models using those methods I tried above.

Any ideas? Am I using “Create new model” correctly? Does this have anything to do with the location of the Userdata folder?

134

Also, how do you show line numbers in the xStream script editor?

135

Is there any way to include libraries? Do I just put a simple #include “lib.h” at the top of the code?

136

Hey, and thanks for the feedback.

When you’re having trouble with such a basic task (creating a model), I would have to ask about whether you’re using a custom userdata folder (since you bring this up - it’s set from the xStream options dialog), and possibly what OS you’re using?

What you are trying to do really shouldn’t cause such problems.

That said, the current version of the tool is not completely without bugs. Who knows, you might have encountered another one.

Also, how do you show line numbers in the xStream script editor?

There is a number of features I’d like to include, but that can’t be done without improving the Renoise API itself. Having line numbers is one such feature.

However, I have been toying with an idea which would enable you to edit xstream files outside Renoise. I think this is actually a much more convenient thing, because no matter how much how we tweak the text editor in Renoise, most people who are dabbling with this tool probably have a text editor that they prefer anyway. So I see no reason why xStream shouldn’t benefit from that. Hell, you could even use the Renoise scripting console then, if you wanted to (I know some people do ^_^)

Is there any way to include libraries? Do I just put a simple #include “lib.h” at the top of the code?

Everything that gets included with xStream is specified in the main.lua But - that doesn’t mean that you can just starting referencing functions and method from within the xStream “main method” (the processing method that transforms pattern data, not to be confused with the “main.lua” file, which is simply the entry point for any renoise tool.

This is so, because xStream wraps the main method in a lua sandbox - which makes certain variables (e.g. the all-important xinc) available to the model, while also hiding potentially ‘dangerous’ things like file system access.

In order to benefit from method in your own files, you need to explicitly define sandbox accessors to your own properties and methods. If it sounds complicated, it’s not - just a question of opening and modifying the file called xStreamModel.lua.

Hope this helps!

137

Hi

Say I have a sequence in Track 1, that I want to simply transpose the note randomly between + {0, 12, 24}, so the original value is read from the src track, then modified and written to Track 2, how would I do that?

The problem is that the original note value is being modified and it becomes the base note for the next round of modifying…I want to refer to the original note value all the time.

Is it possible to read from one track and apply a modified result to another, or do I have to approach this differently?

thanks

138

The problem is that the original note value is being modified and it becomes the base note for the next round of modifying…I want to refer to the original note value all the time.

Ah, so you would essentially be looping over the same notes over and over again?

Yes, then the result would gradually slide further and further away from the original state.

There is no problem in writing notes to a different track though, you should have the necessary information already -

for example, using xpos and xline together, you should be able to call a method like xline:do_write (documentation).

The only thing to be aware of is that xStream does not manage this “shadowtrack”,

The next version of xStream will indeed feature a workflow that integrates track routings, but for now you will have to that yourself.

139

How do I use the do_write method?

Please could you show me an model example of reading a src track (1) and writing the modified data destination trk 2?

for k = 1,rns.tracks[track_index].visible_note_columns do

if (xline.note_columns[k].note_value < 120) then

local note = xline.note_columns[k].note_value

xline:do_write(xpos.seq, xpos.line, 2) _ <-- if this is pointing to the right track, where do I plug the note data in? _

end

end

Thanks

140

The note data is specified by the xline itself. So you could basically take the incoming xline, make a copy of it, modify the properties and then write it to your target track.

If you didn’t make a copy then the modification you make would also be written to the source track - not what you want in this case.

A copy can be made simply by creating an xLine using another xline as argument.

I can make a quick test, let you see how it’s done.