New Tool (3.0,3.1): xStream

[danoise]

I just committed my stuff. It more or less wraps up the merged buffer implementation.

Btw: I found what seems to a typo in the xLine read method you contributed. Not sure if you’re using the code elsewhere, so I wanted to point it out:
https://github.com/renoise/xLib/commit/44b1fd5765da763a65dea3365ac8cc1fd00b20a5#diff-c4c998a9605b5b8a705d6f73f70f4165R303

I also did away with the variables pointing to functions. It’s my impression that there is no overhead when calling functions that are properties of an object.
But again, maybe you had some reason to make this change? If not, I think it just makes the code slightly harder to read.

This is one of those times I wish there was a real way to trace function calls backwards in some IDE. The whole xStream code (like any larger project) isn’t very easy to penetrate and get a full birds-eye perspective on.

Yep, manual tracing goes some of the way (you can identify often-called functions in a logging tool). But it has downsides - maintenance, mostly - and isn’t proper debugging either.
Note: I usually strip away the TRACE statements once a tool goes on tools.renoise.com, so there’s no a performance penalty to speak of

[joule]

It doesn’t make a big difference localizing these things, no. I just do it on things that are very frequently accessed since it’s easy and it’s faster. It seems each table/object depth adds the same time it takes to call a same-scope variable (n_time*depth). I mean, if optimizing the most frequently used functions would make the software 25% less hungry, why not

One point I’d like to stress though, that could probably be very easy to fix, is to reuse xline (and args if that is an object?) instead of creating new instances on every line. That’s an object (tables) creation (and lots of init+gc stuff) that can be avoided by reusing their “skeleton” with some simple :clear() function. E g. create if xinc == 0 or xline:clear()

[danoise]

Ah, yes, the “re-usable pool of objects” approach.
This works well when things are predictable, but it’s made more complex in xStream by the fact that you can do stuff like this:

xline.note_columns[1] = {
 note_string = "C-4"
}

You’re basically allowed to redefine xline properties (note/effect columns), which are then instantiated as classes during execution (with default values for panning, etc. being added on the fly).

But you’re right - the xline itself could indeed be turned into a resource pool. Something to think about!

I also think I know what you’re looking for here: some “middle road” solution with a shared codebase for “xStream light” and then the full version?

[joule]

I’m actually just thinking what would happen if 8 tracks are rendered simultaneously, with something like remapped chord voices via midi. “Would it be possible with just one line write-ahead?”

I’m not sure I get the dilemma of your example, but maybe you’re right. I thought things that must be “initialized” could be so on :clear(), that it should in any case be easy to re-generate the object into something identical to a normal init. Maybe you mean that you have some clever way of ‘copying’ a mother-xline into temporary children-xline, I’m not sure.

Anyway, fun fact that might be relevant for this topic: when clearing a fixed table, it’s faster to set individual values to nil if they are fewer than 8, compared to setting the table to a new { }

PS. Let us all know when/if we should download the fresh stuff!

[danoise]

8 tracks are rendered simultaneously, with something like remapped chord voices via midi

Ah, OK. Yeah, it’s possible to write directly to other tracks (you even did a proof of concept, didn’t you?). I rarely would need this, I would be much more likely to have different models running side by side.

Also related to this topic + performance - in many cases, the fact that xStream reads pattern data is also a waste of CPU cycles.

It’s just that I haven’t realized how to make this configurable in an easy to understand manner (write-only models? just adds to the scare factor IMO)

Stacking, on the other hand, is simple enough as the concept implies that you are reading pattern data - from the track OR passed down from a different model.

Hm, now I opened the topic of stacking but I don’t really feel like discussion that right now.

Instead I’ll spend some time tonight taking the current version for a good spin

[joule]

Also related to this topic + performance - in many cases, the fact that xStream reads pattern data is also a waste of CPU cycles.
It’s just that I haven’t realized how to make this configurable in an easy to understand manner (write-only models? just adds to the scare factor IMO)
Stacking, on the other hand, is simple enough as the concept implies that you are reading pattern data - from the track OR passed down from a different model.

If you’d change xLinePattern to somehow work on a need-to-read basis (using __index metamethod), the current (and fast) line string reading would be superfluous in those cases. So it should be avoided then…

Hmm. this might be an idea:

1. Only generate the env.xline when/if it is first accessed in the sandbox. This should be simple with an if statement inside the sandbox environments __index metamethod. The default value of env.xline would be just “false”.
2. Store the actual renoise.PatternLine into some key in the sandbox environment, as an alternative for write-only models to access. Is this really too scary?

This seems like quite an easy fix, but I haven’t looked at where you update the sandbox environment.

Click to view contents

Something like:

env.mt.__index = function(t, k)
  if k == "xline" then
   if not env[k] then
   env[k] = 
   end
   return env[k]
  end
  -- all the other stuff
end

And then, of course, just clear that key for every new line

You might have to do the similar thing with __newindex to cover all uses. I’m not entirely sure.

About generating the xline there, I understand that you probably need access to some objects that aren’t within the scope (including the current renoise.Patternline). The dirty way is to use some global variable(s), but IMO the tidy way would be to always pass the “parent class” (self) downwards via args into a property when creating new objects. This, kind of, makes everything accessible from anywhere.

EDIT:

Or you could implement the approach I’m suggesting (need-to-read basis) on the note_columns and effect_column getter/setter properties inside the xLine class? Let self.pattern_line be nil until someone tries to access it. That might be much simpler than messing about in the sandbox environment. Unless you’re aiming for really optimizing everything and avoid any unnecessary object creations.

Click to view contents

Principle:

...
self.pattern_line = nil
...
function xLine:get_note_columns()
 self.pattern_line = self.pattern_line or xLinePattern(
 self.args.note_columns,
 self.args.effect_columns
 )
 return self.pattern_line.note_columns
end

[danoise]

Yes, many roads lead to Rome here.

When I mentioned scary, what I meant was that I didn’t want to push this over to the person using xStream - having to explicitly set a model as “write only” (WTF?)

So yeah, anything that gets done needs to work automatically and transparently.

I just added MIDI device hot-plugging, btw. Nice and easy addition to the xMidiIO class.

[joule]

Nice to see some development! It’s damn hard to penetrate the code though (follow the ‘stream’ of data and understanding all the ‘detours’). I’ve pretty much given up for the time being.

Some simple flow chart of all classes would be highly appreciated

[danoise]

At long last, a new release of xStream
http://www.renoise.com/tools/xstream

This update contains both the workaround for the “preferences is nil” error that has plagued people since ages,
but is also a pretty thoroughly tested general update with a lot of enhancements under the hood.

Enjoy, and keep me posted if you encounter any problems with it!

## 1.6 changelog
- Core: update cLib/vLib/xLib classes 
- Core: using xStreamBuffer 
- Added: support MIDI device hot-plugging
- Changed: using popup to choose render mode (global toolbar)
- Changed: updated / cleaned up models + documentation 
- Fixed: unable to start tool (preferences is nil) (#102)
- Fixed: error when editing argument, applying new settings (#103)
- Fixed: when renaming model, watch out for certain characters (#104)
- Fixed: renaming userdata should update other userdata (#109)
- Fixed: error when adding preset via favorite dialog bug (#114)

took the time to implement this suggestion as well:

I think the xStream GUI has quite an intense scare factor to new users, with all its custom buttons and symbols.

xgui.gif

[Jonas (Djeroek)]

^ I don’t see any model presets in the latest update, am I doing it wrong?

[danoise]

My guess is that you’ve configured the tool to user some custom userdata folder at some earlier point?

Try going into Options > General > Userdata, press “Reset” and see if this helps.

[joule]

I have a very small suggestion about improvements to dialog titles. Quite a few of them only has “xStream” as a title. It would be good if these had some explanatory window titles, further shaving off some of the scare factor. (The utopia being that you can explore and learn a software without reading the manual). Unless you really know how xStream is laid out, some of the windows popping up will leave you clueless.

[Jonas (Djeroek)]

Try going into Options > General > Userdata, press “Reset” and see if this helps.

That did the trick. Got this notice when trying to run the granular model with no instrument loaded:

"‘C:\Users\pluge\AppData\Roaming\Renoise\V3.1.1\Scripts\Tools\com.renoise.xStream.xrnx\main.lua’ failed in one of its notifiers.

The notifier will be disabled to prevent further errors.

Please contact the author (danoise [bjorn.nesby@gmail.com]) for assistance…

.\source/xLib/classes/xEffectColumn.lua:243: attempt to perform arithmetic on a nil value

stack traceback:

.\source/xLib/classes/xEffectColumn.lua:243: in function ‘number_string_to_value’

.\source/xLib/classes/xLinePattern.lua:307: in function ‘do_read’

.\source/xLib/classes/xLine.lua:271: in function ‘do_read’

.\source/xLib/classes/xStreamBuffer.lua:276: in function ‘_obtain_and_store_input’

.\source/xLib/classes/xStreamBuffer.lua:527: in function ‘_read_input_buffer’

.\source/xLib/classes/xStreamBuffer.lua:446: in function ‘_create_content’

.\source/xLib/classes/xStreamBuffer.lua:602: in function ‘write_output’

.\source/xStreamProcess.lua:278: in function ‘output’

.\source/xStream.lua:197: in function <.\source/xStream.lua:196>

[C]: in function ‘bang’

.\source/xLib/classes/xStreamPos.lua:117: in function ‘start’

.\source/xStream.lua:255: in function ‘start’

.\source/xStream.lua:265: in function ‘start_and_play’

.\source/xStreamUIGlobalToolbar.lua:127: in function <.\source/xStreamUIGlobalToolbar.lua:126>"

[Jonas (Djeroek)]

Terminating script gives:

"‘C:\Users\pluge\AppData\Roaming\Renoise\V3.1.1\Scripts\Tools\com.renoise.xStream.xrnx\main.lua’ failed in one of its notifiers.

Please contact the author (danoise [bjorn.nesby@gmail.com]) for assistance…

.\source/xLib/classes/xPlayPos.lua:65: Script execution terminated by user.

stack traceback:

[string “do…”]:37: in function <[string “do…”]:35>

.\source/xLib/classes/xPlayPos.lua:65: in function <.\source/xLib/classes/xPlayPos.lua:43>

[C]: in function ‘xPlayPos’

.\source/xLib/classes/xStreamPos.lua:156: in function ‘reset’

.\source/xLib/classes/xStreamPos.lua:118: in function ‘start’

.\source/xStream.lua:255: in function ‘start’

.\source/xStream.lua:265: in function ‘start_and_play’

.\source/xStreamUIGlobalToolbar.lua:127: in function <.\source/xStreamUIGlobalToolbar.lua:126>"

[danoise]

Thanks djeroek - keep 'em coming!

Btw:

Terminating script gives:

What do you mean by “terminating” - How did the error occur?

[Jonas (Djeroek)]

I mean when processing is taking to long and you get a notice asking if you want to abort and terminate the script or keep waiting.

[danoise]

Ah, in this case you will always get a notice of some kind. I don’t think I can prevent that.

[kingdoobie]

I can’t seem to get this going, as a test with arpeggiator.
Blank instrument track - > Midi Out - IAC Bus
XStream options → Midi In → IAC Bus, Midi Out Renoise In
Instrument → Midi In → Renoise In
Renoise prefs → Midi input → none

I keep getting a feedback loop and it goes up to g9 repeating notes? any help ?

EDIT: I found the GitHub manual, ill take a read!

[Jonas (Djeroek)]

@danoise I was playing around with xstream today and when pressing the M icon in the top of the tool gui, I got the following error notice;

"‘C:\Users\pluge\AppData\Roaming\Renoise\V3.4.3\Scripts\Tools\com.renoise.xStream.xrnx\main.lua’ failed in one of its notifiers.

Please contact the author (danoise [bjorn.nesby@gmail.com]) for assistance…

.\source/xLib/classes/xStreamBuffer.lua:384: attempt to perform arithmetic on field ‘playing’ (a boolean value)
stack traceback:
.\source/xLib/classes/xStreamBuffer.lua:384: in function ‘_get_immediate_xinc’
.\source/xLib/classes/xStreamBuffer.lua:409: in function ‘mute’
.\source/xStreamProcess.lua:289: in function ‘maintain_buffer_mute_state’
.\source/xStreamProcess.lua:153: in function <.\source/xStreamProcess.lua:151>
[C]: in function ‘__newindex’
[string “do…”]:22: in function <[string “do…”]:9>
.\source/xStreamUIGlobalToolbar.lua:164: in function <.\source/xStreamUIGlobalToolbar.lua:163>"

This is the model & settings used;

Btw crazy deep (underrated?) tool, I like how it can automatically draw in graphical animation while playing back the patterns with the lfo model. Curious to how this project started. Feels like something for an electro acoustic university project .

[danoise]

@Jonas

when pressing the M icon in the top of the tool gui, I got the following error

I’m sure there are any number of bugs lurking in this tool
But indeed, I should take a look - particularly if it’s so easy to trigger the error.

@Jonas

Curious to how this project started

The idea came to me when I expanded Conner_Bw’s Grid Pie. I realized that generating entire patterns would mean that Grid Pie would never be fit for live use - precalculating massive amounts of pattern data could always lead to a timeout at any given moment. But it would be possible to realize through a tool that could “stream” those data - building the road ahead as you were driving on it, so to speak.
Of course, I then proceeded to create tool so complex that it has some issues of its own ¯_(ツ)_/¯