Jump to content


Photo

New Tool (3.0,3.1): xStream

live coding sandbox

  • Please log in to reply
118 replies to this topic

#101 d.vyd

d.vyd

    New Member

  • Normal Members
  • Pip
  • 4 posts

Posted 04 April 2017 - 14:44

Hi Danoise, thank you for checking in. I did get MIDI to work with your examples. But, I think the learning curve for Renoise, the tracker paradigm in general, and xStream together is too much for me right now. Instead, I've made progress creating a sequencer that works with my hardware controller in Pure Data. I'm sending the output to my DAW (MuLab) to generate audio. 


  • danoise likes this

#102 kopias

kopias

    Super Advanced Member

  • Normal Members
  • PipPipPipPip
  • 154 posts
  • Gender:Male
  • Location:Łask, Poland

Posted 10 April 2017 - 22:27

From a workflow standpoint I imagine this tool working in a phrase mode.

 

Imaginary usage example1: The Tool is "bind" to current instrument. I get a one phase as input and output is another phrase. Settings like phrase in/out and maybe what "program" is run is stored inside instrument. Changing instrument deactivates the "program" execution and reads those settings.

 

I`m writing this as a encouragement and try to do the actual work. Hopefully this makes sense and be continued.


Edited by kopias, 10 April 2017 - 22:36.


#103 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6250 posts
  • Gender:Male
  • Interests:wildlife + urban trekking

Posted 24 May 2017 - 15:22

██╗  ██╗███████╗████████╗██████╗ ███████╗ █████╗ ███╗   ███╗
╚██╗██╔╝██╔════╝╚══██╔══╝██╔══██╗██╔════╝██╔══██╗████╗ ████║
 ╚███╔╝ ███████╗   ██║   ██████╔╝█████╗  ███████║██╔████╔██║
 ██╔██╗ ╚════██║   ██║   ██╔══██╗██╔══╝  ██╔══██║██║╚██╔╝██║
██╔╝ ██╗███████║   ██║   ██║  ██║███████╗██║  ██║██║ ╚═╝ ██║
╚═╝  ╚═╝╚══════╝   ╚═╝   ╚═╝  ╚═╝╚══════╝╚═╝  ╚═╝╚═╝     ╚═╝ v1.55

 
xStream is back!
 
Since the last version, which I released almost a year ago, I've used this tool - but not as much as I've wanted to.
Too often, the party got spoiled by a number of annoying, recurring issues - all of which should now be gone with this release (well, hopefully).
 
When it comes to this, I'd like to thank to pat for reporting everything he came across in just a couple of days of xStreaming  ;)  
 
Also, there was some much-needed internal reorganization that needed to happen. Joule and others has made it clear how multi-track operation is the future for this tool. 
And so, considering how much extra work this would involve - also resulted in xStream development being put on the back burner for a while. 
 
But now, the tool is back, and I dare say, better than ever. It's pretty much the same featureset as the previous release, but with a more solid foundation. 
 

Download from the tool page

http://www.renoise.com/tools/xstream

 

As the featureset has stabilized since that "sprint" of last year, I've also had time to update the documentation.
It's now spread across topics, with a better introduction to xStream coding than before, and generally just better organized.
 
You can check out the new documentation here (on github). Some of the pictures are a little outdated, I'm working on fixing that.
https://github.com/r...se.xStream.xrnx
 
Changelog for this release:

- Core: refactored several internal classes
- Core: more solid, simpler streaming implementation
- Fixed: loading favorites.xml was broken
- Fixed: selecting [no argument] would throw an error
- Fixed: table constants (e.g. EMPTY_XLINE) are now returned as a copy
- Fixed: read-only value arguments are not MIDI assignable
- Fixed: error when trying to create argument with just one "item"
- Fixed: failure to export presets when arguments are tabbed
- Fixed: setting custom userdata folder is now applied immediately
- Added: ability to migrate userdata to a custom folder
- Added: xLFO class + demonstration model
- Added: RandomScale model 
- Added: Updated documentation 

  • Djeroek, joule and pat like this

Tracking with Stuff. API wishlist | Soundcloud


#104 joule

joule

    Guruh Motha Fakka is Levitating and Knows Everything About Renoise Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • 1382 posts
  • Gender:Not Telling
  • Location:Sweden
  • Interests:music, philosophy, engineering

Posted 25 May 2017 - 07:59

So great to see an update! I'm trying the Euclidean Rhythms 1.2 model and it's awesome :)

 

I really like the preset system. With this model, I notice that it would be helpful being able to copy/paste the individual abcde tabs. I guess these are adhoc for the Euclidean model, and that it would be difficult to achieve. How are they even defined? (by a "tabs" variable in the sandbox?)



#105 Djeroek

Djeroek

    Probably More God or Borg Than Human Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6684 posts
  • Gender:Male
  • Location:Borneo

Posted 25 May 2017 - 08:16

Was switching models on the fly while running through patterns in playbackmode, also with 'auto-clone patterns' enabled and got this;

 

 

'C:\Users\pluge\AppData\Roaming\Renoise\V3.1.0\Scripts\Tools\com.renoise.xStream.xrnx\main.lua' failed in one of its notifiers.

The notifier will be disabled to prevent further errors.
 
Please contact the author (danoise [bjorn.nesby@gmail.com]) for assistance...
 
std::logic_error: 'ViewBuilder: invalid index for popup: '0'. value must be [1 - 4].'
stack traceback:
  [C]: in function 'popup'
  .\source/xStreamUIArgsPanel.lua:348: in function 'build_args'
  .\source/xStreamUI.lua:1479: in function 'on_idle'
  .\source/xStream.lua:352: in function 'on_idle'
  .\source/xStream.lua:202: in function <.\source/xStream.lua:201>


#106 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6250 posts
  • Gender:Male
  • Interests:wildlife + urban trekking

Posted 25 May 2017 - 11:26

How are they even defined? (by a "tabs" variable in the sandbox?)

 

Haha, guess it's time to check the documentation: 

 

 

When there are too many arguments to fit on the screen, you can organize them in a tabbed interface, simply by prefixing the name with the tab name.

For example, "voice1.volume" and "voice2.volume" will create two tabs, labelled voice1 and voice2, and add a volume inside each one.

 

Behind the scenes, the tab name and argument name is indeed kept separate. But from the main method, such arguments can be accessed as a table structure. 

-- accessing a tab with variable name
local tab_name = "voice1"
print(args[tab_name].pulses)

Especially for something like the euclidean model, this has simplified things a lot. 

 

And you're right, presets are dealing with the entire model. At least, that's true for newly saved presets - because, if you add additional arguments to a model, these will only be saved along with a newly made preset - if you load an older preset, it will also work, but obviously not be able to update the newly added arguments (the scripting console even tells you if this is the case). 

 

So, let's say that you were to create a special preset manually, by editing the model .lua file... then you could load that preset and it would apply only the specified values. Of course, this is far from an optimal solution, but it should work... 

 

Was switching models on the fly while running through patterns in playbackmode, also with 'auto-clone patterns' enabled and got this;

 

Pretty sure the auto-clone tool isn't to blame   :)

I can't replicate it here ... perhaps you remember which model that caused the error? Or able to repeat it? 

 

Edit: OK, I believe "ChordMemory" is the culprit, as it reads directly from the pattern and updates arguments. 


Edited by danoise, 25 May 2017 - 15:46.

  • Djeroek and joule like this

Tracking with Stuff. API wishlist | Soundcloud


#107 joule

joule

    Guruh Motha Fakka is Levitating and Knows Everything About Renoise Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • 1382 posts
  • Gender:Not Telling
  • Location:Sweden
  • Interests:music, philosophy, engineering

Posted 25 May 2017 - 12:04


And you're right, presets are dealing with the entire model. At least, that's true for newly saved presets - because, if you add additional arguments to a model, these will only be saved along with a newly made preset - if you load an older preset, it will also work, but obviously not be able to update the newly added arguments (the scripting console even tells you if this is the case).

 

Thanks for clarifying! I guess what I'm waiting for then is stacked models, and then handling my need by stacking single-layer euclideans. It's the combination of trial'n'error + preset system that I'm after.



#108 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6250 posts
  • Gender:Male
  • Interests:wildlife + urban trekking

Posted 27 May 2017 - 20:41

Already, I just updated the tool page with v1.57.
- Fixed: error when updating view with out-of-range value #99
- Fixed: error when trying to select data from editor popup #100
- Fixed: error is thrown when entering "return" into main method #97
- Fixed: model doesn't work when last line is comment #95
- Fixed: expecting "models" to be present in custom user-folder #87
- Fixed: favorite icons and preset highlighting (got broken in 1.55)
- Fixed: ChordMemory model had a few flaws
- Added: "Apply to Line", for quick single-line output
- Added: additional keyboard shortcuts and midi mappings 
- Changed: more compact, cleaned up GUI
Seems to work allright, but then I did refactor things a bit more...there might be new bugs lurking?!

The GUI was tweaked too. Got sort of triggered by joule explaning how to remove the focus border around the textfield, I also made some other things more clean and compact.

Looks like this now:
v1.57-expanded.png
 
A bit retro B)

Tracking with Stuff. API wishlist | Soundcloud


#109 Djeroek

Djeroek

    Probably More God or Borg Than Human Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6684 posts
  • Gender:Male
  • Location:Borneo

Posted 27 May 2017 - 21:38

Already, I just updated the tool page with v1.57.

- Fixed: error when updating view with out-of-range value #99
- Fixed: error when trying to select data from editor popup #100
- Fixed: error is thrown when entering "return" into main method #97
- Fixed: model doesn't work when last line is comment #95
- Fixed: expecting "models" to be present in custom user-folder #87
- Fixed: favorite icons and preset highlighting (got broken in 1.55)
- Fixed: ChordMemory model had a few flaws
- Added: "Apply to Line", for quick single-line output
- Added: additional keyboard shortcuts and midi mappings 
- Changed: more compact, cleaned up GUI
Seems to work allright, but then I did refactor things a bit more...there might be new bugs lurking?!

The GUI was tweaked too. Got sort of triggered by joule explaning how to remove the focus border around the textfield, I also made some other things more clean and compact.

Looks like this now:
attachicon.gifv1.57-expanded.png
 
A bit retro B)

 

Got this when installing;

 

C:\Users\pluge\AppData\Roaming\Renoise\V3.1.0\Scripts\Tools\com.renoise.xStream.xrnx\main.lua' failed in one of its notifiers.

The notifier will be disabled to prevent further errors.
 
Please contact the author (danoise [bjorn.nesby@gmail.com]) for assistance...
 
main.lua:155: attempt to index field 'preferences' (a nil value)
stack traceback:
  main.lua:155: in function <main.lua:153>

Opening the tool after install gives;

 

'C:\Users\pluge\AppData\Roaming\Renoise\V3.1.0\Scripts\Tools\com.renoise.xStream.xrnx\' failed to execute in one of its menu entry functions.

 
Please contact the author (danoise [bjorn.nesby@gmail.com]) for assistance...
 
.\source/xStream.lua:42: attempt to index field 'prefs' (a nil value)
stack traceback:
  .\source/xStream.lua:42: in function <.\source/xStream.lua:28>
  [C]: in function 'xStream'
  main.lua:116: in function 'show'
  main.lua:142: in function <main.lua:141>


#110 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6250 posts
  • Gender:Male
  • Interests:wildlife + urban trekking

Posted 28 May 2017 - 01:58

Got this when installing;

Opening the tool after install gives;

 

I'm not getting that error, but maybe I have an idea - try replacing main.lua with this version?

 

Attached File  main.lua   10.2KB   9 downloads


  • Djeroek likes this

Tracking with Stuff. API wishlist | Soundcloud


#111 Djeroek

Djeroek

    Probably More God or Borg Than Human Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6684 posts
  • Gender:Male
  • Location:Borneo

Posted 28 May 2017 - 08:10

I'm not getting that error, but maybe I have an idea - try replacing main.lua with this version?

 

attachicon.gifmain.lua

 

that seems to have fixed it :)



#112 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6250 posts
  • Gender:Male
  • Interests:wildlife + urban trekking

Posted 28 May 2017 - 11:19

that seems to have fixed it :)

 

Cool - but also a bit "scary", because I didn't actually fix any errors, just left out the whole keybinding/midi-mapping bits. Which is 100% working here. 

 

Seems I'm running against some internal limitation of the lua runtime here. I've been known to do that...

did you know that our lua engine allows you to create no more than 300 local variables per function? Neither did I, until recently  B)


  • Djeroek likes this

Tracking with Stuff. API wishlist | Soundcloud


#113 Djeroek

Djeroek

    Probably More God or Borg Than Human Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6684 posts
  • Gender:Male
  • Location:Borneo

Posted 28 May 2017 - 11:36

Cool - but also a bit "scary", because I didn't actually fix any errors, just left out the whole keybinding/midi-mapping bits. Which is 100% working here. 

 

Seems I'm running against some internal limitation of the lua runtime here. I've been known to do that...

did you know that our lua engine allows you to create no more than 300 local variables per function? Neither did I, until recently  B)

^I have no to a very slight idea what all that means :) , but as long as you manage to turn in cool useful tools I can't complain. Are these limitations beatable with a new Renoise, or intrinsic to Lua?



#114 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6250 posts
  • Gender:Male
  • Interests:wildlife + urban trekking

Posted 28 May 2017 - 11:49

Are these limitations beatable with a new Renoise, or intrinsic to Lua?

 

Beatable, for sure. We have options when compiling the Renoise-flavored lua.

It's just that I've not come across this before. But I have experienced similar problems in other embedded languages.


Tracking with Stuff. API wishlist | Soundcloud


#115 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6250 posts
  • Gender:Male
  • Interests:wildlife + urban trekking

Posted 10 June 2017 - 17:39

Teasing what might be the most important step so far for this tool: 

 

xstream-stack-sneak-preview.gif

 

The idea have been around for a while: allow models to pass input from one to the next, in a serial or parallel fashion. 

 

So you can have something like a sequencer, and plug it into another model that transposes the notes. 

And not limited to a single track either - the first model could output to one track, and the transposed version could be written to a different track. 

 

It opens up the door to a multitude of possibilities, as well as making it much more attractive to write small, focused models that perform a single task. 

 

The processing would be linear, a bit like the left -> right processing in Renoise. 

But even with that limitation, extensive routing is possible - if you could then specify for each model, where it should get its input from. 

 

Here's a flowchart which describe the signal flow: 

xstream-flowchart.gif

 

Edit: maybe not so clear from the illustration, but those lines are arrows, going in one particular direction

 

The tricky thing was to come up with a reasonably simple way to express this visually - but I'm pretty happy with the result (see GIF above). 

 

Now, as stacks introduce the idea of connecting multiple models, obviously they need to be able to be imported & exported, just like regular models. 

But even better, I wanted this to be something that you didn't necessarily have to worry about - so the tool will be able to save the entire state in the song,

and recall it the next time you load the song. 

 

Now, this feature will take considerable time and effort to finish - it touches upon about a million small things and will need a LOT of testing. 

Until then, v1.57 is the premium choice.


Edited by danoise, 10 June 2017 - 19:23.

Tracking with Stuff. API wishlist | Soundcloud


#116 joule

joule

    Guruh Motha Fakka is Levitating and Knows Everything About Renoise Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • 1382 posts
  • Gender:Not Telling
  • Location:Sweden
  • Interests:music, philosophy, engineering

Posted 12 June 2017 - 09:41

Best tool with a bright future! I'm very much looking forward to this kind of modularity.

 

Here is an example of a user case that I hope will eventually be possible with stacked models:

 

1) Select a model - "Chord input" - as model A. Recognition of chord and inversion happens here.

2) Select a model - "Invert chord" as model B. Basically you can use some effect number in track x, to modify the chord inversion from model A here.

3) Select a model - "Chord to arp" as model C. This can be some kind of scheme to output the chord according to some kind of indexed notes from track y, for example.

 

Some issues here:

* It would require voice handling to determine the "active chord". Essentially, logic that spans over multiple lines - keeping a table unless pattern data changes it. Or maybe this is meant to be covered by the "event" system?

* Also, the function described in point 2 would require an 'effect voice' handling. The inversion should be active until some other effect number happens.

* Multiple track inputs is still not addressed? This would be required by model C above. Maybe this is covered in your flowchart explanation - I'm not sure.

* Some auto-rerender feature would be good. Auto-generating data (thru-out song??) whenever some arg or code is changed.

 

Just throwing these ideas here, in case they have something that requires consideration.


Edited by joule, 12 June 2017 - 09:48.


#117 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6250 posts
  • Gender:Male
  • Interests:wildlife + urban trekking

Posted 12 June 2017 - 12:20

Yes - please bring on the scenarios.

 

As you mention voice handling specifically, this is something I thought about too. 

 

See, there is a realtime voice manager in xLib and it does a great job at keeping track of voices - which track, instrument, etc. 

But xStream is a different kind of beast. Here, the output exists in a kind of "quantum state" because it's written to the track ahead of time and this content might change at any given moment. 

 

Which means, you can't just decide that "ah ok, got some voices here", because with the flick of a switch, those voices would never have existed. 

Same goes for output, you can't just cancel a voice because the output contained a note-off at some point. 

 

I know, this is not relevant for you because you're not interested in the streaming aspect. 

But voice-management with streaming support is indeed something that will need special attention. 
And this could all be prototyped as a model, btw.  ^_^
Multiple track inputs is still not addressed?

A model will still be passed a single xline - the one of it's designated "read track", or the one passed from another model (by means of the routing as illustrated in the GIF).

But in addition, models will be able to directly access the buffer of any previous model - in the flowchart, model C has access to any track as the xline, and/or the output from models A+B.

 
And of course, this is just the "managed" part of the tool. You have always been able to read the input from any track -
the problem is that you're then hard-coding track indexes into models - which is a bad practice. Better to have some degree of management here. 

 

--

 

Another thing you bring up is "indexed notes". Now, this happens to fit nicely with the idea of an xline - using columns to represent indices. 

 

But there is a bigger topic here: sometimes you need to pass data around, in one form or the other. 

So, in which other ways can the xline can be used as a "protocol" between models? 

 

imagine for example, that the first model is an LFO. It creates high-res data and writes it into the "xline.automation" property? 

Now, this automation is not going anywhere before the xline is actually written. So if it got passed on to Model B, then Model B would be able to interpret the automation and use it for "whatever" it wanted.  

For example, writing Sxx commands using an LFO is quite fun. 

 

What do you think of treating the "xline as protocol", rather than "passing arbitrary sets of data" - can you think of some immediate limitations?

Take MIDI as the example, they implemented sysex because they knew that the protocol would otherwise be too strict. 

 

Some auto-rerender feature would be good. Auto-generating data (thru-out song??) whenever some arg or code is changed.

 

Yes, or simply yet another output scope: MTX (matrix). Complementing TRK/SEL/LINE

I mean (and I'm guessing here): what you're really looking for is the ability to have something that evolve over time, in more than a single pattern? Like when you're streaming.

 

Auto-render would then be a simple checkbox, somewhere. 


Tracking with Stuff. API wishlist | Soundcloud


#118 joule

joule

    Guruh Motha Fakka is Levitating and Knows Everything About Renoise Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • 1382 posts
  • Gender:Not Telling
  • Location:Sweden
  • Interests:music, philosophy, engineering

Posted 14 June 2017 - 16:35

What do you think of treating the "xline as protocol", rather than "passing arbitrary sets of data" - can you think of some immediate limitations?
Take MIDI as the example, they implemented sysex because they knew that the protocol would otherwise be too strict.

 
In a private tool of mine that has this kind of serialness, this is pretty much what I do. In addition, I have a table inside my "fline" named _meta, where I can store custom stuff like origin, previously analyzed chord quality and such.
 
Technically, I feel that an xline is a bit too specific for this task, though. Not sure about your data structure, but maybe just dump a table from one model to another (and if the table happens to contain an xline key, so be it).
 
I'm thinking.. In model B, you could access model_a["xline"] for example. Maybe all models could share the same sandbox? (or kind of a "parent sandbox / evironment" for __newindex. Of course, model "A" being executed first, then B, C et c). This would allow something like the following scenario:
Model A and Model B has different inputs. Model C outputs to a track, and alternates between data from model A and B (depending on some criteria).

I mean (and I'm guessing here): what you're really looking for is the ability to have something that evolve over time, in more than a single pattern? Like when you're streaming.


Not really, I think. Personally, I'm only interested in abusing xstream for generating pattern data (offline). So the main use for "songwide auto-generate on arg change" is to make it fit into a trial'n'error workflow.

Edited by joule, 14 June 2017 - 16:44.


#119 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6250 posts
  • Gender:Male
  • Interests:wildlife + urban trekking

Posted 14 June 2017 - 17:34

I have a table inside my "fline" named _meta, where I can store custom stuff like origin, previously analyzed chord quality and such.

 

Great that you bring up a concrete scenario. 

In case of the chord analyser, "origin" is what - the originating track index? This we've got covered, as the stack will specify the actual track index for each member - both for reading and writing. 

The "chord quality" would obviously be some kind of data associated with the model, and as such, also directly accessible. 

 

And we need a streaming voice-manager as well. I mean, you mentioned this: 

 

It would require voice handling to determine the "active chord"

 

This is where I'd imagine that a voice-manager would allow you to figure out the active voices. What would make it a "streaming" voice-manager is that the xinc decides what results you will get. 

Therefore, it needs to be a process, automatically running in the background. I imagine it would take a few cues from the regular xVoiceManager, perhaps even include some of the methods - release_all() could write note-offs in relevant columns, etc. 

 

And the streaming is also my biggest concern when it comes to arbitrary data. Accessing these data could (should) yield a result, depending on the xinc that you're currently located at.

This is not too bad with a voice-manager, but it could get messy with arbitrary data. Definitely hard to come up with an efficient, elegant solution here. 

 

Other than that, it really sounds like the approach I've taken would be working for you as well. 


Tracking with Stuff. API wishlist | Soundcloud






Also tagged with one or more of these keywords: live coding, sandbox