Jump to content


Photo

New Tool (3.0,3.1): xStream

live coding sandbox

  • Please log in to reply
158 replies to this topic

#126 joule

joule

    Guruh Motha Fakka is Levitating and Knows Everything About Renoise Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • 1487 posts
  • Gender:Not Telling
  • Location:Sweden
  • Interests:music, philosophy, engineering

Posted 09 August 2017 - 07:53

I have pretty much done what Oopslfly is talking about, but a bit more elaborate - passing each line notifier event thru a sandboxed function.

 

Be aware that the biggest obstacle when it comes to this kind of operation (track1+track2=track3), is that chances are very low that you'll get it working with pattern aliases. So the tool will never be perfect. This is for various design reasons (and bugs, I think) in the API. At least, that's my experience..



#127 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6414 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted 09 August 2017 - 12:11

chances are very low that you'll get it working with pattern aliases. So the tool will never be perfect

 

Don't forget that matrix slots can be aliased as well. It's all very tricky in the detail, but possible.

Don't know what bugs you are referring to? 


Tracking with Stuff. API wishlist | Soundcloud


#128 joule

joule

    Guruh Motha Fakka is Levitating and Knows Everything About Renoise Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • 1487 posts
  • Gender:Not Telling
  • Location:Sweden
  • Interests:music, philosophy, engineering

Posted 09 August 2017 - 12:21

There are several issues regarding pattern aliases. I've tried to explain it in a previous thread, but it's a bit difficult. http://forum.renoise...dex-observable/

 

Short story:

 

1) Keeping track of all aliases is generally quite tricky (but possible).

2) alias_pattern_index_observable will bang when a sequence is removed. This should be a bug, or at the very least a design flaw.

3) alias_pattern_index_obserfvable will bang when a sequence is inserted. So, yet more work-around logic is needed to circumvent this, if it's even possible.

 

Eventually, you'll find that you have to use lots of workarounds and complicated schemes. Everything would have been solved with some simple change in the API itself. So.. the motivation you're left with after trying, is only enough for writing a crappy forum post about it :)


Edited by joule, 09 August 2017 - 12:21.


#129 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6414 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted 09 August 2017 - 12:51

I've tried to explain it in a previous thread

 

Hey man, thanks. That's a constructive post you made there...good points.

You should've expected at least *some kind* of response for that. For me, it somehow slipped under the radar. 

 

And while I'm personally not afraid to implement workarounds, I agree that anything that can't be done cleanly/properly needs a critical examination. 


Tracking with Stuff. API wishlist | Soundcloud


#130 OopsIFly

OopsIFly

    Guruh Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 924 posts
  • Gender:Male
  • Interests:...daydreams... -VS- ...propaganda...

Posted 09 August 2017 - 16:04

Oh, it's not a bug, it's a quirk then - if with clever filtering and double checking every relevant event can be collected somehow, then things are finde with me. Ofc I can understand its problematic if you don't know the whereabouts, and also think that a proper api that don't need quirkyness workarounds is the best, but well, in my experience almost any complex software api has such strange behaviour somewhere in its guts.

 

Hm, so I thought I'd first do the processing code until it works fine. It would also set me going as I could compose a tune straight or with some native groove, and then experiment with grooves, single-run applying them on command. This would also be useful for old (quasi finished) tunes, that are to get some a posteriori ants into their pants...

 

Then I thought about an event catcher func that is as slim as ever possible, just filtering/collecting relevant events to some stack or so. Then a processing routine that goes in the background that will apply the action - is the idle notifier background threaded, or would I have to split the workload in packets for the whole thing not to block the engine at any point more than absolutely nessecary? Is it generally advisable to put processing out of the notifier handlers and schedule into idle routines instead?

 

I mean also, the nature of the task I have in mind is very simple, and with the right concept it could work out well. I don't know how robust I would want to make this. For example the effected copy of a track is never to be touched by the user, I'd for example go for assuming the user not to touch it and periodically and by demand overwrite whatever crap happened to the track just to make sure. Maybe the processing also can be kept rather simple, by for example always processing the whole patterntrack on a change instead of trying to weed out just single notes that were changed. I will also try looking for nudge/shift tools, or similar, to lend some note overlap handling for shifting notes to (new) columns. My main concern on the live grooving approach is to keep it, for the user, as close to a normal workflow as possible, I have some Ideas going on about this. But some concepts will imply my own preffered workflow I think, or also have to add quite some visible baggage to the project in order for it to work nicely.

 

Maybe you want to split the discuss to another thread as its not related to your tool any more. I also have to say there will be some time passing until I can try working on this. Might have to wait for nothern hemisphere winter... I am no seasoned programmer, just an enthusiast, and I have other things to waste my time with, and want to get warm with another tool that aims accelerating sample creation/editing/processing workflow with the convolver first.



#131 joule

joule

    Guruh Motha Fakka is Levitating and Knows Everything About Renoise Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • 1487 posts
  • Gender:Not Telling
  • Location:Sweden
  • Interests:music, philosophy, engineering

Posted 09 August 2017 - 16:21

In this case, using an idle notifier/process slicer is not necessary if you have a fairly fast processor IMO. I'd say go for a simple approach - you can always add such things later on if really needed. It's more sensible to first focus on optimizing any .song() access, as this often will suffice.

#132 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6414 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted 10 August 2017 - 11:05

Then I thought about an event catcher func that is as slim as ever possible, just filtering/collecting relevant events to some stack or so. Then a processing routine that goes in the background that will apply the action - is the idle notifier background threaded, or would I have to split the workload in packets for the whole thing not to block the engine at any point more than absolutely nessecary? Is it generally advisable to put processing out of the notifier handlers and schedule into idle routines instead?


When you're processing a whole song, or basically anything that takes enough time that a script timeout could happen (a dozen seconds or so), you can "slice" your process.
This is similar to a thread (co-routine), and really isn't that hard to implement. It's just a matter of creating a self-contained method that does the processing and periodically hands over time to the main process via a yield(). Here is a working example:
https://github.com/r...cedProcess.xrnx

 

Maybe you want to split the discuss to another thread as its not related to your tool any more. I also have to say there will be some time passing until I can try working on this. Might have to wait for nothern hemisphere winter... I am no seasoned programmer, just an enthusiast, and I have other things to waste my time with, and want to get warm with another tool that aims accelerating sample creation/editing/processing workflow with the convolver first.

 

You'll probably want to create a topic once the tool takes shape, but having it here for now is fine with me  :)


Tracking with Stuff. API wishlist | Soundcloud


#133 oise

oise

    Member

  • Normal Members
  • PipPip
  • 39 posts
  • Gender:Male
  • Location:Berlin

Posted 24 September 2017 - 15:59

Hello danoise and all the guys around this wonderful topic.

Many thanks for this great tool!

 

Recently i tried an updated version and now i can't use it because of the error (OSX). Please see attached screenshot.

Can you help with this?

Attached Images

  • Renoise-Screen Shot 2017-09-24 at 16.52.41.png


#134 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6414 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted 24 September 2017 - 17:02

Hello danoise and all the guys around this wonderful topic.

Many thanks for this great tool!

 

Recently i tried an updated version and now i can't use it because of the error (OSX). Please see attached screenshot.

Can you help with this?

 

Yeah, this bug happens for an unknown reason.

 

Still, there is a known fix/workaround here. 

http://forum.renoise...tream/?p=357808


  • oise likes this

Tracking with Stuff. API wishlist | Soundcloud


#135 g8tr1522

g8tr1522

    Member

  • Normal Members
  • PipPip
  • 15 posts
  • Gender:Male
  • Location:Gainesville, FL

Posted 24 September 2017 - 22:41

I'm taking a crack at this again after I saw the updated documentation. I tried a while ago and gave up cause I couldn't find enough documentation, so I really appreciate that update!

 

So, I'm trying to open my previously saved models, but I can't.

 

I have a model on my desktop named "mymodel.lua" and is exactly the same as the "Untitled model.lua" model. When I try to open it using

(models toolbar >) create new model > Locate a file on disk > mymodel.lua

a "Scripting Tool Error" window pops up with the message:

 

'C:\Users\Owner\AppData\Roaming\Renoise\V3.1.0\Scripts\Tools\com.renoise.xStream.xrnx\main.lua' failed in one of its notifiers.

 
Please contact the author (danoise [bjorn.nesby@gmail.com]) for assistance...
 
.\source/xStreamModel.lua:34: Wrong type of parameter
stack traceback:
  [C]: in function 'assert'
  .\source/xStreamModel.lua:34: in function <.\source/xStreamModel.lua:31>
  [C]: in function 'xStreamModel'
  .\source/xStreamUIModelCreate.lua:292: in function 'navigate_to_model'
  .\source/xStreamUIModelCreate.lua:239: in function 'show_next_page'
  .\source/xStreamUIModelCreate.lua:143: in function <.\source/xStreamUIModelCreate.lua:142>
 
I get the same thing when I copy and paste the file's contents using the "Paste from clipboard" option. I also tried uninstalling xStream, closing Renoise, and reinstalling. 
This also happens when I try opening up one of the example models using those methods I tried above.
 
Any ideas? Am I using "Create new model" correctly? Does this have anything to do with the location of the Userdata folder?
 


#136 g8tr1522

g8tr1522

    Member

  • Normal Members
  • PipPip
  • 15 posts
  • Gender:Male
  • Location:Gainesville, FL

Posted 25 September 2017 - 00:01

Also, how do you show line numbers in the xStream script editor? 



#137 g8tr1522

g8tr1522

    Member

  • Normal Members
  • PipPip
  • 15 posts
  • Gender:Male
  • Location:Gainesville, FL

Posted 25 September 2017 - 18:01

Is there any way to include libraries? Do I just put a simple #include "lib.h" at the top of the code?



#138 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6414 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted 26 September 2017 - 10:16

Hey, and thanks for the feedback.  

 

When you're having trouble with such a basic task (creating a model), I would have to ask about whether you're using a custom userdata folder (since you bring this up - it's set from the xStream options dialog), and possibly what OS you're using?

 

What you are trying to do really shouldn't cause such problems. 

That said, the current version of the tool is not completely without bugs. Who knows, you might have encountered another one.

 

Also, how do you show line numbers in the xStream script editor? 

 

There is a number of features I'd like to include, but that can't be done without improving the Renoise API itself. Having line numbers is one such feature.

 

However, I have been toying with an idea which would enable you to edit xstream files outside Renoise. I think this is actually a much more convenient thing, because no matter how much how we tweak the text editor in Renoise, most people who are dabbling with this tool probably have a text editor that they prefer anyway. So I see no reason why xStream shouldn't benefit from that. Hell, you could even use the Renoise scripting console then, if you wanted to (I know some people do ^_^)

 

Is there any way to include libraries? Do I just put a simple #include "lib.h" at the top of the code?

 

Everything that gets included with xStream is specified in the main.lua  But - that doesn't mean that you can just starting referencing functions and method from within the xStream "main method" (the processing method that transforms pattern data, not to be confused with the "main.lua" file, which is simply the entry point for any renoise tool. 

 

This is so, because xStream wraps the main method in a lua sandbox - which makes certain variables (e.g. the all-important xinc) available to the model, while also hiding potentially 'dangerous' things like file system access.  

In order to benefit from method in your own files, you need to explicitly define sandbox accessors to your own properties and methods. If it sounds complicated, it's not - just a question of opening and modifying the file called xStreamModel.lua.

 

Hope this helps!


Tracking with Stuff. API wishlist | Soundcloud


#139 boonier

boonier

    Advanced Member

  • Normal Members
  • PipPipPip
  • 113 posts

Posted 11 October 2017 - 20:12

Hi

 

Say I have a sequence in Track 1, that I want to simply transpose the note randomly between + {0, 12, 24}, so the original value is read from the src track, then modified and written to Track 2, how would I do that?

 

The problem is that the original note value is being modified and it becomes the base note for the next round of modifying...I want to refer to the original note value all the time.

 

Is it possible to read from one track and apply a modified result to another, or do I have to approach this differently?

 

thanks


Edited by boonier, 11 October 2017 - 20:27.


#140 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6414 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted 12 October 2017 - 13:24

The problem is that the original note value is being modified and it becomes the base note for the next round of modifying...I want to refer to the original note value all the time.

 

Ah, so you would essentially be looping over the same notes over and over again? 

Yes, then the result would gradually slide further and further away from the original state. 

 

There is no problem in writing notes to a different track though, you should have the necessary information already - 

for example, using xpos and xline together, you should be able to call a method like xline:do_write (documentation). 

 

The only thing to be aware of is that xStream does not manage this "shadow track", 

The next version of xStream will indeed feature a workflow that integrates track routings, but for now you will have to that yourself. 


Tracking with Stuff. API wishlist | Soundcloud


#141 boonier

boonier

    Advanced Member

  • Normal Members
  • PipPipPip
  • 113 posts

Posted 12 October 2017 - 23:11

How do I use the do_write method?

 

Please could you show me an model example of reading a src track (1) and writing the modified data destination trk 2?

 

for k = 1,rns.tracks[track_index].visible_note_columns do
  if (xline.note_columns[k].note_value < 120) then
    local note = xline.note_columns[k].note_value
    xline:do_write(xpos.seq, xpos.line, 2) <-- if this is pointing to the right track, where do I plug the note data in?
  end
end
 
Thanks


#142 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6414 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted 13 October 2017 - 11:40

The note data is specified by the xline itself. So you could basically take the incoming xline, make a copy of it, modify the properties and then write it to your target track. 

If you didn't make a copy then the modification you make would also be written to the source track - not what you want in this case. 

 

A copy can be made simply by creating an xLine using another xline as argument. 

I can make a quick test, let you see how it's done. 


Tracking with Stuff. API wishlist | Soundcloud


#143 boonier

boonier

    Advanced Member

  • Normal Members
  • PipPipPip
  • 113 posts

Posted 13 October 2017 - 16:31

Thanks that'd be most helpful, it's more of a syntax issue for me - Lua is a relatively new one for me.



#144 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6414 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted 13 October 2017 - 21:49

Thanks that'd be most helpful, it's more of a syntax issue for me - Lua is a relatively new one for me.

 
D'oh, maybe my example was also not that helpful. Cloning, like I described it, is only available in the development branch. 
I need to release that one anyway - wanted to get rid of this annoying issue for some time now.  
 
But it's still possible to achieve what you want, it's just that it's quite a lot more code (using Renoise API calls).
-- amount of semitones to transpose by 
local transpose = -2

-- define our target track index
local target_track_index = 2

-- figure out the pattern-track in which the line exists...
local patt_idx = rns.sequencer:pattern(xpos.sequence)
local patt = rns.patterns[patt_idx]
local ptrack = patt:track(target_track_index)

-- we have our pattern-track and can figure out the line
local rns_line = ptrack:line(xpos.line)

-- finally, we loop/iterate through all visible note columns 
-- in the source track, transpose the note (clamping it to
-- make sure its between C-0 and B-9) and write it to 
-- the line that we previously figured out. 

for k = 1,rns.tracks[track_index].visible_note_columns do
  if (xline.note_columns[k].note_value < 120) then
    local new_note = xline.note_columns[k].note_value + transpose
    rns_line.note_columns[k].note_value = cLib.clamp_value(new_note,0,119)
  end
end
Ed: now here's why cloning would be nicer -
the above example could be reduced to something like this:
 
-- amount of semitones to transpose by 
local transpose = -2

-- define our target track index
local target_track_index = 2

-- clone the xline 
local new_xline = xLine(xline)

for k = 1,rns.tracks[track_index].visible_note_columns do
  if (xline.note_columns[k].note_value < 120) then
    local new_note = xline.note_columns[k].note_value + transpose
    new_xline.note_columns[k].note_value = cLib.clamp_value(new_note,0,119)
    new_xline:do_write(xpos.sequence,xpos.line,target_track_index)
  end
end

Edited by danoise, 13 October 2017 - 23:40.

Tracking with Stuff. API wishlist | Soundcloud


#145 boonier

boonier

    Advanced Member

  • Normal Members
  • PipPipPip
  • 113 posts

Posted 16 October 2017 - 10:07

Awesome, thanks Bjorn! That works great,

 

Yeah I can see that the sugared syntax makes it a lot more succinct. Also, having the tool manage the track routing would be great, like specifying a source and destination track and it won't matter which track is selected, the notes always go from src to dest. I found I had to consciously make sure everything was set up.

 

Another thing, will the next version manage the visible columns shown? At first I was baffled as to where the notes were going in the dest track, only to discover they were hidden!



#146 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6414 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted 16 October 2017 - 10:38

Another thing, will the next version manage the visible columns shown?

 

Obviously, in this example, as we are just writing to the "raw" target track, nothing happens (visibility stays as it is). 

But yes - if the track was managed by xStream, then note columns should be revealed in that track too, as it's being written to.

At least conceptually, everything should works the same with multiple/stacked tracks as they would with a single track. 

 
Of course, once we go into detail, the new stacked-models approach can get tricky. Right now, whether note columns are revealed or not is controlled by the 'expand_columns' option. So the behavior can be set per-model (override the globally set option). But I have to think about what would happen if multiple models were targeting the same track and specified different settings. 

Tracking with Stuff. API wishlist | Soundcloud


#147 joule

joule

    Guruh Motha Fakka is Levitating and Knows Everything About Renoise Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • 1487 posts
  • Gender:Not Telling
  • Location:Sweden
  • Interests:music, philosophy, engineering

Posted 16 October 2017 - 10:56

PS. @Danoise,

If xStream needs optimization when going multi-track, I have some ideas that should speed up the reading/writing of xline a lot.

 

Reading:

Creating an xLine from tostring(renoise.PatternLine) instead of iterating columns.

I'm not sure exactly how you convert it in xlib/xstream now, but it shouldn't be any problem adding an optional argument to xLine:__init() to populate the data quickly from a PatternLine string.

 

Writing:

Comparing xLine (string) to the destination PatternLine (string), only writing what is updated.

 

It might sound like a lot of string operations et c, but it'll be really quick in comparison to accessing renoise.song(). I can submit an example of the xLine class allowing init from patternstring, if it sounds interesting.

 

PS. Another far fetched/vague idea I got is to make xLine some kind of sandbox with meta table, only accessing what is really being accessed by the end user (never populating everything). Perhaps interesting, as well...


Edited by joule, 16 October 2017 - 11:02.


#148 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6414 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted 17 October 2017 - 19:50

It might sound like a lot of string operations et c, but it'll be really quick in comparison to accessing renoise.song(). I can submit an example of the xLine class allowing init from patternstring, if it sounds interesting.

 

Sounds very promising. xLine is at the heart of xStream - it could indeed bring a lot of extra performance  :)

 

I imagine that you could then initialize an xLine in three different ways:

1. Using a table (a "descriptor")

2. Using another xLine instance 

3. Using a string (PatternLine)

 

I will make an attempt at clarifying what the xLine does, and needs to do. That will definitely make it easier to nail down some specs :-)


Tracking with Stuff. API wishlist | Soundcloud


#149 boonier

boonier

    Advanced Member

  • Normal Members
  • PipPipPip
  • 113 posts

Posted Yesterday, 08:34

Is there anyway to lock the tracks i.e. hard code the src track and destination track? 

 

So I replaced the line:

for k = 1,rns.tracks[track_index].visible_note_columns do

with 

local src_trk = args.src_trk

for k = 1,rns.tracks[src_trk].visible_note_columns do

But it seems that you need the source track selected in the Pattern editor. Is that right?



#150 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6414 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted Yesterday, 08:48

it seems that you need the source track selected in the Pattern editor. Is that right?

 

Yep. xStream is currently hard-coded to the selected track. Again, this is something that would change in the planned version 

(I would like to add as a sidenote: it's nice how your questions align so nicely with those plans  ^_^ )


Tracking with Stuff. API wishlist | Soundcloud






Also tagged with one or more of these keywords: live coding, sandbox