Jump to content


Photo

New Tool (3.0,3.1): xStream

live coding sandbox

  • Please log in to reply
218 replies to this topic

#201 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 7009 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted 15 August 2018 - 10:11

It will mostly be improvements to the buffer (the massive speedup we did together), which still lacked proper testing.
And it's well worth the effort - I've already found a few stupid typos and such.

The "stacked models" branch is of course still there too. I'm most likely going to cherry-pick things from it, once this release has settled a bit.
 

didn't know what branch to start from.

I'm currently working on merging the `xstream-new-buffer-implementation` branch into the master
- mostly working on this locally, but already made a few commits to the master, I think.

This is the milestone I'm working towards, btw.
https://github.com/r...rnx/milestone/3


Re the sandbox: yeah, how it was _planned_ to work, and how it _actually_ ended up being used are two slightly different things.
I think by now you've been diving deeper into lua meta-methods than I have, so I'm keen to hear your input/opinions.

AFAIK the "table as a switch" (using associative, string based keys for access) is very efficient in lua, comparable with environment tables.
But perhaps you have evidence that shows otherwise? I must admit, profiling code is often the last thing I do ...

Tracking with Stuff. API wishlist | Soundcloud


#202 joule

joule

    Guruh Motha Fakka is Levitating and Knows Everything About Renoise Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • 1863 posts
  • Gender:Not Telling

Posted 15 August 2018 - 10:20

I will have a look and get back here. It's not at all certain that I'm correct in everything that I say :o

This is one of those times I wish there was a real way to trace function calls backwards in some IDE. The whole xStream code (like any larger project) isn't very easy to penetrate and get a full birds-eye perspective on.

 

At least, I'm pretty positive that this particular __index metamethod is the single one function that is most frequently called within the whole software :) (not counting native/renoise metamethods)


Edited by joule, 15 August 2018 - 10:23.


#203 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 7009 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted 15 August 2018 - 11:09

I just committed my stuff. It more or less wraps up the merged buffer implementation.

Btw: I found what seems to a typo in the xLine read method you contributed. Not sure if you're using the code elsewhere, so I wanted to point it out:
https://github.com/r...f73f70f4165R303

I also did away with the variables pointing to functions. It's my impression that there is no overhead when calling functions that are properties of an object.
But again, maybe you had some reason to make this change? If not, I think it just makes the code slightly harder to read.
 

This is one of those times I wish there was a real way to trace function calls backwards in some IDE. The whole xStream code (like any larger project) isn't very easy to penetrate and get a full birds-eye perspective on.


Yep, manual tracing goes some of the way (you can identify often-called functions in a logging tool). But it has downsides - maintenance, mostly - and isn't proper debugging either.
Note: I usually strip away the TRACE statements once a tool goes on tools.renoise.com, so there's no a performance penalty to speak of
  • joule likes this

Tracking with Stuff. API wishlist | Soundcloud


#204 joule

joule

    Guruh Motha Fakka is Levitating and Knows Everything About Renoise Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • 1863 posts
  • Gender:Not Telling

Posted 15 August 2018 - 11:41

It doesn't make a big difference localizing these things, no. I just do it on things that are very frequently accessed since it's easy and it's faster. It seems each table/object depth adds the same time it takes to call a same-scope variable (n_time*depth). I mean, if optimizing the most frequently used functions would make the software 25% less hungry, why not ;)

 

One point I'd like to stress though, that could probably be very easy to fix, is to reuse xline (and args if that is an object?) instead of creating new instances on every line. That's an object (tables) creation (and lots of init+gc stuff) that can be avoided by reusing their "skeleton" with some simple :clear() function. E g. create if xinc == 0 or xline:clear()


Edited by joule, 15 August 2018 - 11:46.


#205 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 7009 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted 15 August 2018 - 12:05

Ah, yes, the "re-usable pool of objects" approach.
This works well when things are predictable, but it's made more complex in xStream by the fact that you can do stuff like this:

xline.note_columns[1] = {
 note_string = "C-4"
}

You're basically allowed to redefine xline properties (note/effect columns), which are then instantiated as classes during execution (with default values for panning, etc. being added on the fly).

But you're right - the xline _itself_ could indeed be turned into a resource pool. Something to think about! 

 

I also think I know what you're looking for here: some "middle road" solution with a shared codebase for "xStream light" and then the full version? 


Tracking with Stuff. API wishlist | Soundcloud


#206 joule

joule

    Guruh Motha Fakka is Levitating and Knows Everything About Renoise Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • 1863 posts
  • Gender:Not Telling

Posted 15 August 2018 - 12:14

I'm actually just thinking what would happen if 8 tracks are rendered simultaneously, with something like remapped chord voices via midi. "Would it be possible with just one line write-ahead?"

 

I'm not sure I get the dilemma of your example, but maybe you're right. I thought things that must be "initialized" could be so on :clear(), that it should in any case be easy to re-generate the object into something identical to a normal init. Maybe you mean that you have some clever way of 'copying' a mother-xline into temporary children-xline, I'm not sure.

 

Anyway, fun fact that might be relevant for this topic: when clearing a fixed table, it's faster to set individual values to nil if they are fewer than 8, compared to setting the table to a new { } :)

 

PS. Let us all know when/if we should download the fresh stuff!


Edited by joule, 15 August 2018 - 12:22.


#207 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 7009 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted 15 August 2018 - 17:35

8 tracks are rendered simultaneously, with something like remapped chord voices via midi

 

Ah, OK. Yeah, it's possible to write directly to other tracks (you even did a proof of concept, didn't you?). I rarely would need this, I would be much more likely to have different models running side by side. 

 

Also related to this topic + performance - in many cases, the fact that xStream reads pattern data is also a waste of CPU cycles.

It's just that I haven't realized how to make this configurable in an easy to understand manner (write-only models? just adds to the scare factor IMO)

Stacking, on the other hand, is simple enough as the concept implies that you are reading pattern data - from the track OR passed down from a different model. 

 

Hm, now I opened the topic of stacking but I don't really feel like discussion that right now. 

Instead I'll spend some time tonight taking the current version for a good spin :D


Tracking with Stuff. API wishlist | Soundcloud


#208 joule

joule

    Guruh Motha Fakka is Levitating and Knows Everything About Renoise Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • 1863 posts
  • Gender:Not Telling

Posted 15 August 2018 - 20:10

Also related to this topic + performance - in many cases, the fact that xStream reads pattern data is also a waste of CPU cycles.
It's just that I haven't realized how to make this configurable in an easy to understand manner (write-only models? just adds to the scare factor IMO)
Stacking, on the other hand, is simple enough as the concept implies that you are reading pattern data - from the track OR passed down from a different model.

 
If you'd change xLinePattern to somehow work on a need-to-read basis (using __index metamethod), the current (and fast) line string reading would be superfluous in those cases. So it should be avoided then...
 
Hmm. this might be an idea:
 
1) Only generate the env.xline when/if it is first accessed in the sandbox. This should be simple with an if statement inside the sandbox environments __index metamethod. The default value of env.xline would be just "false".
2) Store the actual renoise.PatternLine into some key in the sandbox environment, as an alternative for write-only models to access. Is this really too scary?
 
This seems like quite an easy fix, but I haven't looked at where you update the sandbox environment.
 

Spoiler

 

EDIT:

Or you could implement the approach I'm suggesting (need-to-read basis) on the note_columns and effect_column getter/setter properties inside the xLine class? Let self.pattern_line be nil until someone tries to access it. That might be much simpler than messing about in the sandbox environment. Unless you're aiming for really optimizing everything and avoid any unnecessary object creations.

Spoiler

Edited by joule, 16 August 2018 - 10:48.


#209 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 7009 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted 16 August 2018 - 11:38

Yes, many roads lead to Rome here. 

 

When I mentioned scary, what I meant was that I didn't want to push this over to the person using xStream - having to explicitly set a model as "write only" (WTF?) 

So yeah, anything that gets done needs to work automatically and transparently. 

 

I just added MIDI device hot-plugging, btw. Nice and easy addition to the xMidiIO class. 


  • ghostwerk likes this

Tracking with Stuff. API wishlist | Soundcloud


#210 joule

joule

    Guruh Motha Fakka is Levitating and Knows Everything About Renoise Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • 1863 posts
  • Gender:Not Telling

Posted 16 August 2018 - 15:37

Nice to see some development! It's damn hard to penetrate the code though (follow the 'stream' of data and understanding all the 'detours'). I've pretty much given up for the time being.

Some simple flow chart of all classes would be highly appreciated :)

#211 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 7009 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted 04 October 2018 - 17:40

At long last, a new release of xStream  :)
http://www.renoise.com/tools/xstream
 
This update contains both the workaround for the "preferences is nil" error that has plagued people since ages, 
but is also a pretty thoroughly tested general update with a lot of enhancements under the hood. 
 
Enjoy, and keep me posted if you encounter any problems with it!
## 1.6 changelog
- Core: update cLib/vLib/xLib classes 
- Core: using xStreamBuffer 
- Added: support MIDI device hot-plugging
- Changed: using popup to choose render mode (global toolbar)
- Changed: updated / cleaned up models + documentation 
- Fixed: unable to start tool (preferences is nil) (#102)
- Fixed: error when editing argument, applying new settings (#103)
- Fixed: when renaming model, watch out for certain characters (#104)
- Fixed: renaming userdata should update other userdata (#109)
- Fixed: error when adding preset via favorite dialog bug (#114)
took the time to implement this suggestion as well: 
 

I think the xStream GUI has quite an intense scare factor to new users, with all its custom buttons and symbols.
 
attachicon.gifxgui.gif


  • Djeroek and joule like this

Tracking with Stuff. API wishlist | Soundcloud


#212 Djeroek

Djeroek

    Probably More God or Borg Than Human Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6798 posts
  • Gender:Male
  • Location:Borneo

Posted 06 October 2018 - 06:13

^ I don't see any model presets in the latest update, am I doing it wrong?



#213 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 7009 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted 06 October 2018 - 08:40

My guess is that you've configured the tool to user some custom userdata folder at some earlier point?  

 

Try going into Options > General > Userdata, press "Reset" and see if this helps. 


Tracking with Stuff. API wishlist | Soundcloud


#214 joule

joule

    Guruh Motha Fakka is Levitating and Knows Everything About Renoise Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • 1863 posts
  • Gender:Not Telling

Posted 06 October 2018 - 09:48

I have a very small suggestion about improvements to dialog titles. Quite a few of them only has "xStream" as a title. It would be good if these had some explanatory window titles, further shaving off some of the scare factor. (The utopia being that you can explore and learn a software without reading the manual). Unless you really know how xStream is laid out, some of the windows popping up will leave you clueless.



#215 Djeroek

Djeroek

    Probably More God or Borg Than Human Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6798 posts
  • Gender:Male
  • Location:Borneo

Posted 06 October 2018 - 21:42

Try going into Options > General > Userdata, press "Reset" and see if this helps. 

That did the trick. Got this notice when trying to run the granular model with no instrument loaded:

"'C:\Users\pluge\AppData\Roaming\Renoise\V3.1.1\Scripts\Tools\com.renoise.xStream.xrnx\main.lua' failed in one of its notifiers.

The notifier will be disabled to prevent further errors.
 
Please contact the author (danoise [bjorn.nesby@gmail.com]) for assistance...
 
.\source/xLib/classes/xEffectColumn.lua:243: attempt to perform arithmetic on a nil value
stack traceback:
  .\source/xLib/classes/xEffectColumn.lua:243: in function 'number_string_to_value'
  .\source/xLib/classes/xLinePattern.lua:307: in function 'do_read'
  .\source/xLib/classes/xLine.lua:271: in function 'do_read'
  .\source/xLib/classes/xStreamBuffer.lua:276: in function '_obtain_and_store_input'
  .\source/xLib/classes/xStreamBuffer.lua:527: in function '_read_input_buffer'
  .\source/xLib/classes/xStreamBuffer.lua:446: in function '_create_content'
  .\source/xLib/classes/xStreamBuffer.lua:602: in function 'write_output'
  .\source/xStreamProcess.lua:278: in function 'output'
  .\source/xStream.lua:197: in function <.\source/xStream.lua:196>
  [C]: in function 'bang'
  .\source/xLib/classes/xStreamPos.lua:117: in function 'start'
  .\source/xStream.lua:255: in function 'start'
  .\source/xStream.lua:265: in function 'start_and_play'

  .\source/xStreamUIGlobalToolbar.lua:127: in function <.\source/xStreamUIGlobalToolbar.lua:126>"



#216 Djeroek

Djeroek

    Probably More God or Borg Than Human Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6798 posts
  • Gender:Male
  • Location:Borneo

Posted 06 October 2018 - 21:50

Terminating script gives:

"'C:\Users\pluge\AppData\Roaming\Renoise\V3.1.1\Scripts\Tools\com.renoise.xStream.xrnx\main.lua' failed in one of its notifiers.

 
Please contact the author (danoise [bjorn.nesby@gmail.com]) for assistance...
 
.\source/xLib/classes/xPlayPos.lua:65: Script execution terminated by user.
stack traceback:
  [string "do..."]:37: in function <[string "do..."]:35>
  .\source/xLib/classes/xPlayPos.lua:65: in function <.\source/xLib/classes/xPlayPos.lua:43>
  [C]: in function 'xPlayPos'
  .\source/xLib/classes/xStreamPos.lua:156: in function 'reset'
  .\source/xLib/classes/xStreamPos.lua:118: in function 'start'
  .\source/xStream.lua:255: in function 'start'
  .\source/xStream.lua:265: in function 'start_and_play'

  .\source/xStreamUIGlobalToolbar.lua:127: in function <.\source/xStreamUIGlobalToolbar.lua:126>"



#217 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 7009 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted 07 October 2018 - 08:55

Thanks djeroek - keep 'em coming! 

 

Btw:

 

Terminating script gives:

 

What do you mean by "terminating" - How did the error occur?


Tracking with Stuff. API wishlist | Soundcloud


#218 Djeroek

Djeroek

    Probably More God or Borg Than Human Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6798 posts
  • Gender:Male
  • Location:Borneo

Posted 07 October 2018 - 09:00

I mean when processing is taking to long and you get a notice asking if you want to abort and terminate the script or keep waiting.



#219 danoise

danoise

    Probably More God or Borg Than Human Member

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 7009 posts
  • Gender:Male
  • Location:Berlin
  • Interests:wildlife + urban trekking

Posted 07 October 2018 - 10:23

Ah, in this case you will always get a notice of some kind. I don't think I can prevent that. 


Tracking with Stuff. API wishlist | Soundcloud






Also tagged with one or more of these keywords: live coding, sandbox