I think this is unnecessary and adds complexity - simply having the most recent change override another change is probably much simpler and perfectly acceptable… a song is a big playing field - the odds of two players colliding is really rather minimal.
Have you ever worked two people on a spreadsheet in Google Docs, at the same time? It’s a fascinating experience, you should try it … The tracker is basically a spreadsheet, so you can get a feel for how this could look and work with a spreadsheet in Google Docs - you can see the other “player’s” cursors (in different colors), you can see their edits in real-time, you can see them enter and leave the session, and you can chat with them. If you could see a ghosted version of the other collaborators’ mouse pointers too, that would be awesome. If the cursor left a subtle trail when changes were made (that fades or disappears after a few seconds), that would take some of the “spookiness” out, so you don’t suddenly realize that someone made a change where you were looking.
With the low latency on modern internet connections, you don’t need to worry much about the lag - participants will see the other editors’ cursors and pointers, and they will naturally get out of the way. I’ve been working with 4 different people on the same spreadsheet (relatively small, perhaps the eqivalent of 3-4 patterns in size), and we never once collided or overwrote each other’s changes. I don’t think manual “reservations” are necessary or pratical.
I seem to have a hard time not thinking about this - here are some thoughts I had last night…
Initially, my thinking was that one instance of Renoise would be the “master” and others would connect to it, and they would try to keep things in sync between them.
There are two problems with that approach, that stem from the same issue:
Add a simple “version control” system - meaning, whenever the song is “committed”, we keep a numbered copy of the previous version of the song. Pretty simple, and should be optional of course, in case you’re working with a huge song file.
Found this thread while looking for something similar.
What’s the status of renoise not being able saving to/writing from dirs (as opposed to .xrns-files)?
I’m looking at git with hooks and smudge/clean. It should be able to work with xrns files and zip/unzip behind-the-scenes, I just can’t get my head around it. Anyone tried this as well?
The idea of using a version control system had crossed my mind, but I don’t think it’s going to work well in practice - version control systems are designed for working with source-code, and they are not designed for real-time applications. They are also not guaranteed to automatically resolve many conflicting changes, as would occur in a system trying to use a version-control service in real-time.
It might be feasible to create an integration for a version control system for version control, however - which might be a really cool thing to have, too. So that saving your song would actually create a revision, and so that you could undo changes made by a particular revision. I’m sure there are references between objects stored in a Renoise XML file though - for example, instruments reference samples, so even if the version control system could successfully track changes and patch/remove change sets, you might end up with corrupted files that won’t actually load.
I’m not going to say it wouldn’t be cool if you could get that working - but you should not expect it to work in real-time…
It looks like the best algorithm around for doing this kind of thing, is Operational Transformation:
This algorithm eliminates favoritism - everyone is editing the shared data on equal terms.
Here’s an implementation with a server written in CoffeeScript and running under Node on a webserver, with a client-library (also written in CS) running under JavaScript engines in conventional web browsers:
Pull up two browser windows side by side and edit the text in either window - ask a friend to pull it up and do the same. Pretty amazing. Scroll down that page for more links to far more impressive examples.
It supports text and JSON, so both one-dimensional and two-dimensional data - one dimensional operations would be required for samples, while two dimensional operations could be used for patterns.
The JS client library is only 4KB of JavaScript, so it must be relatively simple. Or in other words, it’s not unthinkable that you could write a client in Lua and actually run it against a ShareJS server. This is all open source - so you could package Node and ShareJS and create a self-contained server distribution.
So the hard part here would be implementing the actual operations, or more accurately, the operational transformations on those - I guess in some sense the operations are already implemented by Renoise.
I’m not going to say this would be easy, but it’s feasible
By the way, I still think my original idea would work, and probably much simpler to implement - but I should highlight one weakness of that approach: it interferes with undo/redo. That is, if you undo an operation, you’re not undoing an operation done by you, necessarily, just the last operation performed by any of the participants, which is probably not the desired effect.
For that matter, if an OT approach was used, undo/redo would be considered operations in their own right - that is, you would have to implement operational transformations for the undo and redo operations. I could be wrong, but I’m not sure the scripting API is up to the job - this may require tighter, more low-level integration with the Renoise editor than is possible with the scripting API. Or not - I don’t know it that well, and definitely have not been keeping up with new developments in the past 6 months
If anybody has the time (and gusto!) to attempt something like this, here’s a very good explanation of OT:
Just a quick follow-up with a couple of observations I’ve gathered from learning some new things since posting in this thread.
If we’re talking real-time collaboration, Operational Transformation, in it’s simplest form, would only work if every possible operation in Renoise can be expressed as a “flat” operation - which may be possible, because almost every collection (instruments, samples, patterns, tracks) in Renoise (as far as I understand) is “addressable” using a flat index. Operational Transformation on anything with structure (such as nodes in an HTML, XML or JSON document) are not simple - they are extremely complicated, and most open source OT libraries do not support it.
If we’re talking on-demand synchronization/merge, a straight diff (such as provided by common diff tools and version control systems) will not work, again, because an XML file, while stored as a flat text-file, does contain structured data, which is much (MUCH) harder to synchronize - which is why you will often experience merge failures in e.g. HTML, XML or JSON files with version control tools. There are specialized diff/merge tools out there designed specifically for XML (such as this one for PHP) but even these can’t always successfully/correctly merge changes to structured data.
To give an example of the “unsolvable” problem, consider an operation where an item is renamed or changes position, versus an operation where an item is deleted and another item is added - without knowledge of the operation that happened, with the end result of both operations looking the same, one can only guess at which operation occurred. Operational Transformation addresses this by working with operations rather than the result of those operations - an XML file is only going to reflect the result, not the actual operation, so an approach using the XML file is always going to be brittle or limited in some way.
The possible exception in the case of Renoise, is a custom XML diff/merge tool specifically designed for Renoise XML files - since all collections (AFAIK) in Renoise are “addressable” (as discussed above) it should be possible to treat the data in the XML structures as flat data rather than as structures, and thereby implement working diff/merge operations, though still with the limitations covered above, as far as being unable to know which operations occurred, and having to make assumptions or guesses, which could be difficult.
In my opinion, real-time synchronization may actually be more feasible to implement than on-demand diff/merge.
But as I see, the Renoise scripting API, as it is, is currently not suitable for implementing this… in order to implement real-time synchronization, the command pattern (which Renoise uses internally to implement e.g. undo/redo) needs to be fully exposed via the scripting API. You need to be able to construct, serialize/unserialize, and execute commands - and command events need to be cancelable. Until the scripting API exposes these features, you can’t implement real-time synchronization.
WoW , I didn’t have time to read through all the topic, but does this mean that then you could have collaboration-gig with your fellow Renoisers (almost)realtime in different countries?
Maybe out of the box thinking is needed, have a camera pointed at the screen or some kind of screen capturing going on and have optical recognition algorithms translate what’s happening realtime, send converted OSC instructions to a arduino controlled robot on the other side, auto-mirroring, tracking the results.
Reading through this post a couple years later - it popped into my head again after tinkering with deepstream a couple of weeks ago.
This thing is basically the back-end we need for real-time collaboration - no need to develop a custom Renoise server or anything like that.
It uses a “last in wins” strategy, which should be fine with today’s internet speeds and the likelihood of actually making conflicting changes.
It’s open-source so anyone could host their own.
It was extremely easy to implement in JavaScript, since they have a JS client library ready to use.
Under Lua, hmm, not so much? Lue is a pretty exotic language - it’s not likely we’ll have a client for that anytime soon. It’s too bad they selected Lua as the scripting back-end. JS would have been much more accessible to a larger audience, and would have made something like this readily available to implement.
Well, either way, I poked through the API reference, and it looks like there would still be some work to do before a script could provide this integration.
On the other hand, maybe with this as a back-end, the Renoise team itself might be interested in making this happen? Not that there’s a client-library for C yet either, so I guess, either way, even though the back-end is there and ready to use, implementing a client-library might be a pretty substantial first hurdle…
Well, the file-format is documented, sure - for a diff/patch-facility and git-support, that’s nice, but doesn’t really help with real-time collaboration. For that, we need the ability to basically subscribe and block all events, then apply them in the order the server echoes them back.
Depending on how central the event-bus in the Renoise architecture is, internally, that could be either really easy, or really difficult - if basically every command/change goes through a unified event-bus, it should actually be quite easy.
… The tracker is basically a spreadsheet, so you can get a feel for how this could look and work with a spreadsheet in Google Docs - you can see the other “player’s” cursors (in different colors), you can see their edits in real-time, you can see them enter and leave the session, and you can chat with them. If you could see a ghosted version of the other collaborators’ mouse pointers too, that would be awesome. If the cursor left a subtle trail when changes were made (that fades or disappears after a few seconds), that would take some of the “spookiness” out, so you don’t suddenly realize that someone made a change where you were looking.
