Collaboration/syncronisation

Renoise Tool Development Renoise XML Fileformats & Tools
1

I’m not sure if you could do it ‘live’ with SOAP. But I do have an idea to create a usable ‘collaboration environment’. Which is technically quite acheivable.

This is the workflow I have in mind: (forgive me if I refer to myself in the 2nd person, I’m not going crazy, I swear twitch twitch :P)

  1. Via a web interface, MickRip initiates a “collaboration tune”. It’s stored remotely as a clean slate - the only significant data (thus far) is the “Song Name”, “ID”, and certain security parameters. MickRip is now the primary author, and he can now assign a collaboration team. He selects Conner_Bw. For now, it’s just a 2 person co-op.

  2. MickRip makes a start on his tune. Just like normal. He tracks the first 5 patterns. He saves as “backspace.xrns”. It’s 8mb in size.

  3. I’ve been figuring out how to get 8mb+ of data from my machine to a web server. Obviously the xrns file needs to be stored in a remote location. That in itself is easy, but looking ahead, we have syncronisation issues to consider.

[indent] .oO( … Thinking about how XRNS will be stored on a DB …)

  • All Flac files are stored separately in binary, on the database.
  • All Patterns are stored seperately. (DSP/Automation same umbrella?)
  • Instrument data (looping, envelopes)
  • Global data
  • VST info and parameters
  • What else? [/indent]

Sending 8mb over HTTP is a pain in the arse. Using FTP, although a bit better, but presents security issues. The best option I see, is create a local (client-side) peice of software which has the ability to …

[indent] - Break apart your tune into its significant/tangable components (flacs, patterns, instruments, dsp info, etc)

  • Probe the server, looking for differences between the server copy, and the local copy.
  • Upload only what is needed. Extra patterns, modifications, etc.
  • Become available for other authorisised participants to Sync with their local copies.[/indent]

The same peice of software should be able to work in reverse, and modify your local copy with any changes that are made on the server. Local software can be written and compiled in PHP. (Not sure about a mac)…

It seems to me, the most difficult aspect of this is turning the whole XRNS file into data, and then building an XRNS file from whatever it finds from the database. It’s not rocket science, but to me to represents the most time.

  1. (Back to reality) Once MickRip has uploaded his tune using the client side software, the tune should be on the server in a format which can be rebuilt into an XRNS file again. Conner_Bw’s client side software will make SOAP requests, first requesting the flac data, the patterns, the instruments and so on. The software will then build an XRNS file, ready for him to edit.

  2. Conner_Bw adds another 6 patterns. He’s also changed a snare sample, more suited to his tastes. He’s also lengthened my intro by a pattern, and changed some notes here and there. He’s also added a vocal sample. He then saves his work, (the tune is now 11mb) … he goes to our client side interface and presses the upload button.

Now, this is where it gets tricky and my brain hurts a bit. Obviously there’s not 11mb to upload, as most of the flac files are intact. So the sync software looks at the instrument data, compares local and remote “Horn Sample”, and will only upload if it there has been a modification. What if you swap instrument 1 and 2? Should I identify the instrument by name rather than the slot? What if I change the name? These issues need to be delt with.

It seems to me, the only solution is to append a change rather than overwrite. Conner_Bw’s drum sample that he replaced, should be simply appended, and the old drum sample should remain on the remote server. We should ID everything by checksum, and not by the internal ID inside the XML. This means that the syncronisation process SHOULD cope with people swapping things around, without the syncronisation engine deciding things are different. If you upload a completely different tune, technically it should append the tune to the end of the existing remote copy. If a pattern has been changed by ONE note, then a new copy of that pattern will be created.

This method facilitates an ultimated amount of authors. It’s a bit “messy” though, but I can’t see any other solution to cope with drastic changes.

These are only initial thoughts. I haven’t worked out if DSP info should be ammeded, or replaced.

Some thoughts would be helpful.

Regards,
Mick :)

2

How about this:
1.) you download the song
2.) you edit a copy of it
3.) instead of de/reconstructing the xrns (why… unless you want the option to synchronize only various parts of the song?), simply generate a diff between the xml of 1.) and 2.) and upload that.
4.) other clients download the .diff (and added/changed samples plus a list of those which were removed) and apply the “patch” locally.

For samples use checksums, and as said I don’t think you need to handle instruments or anything specifically.

3

I don’t think there’s any way to avoid deconstructing/reconstructing the xrns file, no matter how you look at it. Even if you want to avoid the complicated type syncronisation I was suggesting before. (which is looking less appealing all the time :P).

… Maybe my time is better spent making music …

Anyway, to “generate the diff” is a technical process. To apply a patch is a technical processes as well. Both require routines which turn the xrns into information it can understand. You can’t compare 2 xrns without first breaking it down - you can’t “insert” the new changes without doing the same thing.

I’m deciding it’s all too ambitious for now. Workflow friendly sycronisation for Renoise is a long way off. I had visions on many people being able to work on the one tune at once at the same time, but it all sounds too esoteric. A co-op system which mirrors the tried and tested “taking turns” method, is quite easy in comparison, but I don’t imagine many people seeing the need.

Maybe I’ll see the need one day, but if there’s ever time to exclaim the phrase “f**** it”, then it’s now…

4

Cool!

That sounds like a Renoise source code version control system to me. (CVS/Subversion/Perforce/Whatever Flavour you know) I’m pretty sure you could tie your idea into one of those systems.

I think being able to load a Renoise module as a directory, instead of as just a .zip file representing a directory, would solve the issue. The CVS would handle the differences, merge, and sharing issues.

If a song is a bunch of files in a directory, and a user checks out the XML to work on a song, then checks it back in when they are done, the CVS can tell the user if things needs to be merged, replaced, whatever. If the user needs to manipulate a sample, instrument, some other part of the song, he checks that part out as well. If it’s a new sample, he/she would do a submit, just like you would in software development. (p4 submit, p4 edit, p4 sync was my daily routine for the last year: http://en.wikipedia.org/wiki/Perforce if i could unzip my renoise song and was able to load the resulting parts into the software, i don’t see why this wouldn’t work as a multi user project)

A lot of the problems you are trying to solve exist in software development and CVS solves them. The binary data is the tricky part, but if the user explicitly checks-out/checks-in each part of a Renoise file/directory for every modification, it would be like collaborating on a big software project.

It’s doable.

5

To expand on the idea (basically a rehash of daily CVS/Perforce experiences with software projects)

Users would check out songs into a “Sandbox”, mess around with them, check them back in to the “Repository”.

Several users could do this at once, with very little coordination. If they don’t like a version of the song, they can revert, branch, fork, just like software. If 1 user checks something in between the time he checked something out and another user checked something in, the CVS offers merge facilities.

If they screw something up, they can force sync to the latest version, or revert to a version that worked and overwrite the head revision to keep working on where the song is at it’s current point.

The key to facilitating all of this is to break the Renoise song into files you can check in and out, load a song by double clicking on a directory, and Renoise has to be aware that “The song has changed, would you like to reload?”

Basically, you run Renoise in one window and something like this in another. Renoise becomes like an IDE responding to changes in your sandbox which you can sync to the server or change on the fly.

Good times?

6

Yeah, definately do’able. I have no experience in the CVS realm. But lets say we were both working on the same class at the same time. We both make wholesale changes to certain functions in that class, and we both add more functions.

I upload my version of the same class, and you upload your version of the same class.

How does the CVS model deal with that? Does your class file overwrite mine?

7

It would tell me something has changed, and I would have to manually intervene.It would give me a diff option to see what has changed, a merge option (which usually sucks), an abort so i could try to manually merge your changes, an overwrite, or a manual edit on the fly option that would submit on save.

If I screw it up, you/and I could just pull back to the previous revision. Say file.flac in head is now #5 and it’s royally screwed up, would do p4 sync /dir/to/file.flac#4 which would change my “Sandbox” copy to #4, i would fix it, force submit that and it becomes #6. So next time you sync, we both have #6.

If we are chatting in IRC, we agree and move on.

:)

8

But you absolutely can (… unless you want to apply only certain changes, which I think doesn’t apply here).

http://en.wikipedia.org/wiki/Diff

No understanding required of whatever it is you’re generating a diff from.

Of course a CVS would make this moot → great idea!! But still, for the sake of correctness I had to insist :)

9

I see where you’re coming from. If people were adding patterms, that’s pretty easy. But what about if I changed the EQ of a pattern?

Turning the “diff” into something practical is tricky. To sync 2 text files seems easier (on the surface) than a 6mb XML file.

You could ‘diff’ the XML file, or you can parse the XML into meaningful data and ‘diff’ that in a propietry way. (which is far as I can see, is the only way)…

The other option is to prevent 2 people from working on the file at once. Someone is the designated updater of the tune, and any other updates are meaningless.

Conner’s idea of treating the zip file as a directory structure and syncing that seems like the easiest way. Treat the XML as one file; when you compress it, it’s not that big anyway.

Having complicated ammendment/syncing schemas is a pain in the arse.

10

This wasn’t actually what I was trying to say, I was saying the Renoise file as it is now if unzipped could be managed via CVS. But, this is definitely a better idea! If the XML nodes were tree structure in a CVS that might even be better!

Yes, I think making something that everyone gets will be very difficult. But, any two or more composers familiar with CVS could collaborate starting TODAY if they could load directory structures into Renoise and Renoise could respond if something on the HD changed.

The problem, today, would be trying to visually sort out the differences between two XML files. I mean, I work 3 hours on a song, then try to do a submit, and i’m shown some XML diff because there is a conflict due to another user checking something else in between, i don’t know if i’d be able to figure it out. If I wrote XML for 3 hours I might, but the XML is the results of me working in an interface that has nothing to do with writing XML code.

The diff/merge schema would be the most difficult challenge. Your interpretation of breaking the XML tree into a directory tree compatible with CVS might be the first step in the right direction.

11

Trying to rebuild the whole XML file from stored information is quite ambitious for someone “with a life”. It’s all conceptually possible in our heads, we can do anything if you design it well enough. At the end of the day however, to make a collaboration system which locks the file for only person to update at a time is about a quarter of the work. At the beginning, I had visions of a “anyone can submit anytime” system being challenging but easy enough, but as the possible collaboration scenarios started creeping into my head, I realised that an xrns ‘CVS’ system with differential ammendment and syncing processes was too ‘out there’. Working on the same pattern at once is very different (in the real world) to working on the same function/class in a programming environment. An xrns syncing system using a ‘one-author-at-a-time’ method is that if you treat the directory structure as a zip file, then the chances of a system like this being finished becomes realistic.

Even with syncing the contents of the xrns/zip file, there’s still some pretty nifty possiblities. You could work on a tune and “share it” with 10 other people who give you critique. People could subscribe to your work-in-progress tune, and your updates are syncing in an arbitary number of locations. You could use this service as a collaboration tool, although you’re the only person tracking. The other collaborators give advice only.

Speaking on “time better spent”, maybe there’s merit in using a system like this to syncronise other features. Perhaps to upload/manage a renoise instrument repository instead. So rather than xrns, maybe xrni instead… if the system works well, then maybe it can be offered native inside the Renoise application. You could make the soap service into a virtual drive when hunting for new samples/instruments.

12

No difference, since in both cases it’s just (assuming you found/adopted code for those two functions, to my surprise I didn’t find anything straightforward for php, at least not at a glance)

$diff = diff($old_xml, $new_xml)
and
$new_xml = patch($old_xml, $diff)

You see, diff handles everything. Who knows, maybe the binary version is even efficient enough to handle samples, as well?

As you say, on the surface If it can do it for a 10 byte string, it can do it for a 10 megabyte string, too. It just takes more time and memory. Again, assuming you find an algorithm you can use and don’t have to reinvent the wheel. But you don’t, so cheers :)

Then there simply has to be a “project option”: either the song gets locked when someone is editing it (with a timeout? :>), or when such a conflict arises a new branch gets started. Yes, that might be less powerful but at least it’s doable. And something that actually sees the light of day is better than a neat idea… I say that because I’m too lazy to code (the possibilites and what I get done drive me mad so I turn to the arts for peace :lol:), not to offend anyone!

13

PHP has xdiff:
http://ca.php.net/manual/en/ref.xdiff.php

I’m going to bed, but keep the ideas coming! :)

14

Ok, so far there’s 2 primary ways to design multi-author online xrns sync.

  1. Many authors can work on the file at the one time… If me and you are working on Pattern 5, and you decide to modify the snare and insert a track, and modify the EQ. And I decide to add 3 tracks, and I add a reverb DSP to track 6 - and we upload at the same time, the system tried to ammend both of our ideas into the one track. The system would likely turn pattern 5 into 2 separate patterns - label one as mine, and the other as yours. It’s chaos in the making.

  2. Only one author can work on a track at once … which means those issues in method 1 will not exist. Only me or you can work on the file (or anyone else for that matter) - If i’ve been charged with working on the file, then nobody can update the file but me.

Idea 1 seemed cool, but after some thought, I decided it’s too ambious and a pain in the arse. The reason why I wanted to pull apart the XML file was so I could deal with the complicated issues of idea 1. You can’t simply diff it, as the XML requires interpreting so it knows that pattern 5 needs duplicating. We’d need to develop a heavily propietry ‘diff’…And even if we developed an amazing algorithm, it was still going to be chaos. Life is too short.

Idea 2 seems a lot more realistic, and involves NO xml parsing whatsoever. Because XML files (compressed) are small, there’s no need to deal with the differences in the XML file, therefore no need to analyse the XML file at all. It’s just a file, like a flac file. You treat the xrns as a directory structure and send/receive the differences. If the XML file has changed, send the entire XML file.

15

Here are some ideas to add some value to this simpler #2 idea.

  1. File history is kept, file can be reverted and restored based on a timestamp query. I want the song from 48 hours ago? Done.

  2. Deadlines. The user gets an email or and SMS message saying their time is up and if they don’t submit their changes within 30 minutes, another user (in a queue?) will have the next exclusive access window to the file. If I’m collaborating, I don’t want my partner to hog all the time on the track. Time windows can be enforced?

  3. Statistics can be generated from the revision history, make cool collaboration chats that look like MS Project time lines. Who worked on what, when? A visual representation. Pie charts? Changes over time? Or at least a (SOAP/XML?) interface to the data so third party users can query the Renoise Collaboration server to gather statistics?

… ?

16

File History. Hmm. I was thinking that. Now the only problem with this is database overhead. I was thinking more of storing the file history on the client side. So … “mytune_081507_1546.xrns”, “mytune_091507_1115.xrns” … obviously the sync app will look at “mytune.xrns” as the tune to sync…

Doing it the database has more advantages though, although if you’re working on a 50mb file - ohh man… 10 versions of the 50mb file is going to be paaainful on mysql. 50mb is painful enough.

The deadline idea! Great idea!!! Time windows are good, I can’t stand waiting around for days and days. I want to track, and I want to now. :P

The visual stuff sounds like a good “rainy day” idea.

17

I’ve been having some of the same thoughts, so I’ll chip in with the approach I had in mind.

I would write up a “session server”, an application that is responsible for maintaining collaboration session state - keeping track of who’s participating, the changes they contributed to patterns, instruments, samples, etc.

I believe the biggest problem here is how to keep things perfectly synchronized. I think we need to think in terms of “eventual consistency” - a data replication principle that is frequently used in clustered databases.

In a sense, you can think of the “session server” as the “database” for the song - and each collaborating client needs to be able to replicate changes submitted to the “session server”, resulting in eventual consistency; every copy of the song being up to date.

The internet being pretty fast these days, we’re talking about synchronization problems that are probably going to be pretty rare - of course, these must be handled, or you would end up with differing versions of the same song on different clients, but with a fast internet connection (or especially on a LAN connection) the number of conflicts encountered should be pretty low.

Here’s an article explaining various patterns for implementing eventual consistency:

I’m not sure which variation would be the most appropriate.

Under any circumstances, I think the pattern-consistency needs to be implemented at the track-level - for example, for every change you submit, the session server would assign it a running serial number unique to that particular track in that particular pattern, and check for a conflict; e.g. check if another user submitted a change to that track/pattern that has not yet been streamed back to the other clients - if so, you have a conflict.

Now, if every client assigns a serial number to every change as it’s made locally, before submitting it to the session server, a conflict can be resolved by telling the client in violation that their change was rejected - the client can then simply undo from it’s current local change number back to the change number in violation; the last change (or maybe a couple of changes) would “roll back” on that client. And with the “rejection” response from the session server, that client would receive the conflicting changes, and apply them immidiately after undoing the rejected changes.

For changes to instruments, samples, automation, etc. the same mechanics can be applied - locking and conflict resolution would happen at the instrument/sample/automation-graph level.

Anyway, those are my thoughts for the time being.

Network tracking was a dream of mine since 15+ years, so I was quite excited to see Renoise adding a scripting engine with a network API. I think it’s possible! And the internet is fast enough to make this truly useful now.

I regret that I’m no longer making music. Although the prospect of being able to collaborate with friends abroad is a tempting prospect that makes me think about picking up tracking again :wink:

18

It’s all about keeping things up to date. What about storing only the diffs instead of storing the whole song zip ? Tip

19

I’d like to chip in an idea, because this seems all too close to a tool I use every day. I’m an architect by profession, and the application camp I’m in is the Graphisoft one; namely ArchiCAD. While the program can on occasion be a pain in the side, it has one really handy feature and that is the current, new version of TeamWork (add copyright sign)(add registered trademark sign)(add corporate logo)(no animals were harmed). It basically means that several people can work on the same project, which is held as a database(of some sort) on a dedicated server. Like this thing you’re discussing. And the way this is pulled off, without the chaos implied in the scenario, is by reservations policy. Everyone can work on the same project, but only if they have “reserved” that specific thing they need to work on. After the thing one reserved has been completed, the reservation on the thing can be released, submitting the changes made. Changes made by the person working on the reserved thing can be submitted at any time, in which case only the ‘delta’ (or diff?) will be sent and handled by the server. Other changes, made by other people working on the project, can be “fetched” likewise. The main catch on this approach is that the reservations on things must be kept in sync. And there’s a local copy of the whole project.
While I have zero knowledge on actual networking mechanics, I’d imagine this COULD be a viable solution for working on renoise tracks too. It could mean for example that one could reserve a specific pattern to work on. Or reserve the comments-section. Or a specific instrument. One feature of said TeamWork©®(x) is that users can freely add stuff, while modifying and removing things must be via reservations.
Well. Just wanted to note this ‘partial reservations stuff’ as an application on the very subject being discussed. Carry on.

20

i think we have the same idea in mind but let me clarify what i think:
most of the suggestions have something to do with the project file being treated as a directory, unpacked, the changes diffed, etc. the CVS approach is ok, if it works for software why shouldn’t it work for other things.

since the scripting engine allows for networking and offers most of the editing functionalities, a clean approach that would even allow for realtime collaboration would be:

  1. person A connects to a server
  2. Renoise monitors all of person A’s edits (events) (i don’t think this is implemented in the API but it shouldn’t be that difficult)
  3. the edits are translated either to XML or a smaller non-readable format (see marshalling) and sent to the server. if someone adds a sample it is uploaded to the server and kept there until all people participating in the collab have downloaded it, then it is deleted. deleting a sample in your project just sends a message to all clients to remove (maybe not physically) the sample from the project.
    two important things: a history is made and uploaded to the server for all clients that were not connected while the changes were made. everything at this point is being compressed!
  4. person B receives the changes, the samples are downloaded, and they are applied by using the API again. this could technically happen (almost) in realtime or by a notification that a new version is available. this way a lot of people could participate and even watch what’s happening in realtime.

throw in the IRC plugin and you’re good to go. ^^

that’s how i would expect an idea like that to work and it does go in the direction of a SOAP approach but i would make it more dynamic … unless you really want to keep it basic.