Jump to content


Photo

Collaboration/syncronisation


  • Please log in to reply
33 replies to this topic

#1 MickRip

MickRip

    Local Chief Member

  • Normal Members
  • PipPipPipPipPip
  • 231 posts

Posted 23 March 2007 - 02:57

Can someone write a Renoise SOAP server? We can finally collaborate using the HTTP protocol! Live mixing into Apache/MySQL.

Anyone?


I'm not sure if you could do it 'live' with SOAP. But I do have an idea to create a usable 'collaboration environment'. Which is technically quite acheivable.

This is the workflow I have in mind: (forgive me if I refer to myself in the 2nd person, I'm not going crazy, I swear *twitch twitch* :P)
------------------------------------------------------------------------
1. Via a web interface, MickRip initiates a "collaboration tune". It's stored remotely as a clean slate - the only significant data (thus far) is the "Song Name", "ID", and certain security parameters. MickRip is now the primary author, and he can now assign a collaboration team. He selects Conner_Bw. For now, it's just a 2 person co-op.

2. MickRip makes a start on his tune. Just like normal. He tracks the first 5 patterns. He saves as "backspace.xrns". It's 8mb in size.

3. I've been figuring out how to get 8mb+ of data from my machine to a web server. Obviously the xrns file needs to be stored in a remote location. That in itself is easy, but looking ahead, we have syncronisation issues to consider.

.oO( .. Thinking about how XRNS will be stored on a DB ..)
- All Flac files are stored separately in binary, on the database.
- All Patterns are stored seperately. (DSP/Automation same umbrella?)
- Instrument data (looping, envelopes)
- Global data
- VST info and parameters
- What else?


Sending 8mb over HTTP is a pain in the arse. Using FTP, although a bit better, but presents security issues. The best option I see, is create a local (client-side) peice of software which has the ability to ..

- Break apart your tune into its significant/tangable components (flacs, patterns, instruments, dsp info, etc)
- Probe the server, looking for differences between the server copy, and the local copy.
- Upload only what is needed. Extra patterns, modifications, etc.
- Become available for other authorisised participants to Sync with their local copies.


The same peice of software should be able to work in reverse, and modify your local copy with any changes that are made on the server. Local software can be written and compiled in PHP. (Not sure about a mac)..

It seems to me, the most difficult aspect of this is turning the whole XRNS file into data, and then building an XRNS file from whatever it finds from the database. It's not rocket science, but to me to represents the most time.

4. (Back to reality) Once MickRip has uploaded his tune using the client side software, the tune should be on the server in a format which can be rebuilt into an XRNS file again. Conner_Bw's client side software will make SOAP requests, first requesting the flac data, the patterns, the instruments and so on. The software will then build an XRNS file, ready for him to edit.

5. Conner_Bw adds another 6 patterns. He's also changed a snare sample, more suited to his tastes. He's also lengthened my intro by a pattern, and changed some notes here and there. He's also added a vocal sample. He then saves his work, (the tune is now 11mb) .. he goes to our client side interface and presses the upload button.

Now, this is where it gets tricky and my brain hurts a bit. Obviously there's not 11mb to upload, as most of the flac files are intact. So the sync software looks at the instrument data, compares local and remote "Horn Sample", and will only upload if it there has been a modification. What if you swap instrument 1 and 2? Should I identify the instrument by name rather than the slot? What if I change the name? These issues need to be delt with.

It seems to me, the only solution is to append a change rather than overwrite. Conner_Bw's drum sample that he replaced, should be simply appended, and the old drum sample should remain on the remote server. We should ID *everything* by checksum, and not by the internal ID inside the XML. This means that the syncronisation process SHOULD cope with people swapping things around, without the syncronisation engine deciding things are different. If you upload a completely different tune, technically it should append the tune to the end of the existing remote copy. If a pattern has been changed by ONE note, then a new copy of that pattern will be created.

This method facilitates an ultimated amount of authors. It's a bit "messy" though, but I can't see any other solution to cope with drastic changes.

These are only initial thoughts. I haven't worked out if DSP info should be ammeded, or replaced.

Some thoughts would be helpful.

Regards,
Mick :)

Edited by MickRip, 23 March 2007 - 02:58.

  • Jalex likes this

#2 Johann

Johann

    Guruh Motha Fakka is Levitating and Knows Everything About Renoise Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • 2774 posts
  • Gender:Not Telling

Posted 23 March 2007 - 09:42

Sending 8mb over HTTP is a pain in the arse. Using FTP, although a bit better, but presents security issues. The best option I see, is create a local (client-side) peice of software which has the ability to ..

- Break apart your tune into its significant/tangable components (flacs, patterns, instruments, dsp info, etc)
- Probe the server, looking for differences between the server copy, and the local copy.
- Upload only what is needed. Extra patterns, modifications, etc.
- Become available for other authorisised participants to Sync with their local copies.

How about this:
1.) you download the song
2.) you edit a *copy* of it
3.) instead of de/reconstructing the xrns (why... unless you want the option to synchronize only various parts of the song?), simply generate a diff between the xml of 1.) and 2.) and upload that.
4.) other clients download the .diff (and added/changed samples plus a list of those which were removed) and apply the "patch" locally.

What if you swap instrument 1 and 2? Should I identify the instrument by name rather than the slot?

For samples use checksums, and as said I don't think you need to handle instruments or anything specifically.

music / photos / renoise scripts


hay guises! action/reaction. I'm not into leaving or avoiding places, I ban 'em by making sure I get banned.. loadsa noise, little signal.. basically anything else makes more sense to spend energy on. thanks for the app, but from here on out we simply don't have any use for each other period. oh, and of course big shoutouts to bit-arts and kaneel and anyone else I forgot :D :D

#3 MickRip

MickRip

    Local Chief Member

  • Normal Members
  • PipPipPipPipPip
  • 231 posts

Posted 24 March 2007 - 02:35

How about this:
1.) you download the song
2.) you edit a *copy* of it
3.) instead of de/reconstructing the xrns (why... unless you want the option to synchronize only various parts of the song?), simply generate a diff between the xml of 1.) and 2.) and upload that.
4.) other clients download the .diff (and added/changed samples plus a list of those which were removed) and apply the "patch" locally.


I don't think there's any way to avoid deconstructing/reconstructing the xrns file, no matter how you look at it. Even if you want to avoid the complicated type syncronisation I was suggesting before. (which is looking less appealing all the time :P).

.. Maybe my time is better spent making music ..

Anyway, to "generate the diff" is a technical process. To apply a patch is a technical processes as well. Both require routines which turn the xrns into information it can understand. You can't compare 2 xrns without first breaking it down - you can't "insert" the new changes without doing the same thing.

I'm deciding it's all too ambitious for now. Workflow friendly sycronisation for Renoise is a long way off. I had visions on many people being able to work on the one tune at once at the same time, but it all sounds too esoteric. A co-op system which mirrors the tried and tested "taking turns" method, is quite easy in comparison, but I don't imagine many people seeing the need.

Maybe I'll see the need one day, but if there's ever time to exclaim the phrase "f**** it", then it's now..

Edited by MickRip, 24 March 2007 - 02:37.


#4 Conner_Bw

Conner_Bw

    Probably More God or Borg Than Human Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 7091 posts
  • Gender:Male
  • Location:Montreal, Quebec, Canada

Posted 24 March 2007 - 03:24

Cool!

That sounds like a Renoise source code version control system to me. (CVS/Subversion/Perforce/Whatever Flavour you know) I'm pretty sure you could tie your idea into one of those systems.

I think being able to load a Renoise module as a directory, instead of as just a .zip file representing a directory, would solve the issue. The CVS would handle the differences, merge, and sharing issues.

If a song is a bunch of files in a directory, and a user checks out the XML to work on a song, then checks it back in when they are done, the CVS can tell the user if things needs to be merged, replaced, whatever. If the user needs to manipulate a sample, instrument, some other part of the song, he checks that part out as well. If it's a new sample, he/she would do a submit, just like you would in software development. (p4 submit, p4 edit, p4 sync was my daily routine for the last year: http://en.wikipedia.org/wiki/Perforce if i could unzip my renoise song and was able to load the resulting parts into the software, i don't see why this wouldn't work as a multi user project)

A lot of the problems you are trying to solve exist in software development and CVS solves them. The binary data is the tricky part, but if the user explicitly checks-out/checks-in each part of a Renoise file/directory for every modification, it would be like collaborating on a big software project.

It's doable.

Edited by Conner_Bw, 24 March 2007 - 03:29.

cpu Lenovo X220, Intel i7-2640M @ 2.80GHz ×4 os Windows 10 / Ubuntu 16.04 LTS
My Homepage » : My Renoise Tools » : Normalize Your Sig » : One million clicks! »


#5 Conner_Bw

Conner_Bw

    Probably More God or Borg Than Human Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 7091 posts
  • Gender:Male
  • Location:Montreal, Quebec, Canada

Posted 24 March 2007 - 03:43

To expand on the idea (basically a rehash of daily CVS/Perforce experiences with software projects)

Users would check out songs into a "Sandbox", mess around with them, check them back in to the "Repository".

Several users could do this at once, with very little coordination. If they don't like a version of the song, they can revert, branch, fork, just like software. If 1 user checks something in between the time he checked something out and another user checked something in, the CVS offers merge facilities.

If they screw something up, they can force sync to the latest version, or revert to a version that worked and overwrite the head revision to keep working on where the song is at it's current point.

The key to facilitating all of this is to break the Renoise song into files you can check in and out, load a song by double clicking on a directory, and Renoise has to be aware that "The song has changed, would you like to reload?"

Basically, you run Renoise in one window and something like this in another. Renoise becomes like an IDE responding to changes in your sandbox which you can sync to the server or change on the fly.

Good times?

Edited by Conner_Bw, 24 March 2007 - 03:46.

cpu Lenovo X220, Intel i7-2640M @ 2.80GHz ×4 os Windows 10 / Ubuntu 16.04 LTS
My Homepage » : My Renoise Tools » : Normalize Your Sig » : One million clicks! »


#6 MickRip

MickRip

    Local Chief Member

  • Normal Members
  • PipPipPipPipPip
  • 231 posts

Posted 24 March 2007 - 03:46

It's doable.


Yeah, definately do'able. I have no experience in the CVS realm. But lets say we were both working on the same class at the same time. We both make wholesale changes to certain functions in that class, and we both add more functions.

I upload my version of the same class, and you upload your version of the same class.

How does the CVS model deal with that? Does your class file overwrite mine?

#7 Conner_Bw

Conner_Bw

    Probably More God or Borg Than Human Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 7091 posts
  • Gender:Male
  • Location:Montreal, Quebec, Canada

Posted 24 March 2007 - 03:50

How does the CVS model deal with that? Does your class file overwrite mine?


It would tell me something has changed, and I would have to manually intervene.It would give me a diff option to see what has changed, a merge option (which usually sucks), an abort so i could try to manually merge your changes, an overwrite, or a manual edit on the fly option that would submit on save.

If I screw it up, you/and I could just pull back to the previous revision. Say file.flac in head is now #5 and it's royally screwed up, would do p4 sync /dir/to/file.flac#4 which would change my "Sandbox" copy to #4, i would fix it, force submit that and it becomes #6. So next time you sync, we both have #6.

If we are chatting in IRC, we agree and move on.

:)

Edited by Conner_Bw, 24 March 2007 - 03:51.

cpu Lenovo X220, Intel i7-2640M @ 2.80GHz ×4 os Windows 10 / Ubuntu 16.04 LTS
My Homepage » : My Renoise Tools » : Normalize Your Sig » : One million clicks! »


#8 Johann

Johann

    Guruh Motha Fakka is Levitating and Knows Everything About Renoise Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • 2774 posts
  • Gender:Not Telling

Posted 24 March 2007 - 03:57

Anyway, to "generate the diff" is a technical process. To apply a patch is a technical processes as well. Both require routines which turn the xrns into information it can understand. You can't compare 2 xrns without first breaking it down - you can't "insert" the new changes without doing the same thing.

But you absolutely can (.. unless you want to apply only certain changes, which I think doesn't apply here).

http://en.wikipedia.org/wiki/Diff

In computing, diff is a file comparison utility that outputs the differences between two files. The program displays the changes made per line for text files. Modern implementations also support binary files. The output is called a diff or more commonly a patch since the output can be applied with the program patch.

No understanding required of whatever it is you're generating a diff from.

Of course a CVS would make this moot -> great idea!! But still, for the sake of correctness I had to insist :)

music / photos / renoise scripts


hay guises! action/reaction. I'm not into leaving or avoiding places, I ban 'em by making sure I get banned.. loadsa noise, little signal.. basically anything else makes more sense to spend energy on. thanks for the app, but from here on out we simply don't have any use for each other period. oh, and of course big shoutouts to bit-arts and kaneel and anyone else I forgot :D :D

#9 MickRip

MickRip

    Local Chief Member

  • Normal Members
  • PipPipPipPipPip
  • 231 posts

Posted 24 March 2007 - 04:11

But you absolutely can (.. unless you want to apply only certain changes, which I think doesn't apply here).

http://en.wikipedia.org/wiki/Diff

No understanding required of whatever it is you're generating a diff from.

Of course a CVS would make this moot -> great idea!! But still, for the sake of correctness I had to insist :)


I see where you're coming from. If people were adding patterms, that's pretty easy. But what about if I changed the EQ of a pattern?

Turning the "diff" into something practical is tricky. To sync 2 text files seems easier (on the surface) than a 6mb XML file.

You could 'diff' the XML file, or you can parse the XML into meaningful data and 'diff' that in a propietry way. (which is far as I can see, is the only way)..

The other option is to prevent 2 people from working on the file at once. Someone is the designated updater of the tune, and any other updates are meaningless.

Conner's idea of treating the zip file as a directory structure and syncing that seems like the easiest way. Treat the XML as one file; when you compress it, it's not that big anyway.

Having complicated ammendment/syncing schemas is a pain in the arse.

#10 Conner_Bw

Conner_Bw

    Probably More God or Borg Than Human Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 7091 posts
  • Gender:Male
  • Location:Montreal, Quebec, Canada

Posted 24 March 2007 - 04:25

Conner's idea of treating the zip file as a directory structure and syncing that seems like the easiest way. Treat the XML as one file; when you compress it, it's not that big anyway.


This wasn't actually what I was trying to say, I was saying the Renoise file as it is now if unzipped could be managed via CVS. But, this is definitely a better idea! If the XML nodes were tree structure in a CVS that might even be better!

Having complicated ammendment/syncing schemas is a pain in the arse.


Yes, I think making something that everyone gets will be very difficult. But, any two or more composers familiar with CVS could collaborate starting TODAY if they could load directory structures into Renoise and Renoise could respond if something on the HD changed.

The problem, today, would be trying to visually sort out the differences between two XML files. I mean, I work 3 hours on a song, then try to do a submit, and i'm shown some XML diff because there is a conflict due to another user checking something else in between, i don't know if i'd be able to figure it out. If I wrote XML for 3 hours I might, but the XML is the results of me working in an interface that has nothing to do with writing XML code.

The diff/merge schema would be the most difficult challenge. Your interpretation of breaking the XML tree into a directory tree compatible with CVS might be the first step in the right direction.

Edited by Conner_Bw, 24 March 2007 - 04:28.

cpu Lenovo X220, Intel i7-2640M @ 2.80GHz ×4 os Windows 10 / Ubuntu 16.04 LTS
My Homepage » : My Renoise Tools » : Normalize Your Sig » : One million clicks! »


#11 MickRip

MickRip

    Local Chief Member

  • Normal Members
  • PipPipPipPipPip
  • 231 posts

Posted 24 March 2007 - 05:49

The problem, today, would be trying to visually sort out the differences between two XML files. I mean, I work 3 hours on a song, then try to do a submit, and i'm shown some XML diff because there is a conflict due to another user checking something else in between, i don't know if i'd be able to figure it out. If I wrote XML for 3 hours I might, but the XML is the results of me working in an interface that has nothing to do with writing XML code.


Trying to rebuild the whole XML file from stored information is quite ambitious for someone "with a life". It's all conceptually possible in our heads, we can do anything if you design it well enough. At the end of the day however, to make a collaboration system which locks the file for only person to update at a time is about a quarter of the work. At the beginning, I had visions of a "anyone can submit anytime" system being challenging but easy enough, but as the possible collaboration scenarios started creeping into my head, I realised that an xrns 'CVS' system with differential ammendment and syncing processes was too 'out there'. Working on the same pattern at once is very different (in the real world) to working on the same function/class in a programming environment. An xrns syncing system using a 'one-author-at-a-time' method is that if you treat the directory structure as a zip file, then the chances of a system like this being finished becomes realistic.

Even with syncing the contents of the xrns/zip file, there's still some pretty nifty possiblities. You could work on a tune and "share it" with 10 other people who give you critique. People could subscribe to your work-in-progress tune, and your updates are syncing in an arbitary number of locations. You could use this service as a collaboration tool, although you're the only person tracking. The other collaborators give advice only.

Speaking on "time better spent", maybe there's merit in using a system like this to syncronise other features. Perhaps to upload/manage a renoise instrument repository instead. So rather than xrns, maybe xrni instead... if the system works well, then maybe it can be offered native inside the Renoise application. You could make the soap service into a virtual drive when hunting for new samples/instruments.

#12 Johann

Johann

    Guruh Motha Fakka is Levitating and Knows Everything About Renoise Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • 2774 posts
  • Gender:Not Telling

Posted 24 March 2007 - 06:08

I see where you're coming from. If people were adding patterms, that's pretty easy. But what about if I changed the EQ of a pattern?

No difference, since in both cases it's just (assuming you found/adopted code for those two functions, to my surprise I didn't find anything straightforward for php, at least not at a glance)

$diff = diff($old_xml, $new_xml)
and
$new_xml = patch($old_xml, $diff)

You see, diff handles everything. Who knows, maybe the binary version is even efficient enough to handle samples, as well?

Turning the "diff" into something practical is tricky. To sync 2 text files seems easier (on the surface) than a 6mb XML file.

As you say, on the surface If it can do it for a 10 byte string, it can do it for a 10 megabyte string, too. It just takes more time and memory. Again, assuming you find an algorithm you can use and don't have to reinvent the wheel. But you don't, so cheers :)

The problem, today, would be trying to visually sort out the differences between two XML files. I mean, I work 3 hours on a song, then try to do a submit, and i'm shown some XML diff because there is a conflict due to another user checking something else in between, i don't know if i'd be able to figure it out.

Then there simply has to be a "project option": either the song gets locked when someone is editing it (with a timeout? :>), or when such a conflict arises a new branch gets started. Yes, that might be less powerful but at least it's doable. And something that actually sees the light of day is better than a neat idea.. I say that because *I'm* too lazy to code (the possibilites and what I get done drive me mad so I turn to the arts for peace :lol:), not to offend anyone!

music / photos / renoise scripts


hay guises! action/reaction. I'm not into leaving or avoiding places, I ban 'em by making sure I get banned.. loadsa noise, little signal.. basically anything else makes more sense to spend energy on. thanks for the app, but from here on out we simply don't have any use for each other period. oh, and of course big shoutouts to bit-arts and kaneel and anyone else I forgot :D :D

#13 Conner_Bw

Conner_Bw

    Probably More God or Borg Than Human Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 7091 posts
  • Gender:Male
  • Location:Montreal, Quebec, Canada

Posted 24 March 2007 - 06:20

(assuming you found/adopted code for those two functions, to my surprise I didn't find anything straightforward for php, at least not at a glance)


PHP has xdiff:
http://ca.php.net/ma...n/ref.xdiff.php

I'm going to bed, but keep the ideas coming! :)

cpu Lenovo X220, Intel i7-2640M @ 2.80GHz ×4 os Windows 10 / Ubuntu 16.04 LTS
My Homepage » : My Renoise Tools » : Normalize Your Sig » : One million clicks! »


#14 MickRip

MickRip

    Local Chief Member

  • Normal Members
  • PipPipPipPipPip
  • 231 posts

Posted 24 March 2007 - 06:37

No difference, since in both cases it's just (assuming you found/adopted code for those two functions, to my surprise I didn't find anything straightforward for php, at least not at a glance)

$diff = diff($old_xml, $new_xml)
and
$new_xml = patch($old_xml, $diff)

You see, diff handles everything. Who knows, maybe the binary version is even efficient enough to handle samples, as well?

As you say, on the surface If it can do it for a 10 byte string, it can do it for a 10 megabyte string, too. It just takes more time and memory. Again, assuming you find an algorithm you can use and don't have to reinvent the wheel. But you don't, so cheers :)


Ok, so far there's 2 primary ways to design multi-author online xrns sync.

1. Many authors can work on the file at the *one time*.. If me and you are working on Pattern 5, and you decide to modify the snare and insert a track, and modify the EQ. And I decide to add 3 tracks, and I add a reverb DSP to track 6 - and we upload at the same time, the system tried to ammend both of our ideas into the one track. The system would likely turn pattern 5 into 2 separate patterns - label one as mine, and the other as yours. It's chaos in the making.

2. Only one author can work on a track at once .. which means those issues in method 1 will not exist. Only me *or* you can work on the file (or anyone else for that matter) - If i've been charged with working on the file, then nobody can update the file but me.

------------

Idea 1 seemed cool, but after some thought, I decided it's too ambious and a pain in the arse. The reason why I wanted to pull apart the XML file was so I could deal with the complicated issues of idea 1. You can't simply diff it, as the XML requires interpreting so it knows that pattern 5 needs duplicating. We'd need to develop a heavily propietry 'diff'..And even if we developed an amazing algorithm, it was still going to be chaos. Life is too short.

Idea 2 seems a lot more realistic, and involves NO xml parsing whatsoever. Because XML files (compressed) are small, there's no need to deal with the differences in the XML file, therefore no need to analyse the XML file at all. It's just a file, like a flac file. You treat the xrns as a directory structure and send/receive the differences. If the XML file has changed, send the entire XML file.

Edited by MickRip, 24 March 2007 - 06:52.


#15 Conner_Bw

Conner_Bw

    Probably More God or Borg Than Human Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 7091 posts
  • Gender:Male
  • Location:Montreal, Quebec, Canada

Posted 24 March 2007 - 06:54

Idea 2 seems a lot more realistic, and involves NO xml parsing whatsoever. Because XML files (compressed) are small, there's no need to deal with the differences in the XML file, therefore no need to analyse the XML file at all. It's just a file, like a flac file. You treat the xrns as a directory structure and send/receive the differences. If the XML file has changed, sent the entire XML file.


Here are some ideas to add some value to this simpler #2 idea.

1) File history is kept, file can be reverted and restored based on a timestamp query. I want the song from 48 hours ago? Done.

2) Deadlines. The user gets an email or and SMS message saying their time is up and if they don't submit their changes within 30 minutes, another user (in a queue?) will have the next exclusive access window to the file. If I'm collaborating, I don't want my partner to hog all the time on the track. Time windows can be enforced?

3) Statistics can be generated from the revision history, make cool collaboration chats that look like MS Project time lines. Who worked on what, when? A visual representation. Pie charts? Changes over time? Or at least a (SOAP/XML?) interface to the data so third party users can query the Renoise Collaboration server to gather statistics?

... ?

cpu Lenovo X220, Intel i7-2640M @ 2.80GHz ×4 os Windows 10 / Ubuntu 16.04 LTS
My Homepage » : My Renoise Tools » : Normalize Your Sig » : One million clicks! »


#16 MickRip

MickRip

    Local Chief Member

  • Normal Members
  • PipPipPipPipPip
  • 231 posts

Posted 24 March 2007 - 07:38

Here are some ideas to add some value to this simpler #2 idea.

1) File history is kept, file can be reverted and restored based on a timestamp query. I want the song from 48 hours ago? Done.

2) Deadlines. The user gets an email or and SMS message saying their time is up and if they don't submit their changes within 30 minutes, another user (in a queue?) will have the next exclusive access window to the file. If I'm collaborating, I don't want my partner to hog all the time on the track. Time windows can be enforced?

3) Statistics can be generated from the revision history, make cool collaboration chats that look like MS Project time lines. Who worked on what, when? A visual representation. Pie charts? Changes over time? Or at least a (SOAP/XML?) interface to the data so third party users can query the Renoise Collaboration server to gather statistics?

... ?


File History. Hmm. I was thinking that. Now the only problem with this is database overhead. I was thinking more of storing the file history on the client side. So ... "mytune_081507_1546.xrns", "mytune_091507_1115.xrns" .. obviously the sync app will look at "mytune.xrns" as the tune to sync..

Doing it the database has more advantages though, although if you're working on a 50mb file - ohh man.. 10 versions of the 50mb file is going to be paaainful on mysql. 50mb is painful enough.

The deadline idea! Great idea!!! Time windows are good, I can't stand waiting around for days and days. I want to track, and I want to now. :P

The visual stuff sounds like a good "rainy day" idea.

#17 Guest_Bantai_*

Guest_Bantai_*
  • Guests

Posted 24 March 2007 - 12:18

Cool to see this idea taking off. If anyone doesn't know about "distributed computing", I suggest you look take a look at the subject and join the conversation.

Some keywords:
- Networked Impulse Tracker 2.14
- mmorpgs
- rmi, skeleton and stub
- distributed transactions
- semaphores
- deadlocks, the dining philosophers problem

#18 mindplay

mindplay

    Advanced Member

  • Normal Members
  • PipPipPip
  • 85 posts

Posted 13 January 2011 - 16:04

I've been having some of the same thoughts, so I'll chip in with the approach I had in mind.

I would write up a "session server", an application that is responsible for maintaining collaboration session state - keeping track of who's participating, the changes they contributed to patterns, instruments, samples, etc.

I believe the biggest problem here is how to keep things perfectly synchronized. I think we need to think in terms of "eventual consistency" - a data replication principle that is frequently used in clustered databases.

In a sense, you can think of the "session server" as the "database" for the song - and each collaborating client needs to be able to replicate changes submitted to the "session server", resulting in eventual consistency; every copy of the song being up to date.

The internet being pretty fast these days, we're talking about synchronization problems that are probably going to be pretty rare - of course, these must be handled, or you would end up with differing versions of the same song on different clients, but with a fast internet connection (or especially on a LAN connection) the number of conflicts encountered should be pretty low.

Here's an article explaining various patterns for implementing eventual consistency:

http://www.allthings...consistent.html

I'm not sure which variation would be the most appropriate.

Under any circumstances, I think the pattern-consistency needs to be implemented at the track-level - for example, for every change you submit, the session server would assign it a running serial number unique to that particular track in that particular pattern, and check for a conflict; e.g. check if another user submitted a change to that track/pattern that has not yet been streamed back to the other clients - if so, you have a conflict.

Now, if every client assigns a serial number to every change as it's made locally, before submitting it to the session server, a conflict can be resolved by telling the client in violation that their change was rejected - the client can then simply undo from it's current local change number back to the change number in violation; the last change (or maybe a couple of changes) would "roll back" on that client. And with the "rejection" response from the session server, that client would receive the conflicting changes, and apply them immidiately after undoing the rejected changes.

For changes to instruments, samples, automation, etc. the same mechanics can be applied - locking and conflict resolution would happen at the instrument/sample/automation-graph level.

Anyway, those are my thoughts for the time being.

Network tracking was a dream of mine since 15+ years, so I was quite excited to see Renoise adding a scripting engine with a network API. I think it's possible! And the internet is fast enough to make this truly useful now.

I regret that I'm no longer making music. Although the prospect of being able to collaborate with friends abroad is a tempting prospect that makes me think about picking up tracking again ;-)

#19 kRAkEn/gORe

kRAkEn/gORe

    Renoise Developer

  • Renoise Team
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 805 posts
  • Gender:Male
  • Location:minefield

Posted 16 January 2011 - 04:00

It's all about keeping things up to date. What about storing only the diffs instead of storing the whole song zip ? Tip
Posted Image

#20 KMaki

KMaki

    Big Super Masta Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPip
  • 672 posts
  • Gender:Male
  • Location:Tampere, Finland
  • Interests:Survival (e.g. breathing). Small scale renoise-tool-coding. The Melvins. Absurdity of it all. Zen / Not-zen. Wifey&Kids.

Posted 13 June 2011 - 18:15

I'd like to chip in an idea, because this seems all too close to a tool I use every day. I'm an architect by profession, and the application camp I'm in is the Graphisoft one; namely ArchiCAD. While the program can on occasion be a pain in the side, it has one really handy feature and that is the current, new version of TeamWork (add copyright sign)(add registered trademark sign)(add corporate logo)(no animals were harmed). It basically means that several people can work on the same project, which is held as a database(of some sort) on a dedicated server. Like this thing you're discussing. And the way this is pulled off, without the chaos implied in the scenario, is by reservations policy. Everyone can work on the same project, but only if they have "reserved" that specific thing they need to work on. After the thing one reserved has been completed, the reservation on the thing can be released, submitting the changes made. Changes made by the person working on the reserved thing can be submitted at any time, in which case only the 'delta' (or diff?) will be sent and handled by the server. Other changes, made by other people working on the project, can be "fetched" likewise. The main catch on this approach is that the reservations on things must be kept in sync. And there's a local copy of the whole project.
While I have zero knowledge on actual networking mechanics, I'd imagine this COULD be a viable solution for working on renoise tracks too. It could mean for example that one could reserve a specific pattern to work on. Or reserve the comments-section. Or a specific instrument. One feature of said TeamWork©®(x) is that users can freely _add_ stuff, while modifying and removing things must be via reservations.
Well. Just wanted to note this 'partial reservations stuff' as an application on the very subject being discussed. Carry on.
Renoise (2.6->) 2.8 / Mac mini 2010 - 2.66GHz Intel Core 2 Duo - 4 Gb 1067 MHz DDR3 / OS X 10.8 / M-Audio Fast Track USB

#21 Mandulin

Mandulin

    Super Advanced Member

  • Normal Members
  • PipPipPipPip
  • 166 posts
  • Gender:Male
  • Location:Germany

Posted 13 June 2011 - 20:08

Network tracking was a dream of mine since 15+ years, so I was quite excited to see Renoise adding a scripting engine with a network API. I think it's possible! And the internet is fast enough to make this truly useful now.


i think we have the same idea in mind but let me clarify what i think:
most of the suggestions have something to do with the project file being treated as a directory, unpacked, the changes diffed, etc. the CVS approach is ok, if it works for software why shouldn't it work for other things.

since the scripting engine allows for networking and offers most of the editing functionalities, a clean approach that would even allow for realtime collaboration would be:
1. person A connects to a server
2. Renoise monitors all of person A's edits (events) (i don't think this is implemented in the API but it shouldn't be that difficult)
3. the edits are translated either to XML or a smaller non-readable format (see marshalling) and sent to the server. if someone adds a sample it is uploaded to the server and kept there until all people participating in the collab have downloaded it, then it is deleted. deleting a sample in your project just sends a message to all clients to remove (maybe not physically) the sample from the project.
two important things: a history is made and uploaded to the server for all clients that were not connected while the changes were made. everything at this point is being compressed!
4. person B receives the changes, the samples are downloaded, and they are applied by using the API again. this could technically happen (almost) in realtime or by a notification that a new version is available. this way a lot of people could participate and even watch what's happening in realtime.

throw in the IRC plugin and you're good to go. ^^

that's how i would expect an idea like that to work and it does go in the direction of a SOAP approach but i would make it more dynamic ... unless you really want to keep it basic.

Edited by Mandulin, 13 June 2011 - 20:21.


#22 mindplay

mindplay

    Advanced Member

  • Normal Members
  • PipPipPip
  • 85 posts

Posted 14 June 2011 - 13:11

It could mean for example that one could reserve a specific pattern to work on. Or reserve the comments-section. Or a specific instrument. One feature of said TeamWork©®(x) is that users can freely _add_ stuff, while modifying and removing things must be via reservations...


I think this is unnecessary and adds complexity - simply having the most recent change override another change is probably much simpler and perfectly acceptable... a song is a big playing field - the odds of two players colliding is really rather minimal.

Have you ever worked two people on a spreadsheet in Google Docs, at the same time? It's a fascinating experience, you should try it :-) ... The tracker is basically a spreadsheet, so you can get a feel for how this could look and work with a spreadsheet in Google Docs - you can see the other "player's" cursors (in different colors), you can see their edits in real-time, you can see them enter and leave the session, and you can chat with them. If you could see a ghosted version of the other collaborators' mouse pointers too, that would be awesome. If the cursor left a subtle trail when changes were made (that fades or disappears after a few seconds), that would take some of the "spookiness" out, so you don't suddenly realize that someone made a change where you were looking.

With the low latency on modern internet connections, you don't need to worry much about the lag - participants will see the other editors' cursors and pointers, and they will naturally get out of the way. I've been working with 4 different people on the same spreadsheet (relatively small, perhaps the eqivalent of 3-4 patterns in size), and we never once collided or overwrote each other's changes. I don't think manual "reservations" are necessary or pratical.

#23 mindplay

mindplay

    Advanced Member

  • Normal Members
  • PipPipPip
  • 85 posts

Posted 15 June 2011 - 14:40

I seem to have a hard time not thinking about this - here are some thoughts I had last night...

Initially, my thinking was that one instance of Renoise would be the "master" and others would connect to it, and they would try to keep things in sync between them.

There are two problems with that approach, that stem from the same issue:

  • This introduces favoritism (the "master" has more control or is more important than the "clients")
  • This approach leads to many edge cases (problems that have to be solved in one of two possible ways, depending on who initiated the action, a "master" or a "client)

The issue is that of equality.

I think, for something like this to succeed, all clients need to be equal - everyone is a client, and all clients are treated the same.

There needs to be a dedicated "session server" that everyone connects to, including the person who starts the session.

The session server of course can be hosted on one of the clients, but that client needs to connect to it, just like other clients, and interact with it in the exact same way as everyone else.

When creating a session, the user who creates it is only "special" in the sense that he might get to pick a password for the session, and upload the "blueprint" song that all clients are going to use as the starting point.

When a client connects, it downloads the "blueprint" song and opens it, and receives all the "change orders" currently in the session.

I mentioned "eventual consistency" earlier, and it can be implemented quite simply, now that all clients are equal:

  • Any "change order" by any client is sent to the server, which numbers all the change orders sequentially, in the order it receives them.
  • The server broadcasts every change order, in sequence, to every client - and this is very important: including the client that submitted the change.
  • Clients apply every change order, in sequence, to their local copy of the song - even a change they just submitted.
Assuming that all change orders are "atomic updates", this pattern should keep everything perfectly in sync.

What I mean by "atomic", is that every change order is simply an update to an object. And object in this case could mean: PatternTrackRow, PatternTrack, Pattern, Sample, Instrument, etc.

In other words, a "change order" would never include things like "cut column", "paste block" or "insert into pattern sequence", which are all abstract operations that can not necessarily be accurately replicated on each client, because they depend on the state of the client's copy of the song. Each client could be "late" (has not received all change orders yet) or "early" (a change order has not been received or processed by the server yet) compared to the server, so change orders must always consist simply of two things: the target object identity and the data to fill it with. By object identity, I mean things like for example "pattern 5", "pattern sequence position 20" or "pattern 4, track 2, row 6".

Simple updates like these, applied in sequence on every client, will never fail - regardless of the state of the client. It doesn't matter if you made a change to a note at precisely the same time as another person - the order in which they are received by the server, determines the order in which they will be received by your client, and your change will stick or be overwritten depending solely on the order in which the server received them, guaranteeing that, even if your client is temporarily out of sync (pending a longer list of change orders), it will be "eventually consistent" with the server.

I believe this describes a working approach, and actually not that difficult to implement. (I wish I had more spare time.)

Other stray thoughts:

  • Have an option to "commit" to the server - only the person who started the server can do this, and it basically means that the server broadcasts any pending change orders to all clients, then clears the list of change orders, and replaces the "blueprint" song with the saved song. This speeds up the connection process for clients that join later, as the list of change orders they have to process to "catch up" with the other clients, will be shorter.
  • Add a simple "version control" system - meaning, whenever the song is "committed", we keep a numbered copy of the previous version of the song. Pretty simple, and should be optional of course, in case you're working with a huge song file.

That's all from me at the moment :-)

#24 a773

a773

    Big Super Masta Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPip
  • 617 posts
  • Gender:Male
  • Location:Copenhagen, Denmark

Posted 06 July 2011 - 19:00

Hi

Found this thread while looking for something similar.

1) What's the status of renoise not being able saving to/writing from dirs (as opposed to .xrns-files)?

2) I'm looking at git with hooks and smudge/clean. It *should* be able to work with xrns files and zip/unzip behind-the-scenes, I just can't get my head around it. Anyone tried this as well?

#25 mindplay

mindplay

    Advanced Member

  • Normal Members
  • PipPipPip
  • 85 posts

Posted 08 July 2011 - 13:59

2) I'm looking at git with hooks and smudge/clean. It *should* be able to work with xrns files and zip/unzip behind-the-scenes, I just can't get my head around it. Anyone tried this as well?


The idea of using a version control system had crossed my mind, but I don't think it's going to work well in practice - version control systems are designed for working with source-code, and they are not designed for real-time applications. They are also not guaranteed to automatically resolve many conflicting changes, as would occur in a system trying to use a version-control service in real-time.

It might be feasible to create an integration for a version control system for version control, however - which might be a really cool thing to have, too. So that saving your song would actually create a revision, and so that you could undo changes made by a particular revision. I'm sure there are references between objects stored in a Renoise XML file though - for example, instruments reference samples, so even if the version control system could successfully track changes and patch/remove change sets, you might end up with corrupted files that won't actually load.

I'm not going to say it wouldn't be cool if you could get that working - but you should not expect it to work in real-time...