Thought on this one a bit last night while I was sat in a bar,
I see your points… but this is still new and probably in dire need of further development.
In the meantime, we should consider this ‘just’ another step into audio technology: future music.
Let’s worry about copyrights once Timbaland got his hands on Melodyne 
But yeah, remixing will NEVER be the same… damn, can’t wait to tear apart my favourite string quartets 
Well he at least makes your work famous… and if he keeps doing that, folks will eventually assume that whatever he produces is not from his own creativity.
HAHAH, yep!
but this is amazing. I already know what I’d like to do with it. Thanks for posting it.
this is reeally that one step i thought would never happen 
im so glad it did…
its honestly insane.
Well if you know anything about spectral analysis, you will understand how this works…not to say that it isn’t farking amazing, 'cause it is…
Basically, he’s using FFT to split the waveform into it’s component sin waves, looking for areas with groupings of strong peak data. Then he’s going “Oh, that must be a note from one instrument” … then taking those groupings and ripping them into their own waveforms as isolated notes
that’s where it gets nutty though, 'cause anyone who’s ever messed with spectral editors can tell you that even if you do get a clean visual of a grouping of strong peaks, simply lassoing them and playing them back will result in a very tin-can sounding noise… that only has a slight resemblance to the original note. This is because almost every instrument humans have devised have harmonics which span quite a bit of the audible soundscape. This is because, quite simply, the wider the frequency range of a note, the better it sounds… this is why supersaws are so huge sounding… because they’ve got a ton of harmonics.
But harmonics are a funny thing, and it seems to me that this genius of a man realized this. Within a single note, they tend to have a mathematical pattern. Quite simply, a square wave, which is harmonically rich compared to a sin wave, has harmonics from many octaves of the note it’s playing… all of which are simply the octave below multiplied by 2.
So that considered, if you can isolate the strongest frequencies of the note, and figure out what basic tones it’s comprised of, it should be fairly simple to scan the rest of the waveform and determine which harmonics belong to it. Sure, sometimes notes will share the same harmonics, but using some more advanced statistical methodologies, one could conceivably determine which oscillations of those harmonics belong to which note.
Come to think of it, it might be pretty simple to look for patterns in spectral data by monitoring the waveform peaks in the same way a compressor does, to see which peaks and valleys match up… this would be even easier than looking for harmonics.
I’ve been thinking about this stuff for a long time… long before this innovation came around, and as far as I’m concerned it was only a matter of time before someone did it. And as evident by this breakthrough, all it took was someone what knew enough about audio processing, fft, and statistical methods to figure it out. BTW, If you feel like raping this post and using it to devise your own VST hackery, I request that you please send me a free copy of your resulting plugin 
harmonics in a nutshell
like relativity theory is just slowing you down, when you thought you were getting a good pace
After watching the video yet again: It all sounds great but I am sceptic how good it will work. I mean, there are alot of overtones and noise for each sound. If I have a guitar-chord f.e. with several different attacks/transients, how will the software know which part of the noise belongs to which note ?
And what I am interested in is how it performs with a vinyl-sample, not a full blown orchestra ofcourse, something more empty.
If you haven’t seen it, check this clip with Peter presentating the software himself:
Thanks for the video, junior.
This is one thing I’d like to check, too.
Yes, the manipulations sound very natural but in all videos the single manipulated notes of a chord are never played alone. So one could assume that you could hear the missing harmonics without the full chord around it.
But hey. This is really revolutionary. Even if it worked a little bit worse. Being able to turn a recorded major chord to minor with FFT is really something different than what I would expect to be possible so quickly.
I wonder if this technology can be used to have even better audio compression ratios (while 44kHz, Stereo @ 17kbps is already possible with aac+).
It looks nice, but I do have my doubts about its workability, since Melodyne’s current monophonic editing doesn’t work that great already. It takes a lot of tweaking to get Melodyne to recognize lines correctly and then you still get at most 75% accuracy. But we’ll see…
yesterday I had a quick second thought about this and came to the same skeptical conclusions as looza; I think that the notes recognition it’s not only done via the obvious harmonics which have been described by Byte-Smasher, but also via attack recognition, so I wonder how this would work on samples with not evident attack, like a bowed instrument could be in some articulations, or with synthetized sounds which can basically have any kind of attack-sustain-decay-release.
on a side note, if it will work it will surely be a great revolution for the world of music, but personally “I would never use it”
if you’ve checked that sonicstate video you’d see he also analysed a string section without hard attacks.
To get this straight, I am really looking forward to this. If 50% of the promised stuff turns out to be true then it is still amazing. “Live” was the first huge leap for people sampling from vinyl, this could be next next one.
apparently they are going to launch a beta test period for registered Melodyne users in “two weeks”; the news is dated September the 2nd, presumably of 2009
it’s magic. a more practical question… there must be loads of old orchestra recordings of classical pieces that are public domain right?
would love to fiddle around with something that got as sick ooomph/schwung as the original recording of bernard herrman’s psycho soundtrack: http://www.youtube.com/watch?v=Jm3ZuwcAI7o (shitty youtube sound tho :< )
those strings have a nice stab! there’s something about those old microphones or something that i really like.
don’t expect too much from this… the only thing you will be able to do, ideally, is to separate notes from inside chords from a single instrument recording.
I personally expect this to be one of the most overhyped applications in the history of music software. We will see.
Time slicing applied to the Amen Break gave birth to a whole new genre of music. I’m wondering if we will see something similar with this new technology and one melodic loop …
I can’t wait to hear the opinion on this when it hits beta. If it does what it is supposed to do then this really is the holy grail of audio processing. But, as an audio software developer, and someone who’s done more than his fair share of spectral twiddling, I’m very skeptical for the exact reasons BYTE-Smasher so eloquently outlined. I just can’t envision a function that can cleanly decompose an FFT into individual notes/instruments.
I guess one way around it would be to have a ‘training’ mode, similar to voice recognition system. So you play it two cleanly picked guitar notes, a fifth apart, and from this it works out which FFT buckets correspond to formants, and which to ‘pitch’. Using this, I guess you could have a reasonable stab at decomposition of taught instruments. Hmmm, may have to fire up Pure Data tonight and see if I can cobble something together…
Well, let’s hope that I’m going to be pleasantly surprised come mid September!