Jump to content


Photo

Writing AIFF files


  • Please log in to reply
29 replies to this topic

#1 afta8

afta8

    Big Super GrandMasta Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPip
  • 709 posts
  • Gender:Male
  • Location:London

Posted 07 January 2017 - 12:48

Anyone here have experience of writing binary files in lua? I am trying to export sample buffer data to an AIFF file using lua.

 

I have been poking around the File Imports Tool (http://www.renoise.c...-import-support) which actually has a AIFF export function but I can't figure out sending Sample Data into it.

 

Any takers?


Edited by afta8, 07 January 2017 - 16:50.


#2 ffx

ffx

    Guruh Motha Fakka is Levitating and Knows Everything About Renoise Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • 2938 posts
  • Gender:Not Telling
  • Interests:macOS fanboying

Posted 07 January 2017 - 14:49

Why you want to support AIFF, isn't it a purely outdated container format nowadays? For example, there is M4A.


Edited by ffx_away, 07 January 2017 - 14:50.

MacOS 10.12.6 Retina, Renoise 3.1 64 bit   -   Tuned Shortcuts | Multi-Jump From/To Send | Quick Template | Insert Native DSP Menu (incl. deprecated)


#3 afta8

afta8

    Big Super GrandMasta Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPip
  • 709 posts
  • Gender:Male
  • Location:London

Posted 07 January 2017 - 17:04

Why you want to support AIFF, isn't it a purely outdated container format nowadays? For example, there is M4A.

 

For a very specific use case, I have an OP-1 and it only works with the AIFF format + custom header with sample slice points etc.

I want to make an XRNI to AIFF convertor with slice points & tunings that will load into the OP-1 with these preserved, lets me create sample packs, building blocks that I can jam with in OP-1..

 

Looks like much of the leg work and techniques have already been cracked by mxb in the file imports tool but much of this is on stuff I have little knowledge about e.g. binary files, LSB, MSB, 2s complements etc. Could probably figure it all out given enough time, but I'm looking for a shortcut ;)

 

Also relevant OP-1 thread here: https://www.operator...n/comment/8080/



#4 4Tey

4Tey

    Chief Above Chief Member

  • Normal Members
  • PipPipPipPipPipPip
  • 382 posts
  • Gender:Male

Posted 07 January 2017 - 17:12

I assume afta8 you mean the 'generating_aiff' function?  Quickly looking that audiodata parameter seems to want to be a char string of the sample data.  I think first you would have to convert normalised -1 +1 renoise sample_buffer data to say 8-bit (or 16-bit) signed sample data into a table, then convert the table to a string for the audiodata parameter.


  • afta8 likes this

#5 afta8

afta8

    Big Super GrandMasta Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPip
  • 709 posts
  • Gender:Male
  • Location:London

Posted 07 January 2017 - 17:57

Thanks 4Tey, that helps, I presume 'signed sample data' is the same as PCM data?

 

Not sure how to convert -1 to +1 values into this, but will research it.. Also I presume Renoise is doing this anyway when saving out as wav.. maybe there is a shortcut by saving out the Wav file using the API and then stripping the PCM data out of that..

 

EDIT: Found this http://stackoverflow...redirect=1&lq=1

but if I understand correctly it looks like Audio quality is degraded if you use this method. hmm always thought WAV to AIFF conversion was lossless  :wacko:


Edited by afta8, 07 January 2017 - 18:01.


#6 4Tey

4Tey

    Chief Above Chief Member

  • Normal Members
  • PipPipPipPipPipPip
  • 382 posts
  • Gender:Male

Posted 07 January 2017 - 18:10

Yes possibly (although don't quote me on it :D) linear PCM is signed sample data and yes you could recompose/hack a saved wav file sample data from renoise into a aiff file afta8 :)

 

(But looking from a purely scripting point of view...)  Note, quickly looking at the file format(s) you posted afta8 you would have to keep in mind that the function mxb shows gives you a basic aiff file.  You would have to modify that function to also output a modified COMM chunk and output a APPL chunk into the file to accommodate for the parameters to the OP-1.


  • afta8 likes this

#7 afta8

afta8

    Big Super GrandMasta Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPip
  • 709 posts
  • Gender:Male
  • Location:London

Posted 07 January 2017 - 18:19

You would have to modify that function to also output a modified COMM chunk and output a APPL chunk into the file to accommodate for the parameters to the OP-1.

 

Yes, you are right in that, I am *hoping* that once I have figured out how to create a basic AIFF file, I can hack mxb's code to do the extra header with a bit of trial and error and a hex editor.

I have the samples generated by the OP-1 to compare against so comparing against these I could do it... but then I could be wrong :)


  • 4Tey likes this

#8 4Tey

4Tey

    Chief Above Chief Member

  • Normal Members
  • PipPipPipPipPipPip
  • 382 posts
  • Gender:Male

Posted 07 January 2017 - 18:29

Not that you can really go by this afta8 as this is stupid crude, but I managed to get mxb's function to save out a aiff file with a mono 16-bit 44100Hz sample with:

local data = {}
local nf = renoise.song().selected_instrument:sample(1).sample_buffer.number_of_frames
local dptr = 0
for n=1,nf do
  local SampNumber = renoise.song().selected_instrument:sample(1).sample_buffer:sample_data(1,n)
  if SampNumber < 0 then
    SampNumber = 65536 - (SampNumber * -1) * 32768 
  else
    SampNumber = SampNumber * 32768
  end
  data[dptr] = string.char(bit.band(SampNumber,0xff))
  dptr = dptr + 1
  data[dptr] = string.char(bit.rshift(SampNumber,8))
  dptr = dptr + 1
end

generate_aiff(1,44100,16,table.concat(data))

  • afta8 likes this

#9 afta8

afta8

    Big Super GrandMasta Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPip
  • 709 posts
  • Gender:Male
  • Location:London

Posted 07 January 2017 - 18:46

Wow! amazing!!  :drummer:

 

Thank you so much! I can work with 'stupid crude' :) 

OP-1 only works with mono samples so this should be enough for my needs, nice work!

 

I see you are doing some bitwise operations there, I don't really understand these but does your approach lead to any degradation of audio quality?


  • 4Tey likes this

#10 4Tey

4Tey

    Chief Above Chief Member

  • Normal Members
  • PipPipPipPipPipPip
  • 382 posts
  • Gender:Male

Posted 07 January 2017 - 18:56

Nah, afta8 the bitwise operations is just a way of shifting the bits down and masking bits to output the low-byte/high byte of a 16-bit number into two 8-bit chars.  Nothing to do with any audio degradation.  The lines you would be most interested in are these:

if SampNumber < 0 then
    SampNumber = 65536 - (SampNumber * -1) * 32768 
  else
    SampNumber = SampNumber * 32768
  end

That's just a silly hack up from me.  There is probably a better way of converting a normalised signed -1..+1 floating point sample into a signed 16-bit integer number.  If there is any 'degradation' it is going to be here.  I'm sure afta8 you can come up with a lot better way :)



#11 afta8

afta8

    Big Super GrandMasta Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPip
  • 709 posts
  • Gender:Male
  • Location:London

Posted 07 January 2017 - 19:06

Ah ok, I get it now, so any errors creeping in will be from the Integer to Float conversion. I suppose the simplest way to test this would be to do a null test with a source wav and the exported aiff. 

I will test your code out tonight, but either way its enough to get me going on writing the custom headers. Thanks again :)


  • 4Tey likes this

#12 afta8

afta8

    Big Super GrandMasta Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPip
  • 709 posts
  • Gender:Male
  • Location:London

Posted 09 January 2017 - 17:49

If there is any 'degradation' it is going to be here.

 

After testing that code and doing a null test, it does introduce noise. Not sure how to fix this, I'm guessing some kind of dither.. lua uses doubles for everything which is 64bit I think..

The final aiff output is 16bit so the introduced noise is probably down to bit rate conversion I think, don't really fancy writing a dither algorithm in lua :)



#13 4Tey

4Tey

    Chief Above Chief Member

  • Normal Members
  • PipPipPipPipPipPip
  • 382 posts
  • Gender:Male

Posted 09 January 2017 - 18:30

Hmm, interesting afta8.  Not sure about dithering/noise algos, but I've done a little binary compare test myself.  I took a simple 16-bit 44100Hz mono sample.

 

Asked Renoise to produce a wav file from this and extracted the 16-bit linear PCM sample data.  I then ran through my above code to produce a aiff file and extracted the PCM linear data from that file.  We are talking a sample size of 19810(bytes) / 2 = 9905 16-bit word size samples.  Then I binary compared the two files.  They are 100% identical.  Now I'm not saying that that is conclusive in any way afta8.  I suppose at the moment what I could possibly conclude is that if there is a rounding/error factor creeping in at some point, it is very small between that and a Renoise wav file?


Edited by 4Tey, 10 January 2017 - 16:54.


#14 afta8

afta8

    Big Super GrandMasta Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPip
  • 709 posts
  • Gender:Male
  • Location:London

Posted 09 January 2017 - 19:12

I'm not comparing the binary files just testing with a Sine wave and loading them in Renoise (it will import aiffs) and can hear audible noise on the aiff version.. see the attached files.

 

I'm using your code exactly as above, also attached the tool file if you want to have a look (it will add an aiff export option to the tools menu) - are you on mac or windows?

Attached Files



#15 4Tey

4Tey

    Chief Above Chief Member

  • Normal Members
  • PipPipPipPipPipPip
  • 382 posts
  • Gender:Male

Posted 09 January 2017 - 19:39

Ah, more interesting afta8 :)  Yes I can see the 'noise' here.  I admit it is a little interesting.  I tend to use Linux btw (well it had to be one of the two afta8 :D)  The first thought I'm thinking here is to convert that 'sine_wav_original' file you have there to a aiff file via a 3rd party program.  I'm just thinking if the unix program 'sox' could do this. 


  • afta8 likes this

#16 afta8

afta8

    Big Super GrandMasta Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPip
  • 709 posts
  • Gender:Male
  • Location:London

Posted 09 January 2017 - 20:05

Ha, well mac is sort of linux :D.. I used Audacity to convert to aiff and it is much better... it doesn't null perfectly but is good enough.. see attached..

 

Yeah, quite interesting and I'm learning more about binary stuff so all good overall :)

Attached Files



#17 4Tey

4Tey

    Chief Above Chief Member

  • Normal Members
  • PipPipPipPipPipPip
  • 382 posts
  • Gender:Male

Posted 09 January 2017 - 20:55

I got sox to give me a aiff file as well, so at the moment I've just spent sometime looking at the two binary files produced.  Quickly though I don't think the header file that mxb's function produces is quite right (well according to sox anyway..)  But that isn't the full problem.  In fact the problem looks kinda strange at the moment, but I've got to leave it here afta8.  But I will have a look tomorrow at this and see if I can find anything more :)   



#18 Djeroek

Djeroek

    Probably More God or Borg Than Human Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 6713 posts
  • Gender:Male
  • Location:Borneo

Posted 09 January 2017 - 21:33

Could a cdp process help with converting to .aiff? Like;

 

Copysfx (copy/convert soundfiles); http://www.ensemble-...fmn.htm#COPYSFX

 

or

 

Housekeep respec (change sample rate, format or properties of a soudfile); http://www.ensemble-...hous.htm#RESPEC

 

Maybe there are better more suitable processes, but these sprung to mind :)


Edited by Djeroek, 09 January 2017 - 21:34.


#19 4Tey

4Tey

    Chief Above Chief Member

  • Normal Members
  • PipPipPipPipPipPip
  • 382 posts
  • Gender:Male

Posted 09 January 2017 - 22:59

Ah okay I think I've worked out what it was afta8 :)  I've attached a modified version for you, it should now produce your sine wave (without the noise! :D )  Basically it was the 'endiness' of the 16-bit word was the wrong way around.  Also I forgot that lua is 1-based indexed on its tables whereas I was thinking in C terms (0-based arrays), so it was missing the first sample of data :rolleyes:  I've corrected that.  The other thing afta8 is that save_aiff function is at the moment hard wired to only work on instrument 0, but you can modify that with the selected instrument number etc...  So give that a go afta8 and see if that is a bit better :)


Edited by 4Tey, 11 January 2017 - 08:52.

  • afta8 likes this

#20 afta8

afta8

    Big Super GrandMasta Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPip
  • 709 posts
  • Gender:Male
  • Location:London

Posted 10 January 2017 - 00:14

So give that a go afta8 and see if that is a bit better :)

 

Thats perfect, even cleaner than the audacity export! :D

 

Couple of questions to better my understanding...

 

At this link http://stackoverflow...irect=1&lq=1 it it says the " range of 16 bit integers is -32768 to 32767" however in your code you are always multiplying by 32768 and the max range looks like it is 65536 (should this be 65535?). Are you taking a different approach here or am I missing something? I notice when I do a null test it zero's completely except for a small section where the sine wave reaches it's peak value.

 

Also you mention 'endiness' and on the wikipedia page for aiff https://en.wikipedia...nge_File_Format it says  "With the development of the Mac OS X operating system, Apple created a new type of AIFF which is, in effect, an alternative little-endian byte order format." - Therefore is your code here creating a 'little-endian" format of the aiff?

 

 

 

Could a cdp process help with converting to .aiff? Like;

 

Yes and could also use sox, however that is only half of what I need to do, the next bit is to 'customise' the meta data in the sample to include slice data and tunings, this is why it is better to try and use mxb's code as it is already writing metadata and hopefully will be easier to modify.

 

Anyway 4Tey has worked some magic here to help me on my way :)


Edited by afta8, 10 January 2017 - 00:16.

  • Djeroek and 4Tey like this

#21 4Tey

4Tey

    Chief Above Chief Member

  • Normal Members
  • PipPipPipPipPipPip
  • 382 posts
  • Gender:Male

Posted 10 January 2017 - 05:13


At this link http://stackoverflow...irect=1&lq=1 it it says the " range of 16 bit integers is -32768 to 32767" however in your code you are always multiplying by 32768 and the max range looks like it is 65536 (should this be 65535?). Are you taking a different approach here or am I missing something? I notice when I do a null test it zero's completely except for a small section where the sine wave reaches it's peak value.

 

Also you mention 'endiness' and on the wikipedia page for aiff https://en.wikipedia...nge_File_Format it says  "With the development of the Mac OS X operating system, Apple created a new type of AIFF which is, in effect, an alternative little-endian byte order format." - Therefore is your code here creating a 'little-endian" format of the aiff?

 

 

Yep, that's fair questions afta8.  About the multiplier, yes it probably is 65535 rather than 65536, so if it produces better PCM values around zero then by all means change that, no trouble.  I just quickly guessed, not much thought applied on my part there afta8 :D

 

Now the 'endiness' business...  At the moment the code produces a big endian PCM data stream.  Old style Amiga/Apple probably favoured this as they were Motorola 680x0 based machines, where the processor architecture was big endian.  However I think what wiki is saying is that because Apple moved to Intel x32/x64 processors which are little endian in nature, Apple changed the AIFF file format to accommodate this.  Now your OP-1?  Does it want little or big endian stream?  I don't own a OP-1 afta8 but with the file I have here it seems to me to want a little endian PCM stream.  Luckily it shouldn't be a problem because all you do is to swap this:

--This is big endian...
data[dptr] = string.char(bit.band(bit.rshift(SampNumber,8),0xff))
dptr = dptr + 1
data[dptr] = string.char(bit.band(SampNumber,0xff))
dptr = dptr + 1

To this:

--This is little endian...
data[dptr] = string.char(bit.band(SampNumber,0xff))
dptr = dptr + 1
data[dptr] = string.char(bit.band(bit.rshift(SampNumber,8),0xff))
dptr = dptr + 1

But you would have to try that when you get to the stage of uploading a sample to your OP-1 :)

 

I'm not an expert on the AIFF file format, but I assume there is something in the header that tells it what 'endiness' the stream is in afta8.

 

[Edit:  Yes there is, I can see it here in this file :)    The other thing to mention is that when you do start to modify the header afta8, I don't think Renoise can load any other format of AIFF other than standard 'old style' big endian format.  So you would probably have to use another program to load those AIFF files.   Ah, hello Djeroek :) ]


Edited by 4Tey, 10 January 2017 - 05:54.

  • afta8 and ffx like this

#22 Conner_Bw

Conner_Bw

    Probably More God or Borg Than Human Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 7084 posts
  • Gender:Male
  • Location:Montreal, Quebec, Canada

Posted 10 January 2017 - 15:13

Great info thanks for sharing.


  • afta8 and 4Tey like this

cpu Lenovo X220, Intel i7-2640M @ 2.80GHz ×4 os Windows 10 / Ubuntu 16.04 LTS
My Homepage » : My Renoise Tools » : Normalize Your Sig » : One million clicks! »


#23 ffx

ffx

    Guruh Motha Fakka is Levitating and Knows Everything About Renoise Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • 2938 posts
  • Gender:Not Telling
  • Interests:macOS fanboying

Posted 10 January 2017 - 15:17

For some reason I love to read such bit shifting operations/code, I dunno, gives me instant nostalgic feelings  ^_^

 

Thanks for that @4tey


Edited by ffx_away, 10 January 2017 - 15:17.

  • 4Tey likes this

MacOS 10.12.6 Retina, Renoise 3.1 64 bit   -   Tuned Shortcuts | Multi-Jump From/To Send | Quick Template | Insert Native DSP Menu (incl. deprecated)


#24 afta8

afta8

    Big Super GrandMasta Member

  • Normal Members
  • PipPipPipPipPipPipPipPipPipPipPip
  • 709 posts
  • Gender:Male
  • Location:London

Posted 10 January 2017 - 18:51

Thanks 4Tey, some useful knowledge there :)

 

Now your OP-1?  Does it want little or big endian stream?  I don't own a OP-1 afta8 but with the file I have here it seems to me to want a little endian PCM stream.

 

It looks like it will accept either, I loaded the sample created by your code (big endian) into the OP-1 and it works fine. If I look at the samples generated in the OP-1 in a hex editor you can see: "sowt)Signed integer (little-endian)" so I presume the OP-1 native format is little endian.

 

I also generated a sample for the OP-1 using a standalone software tool for it: http://now.teenageen...-1-drum-utility and I think that is big endian, but not sure how to tell when looking at it in a Hex editor

 

So overall I don't think the OP-1 is fussed about the "endiness"

 

 

The other thing to mention is that when you do start to modify the header afta8, I don't think Renoise can load any other format of AIFF other than standard 'old style' big endian format.

 

It works fine for me, I tried loading the native sample generated on the OP-1 which is little endian and it loads into Renoise fine.

 

If you are interested I have attached 3 different versions of same sample:

 

1) lua_export.aiff - The file generated by your code

2) xfer_util.aif - The same file exported by the OP1 utility I mentioned earlier

3) op1_native - The same sample re-saved within the OP1

 

You can see in the 2 and 3 the following meta data chunk which I now need to put into the lua export file in a APPL chunk:

op-1{"drum_version":1,"type":"drum","name":"user","octave":0,"pitch":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"start":[0,75324246,75324246,75324246,75324246,75324246,75324246,75324246,75324246,75324246,75324246,75324246,75324246,75324246,75324246,75324246,75324246,75324246,75324246,75324246,75324246,75324246,75324246,75324246],"end":[75320188,117226960,117226960,117226960,117226960,117226960,117226960,117226960,117226960,117226960,117226960,117226960,117226960,117226960,117226960,117226960,117226960,117226960,117226960,117226960,117226960,117226960,117226960,117226960],"playmode":[8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192],"reverse":[8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192],"volume":[8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192,8192],"dyna_env":[0,8192,0,8192,0,0,0,0],"fx_active":false,"fx_type":"delay","fx_params":[8000,8000,8000,8000,8000,8000,8000,8000],"lfo_active":false,"lfo_type":"tremolo","lfo_params":[16000,16000,16000,16000,0,0,0,0]}

Should be fairly straightforward right ;)

 

 

 

 

Attached Files



#25 4Tey

4Tey

    Chief Above Chief Member

  • Normal Members
  • PipPipPipPipPipPip
  • 382 posts
  • Gender:Male

Posted 10 January 2017 - 19:36

 

It looks like it will accept either, I loaded the sample created by your code (big endian) into the OP-1 and it works fine. If I look at the samples generated in the OP-1 in a hex editor you can see: "sowt)Signed integer (little-endian)" so I presume the OP-1 native format is little endian.

Smashing afta8, your OP-1 doesn't mind which way around the bytes are.  The file 'op1_native' is little endian.  The other two files are big endian.  Basically when it hasn't got 'sowt' in the header chances are it is going to be a standard big endian :)

 

 

It works fine for me, I tried loading the native sample generated on the OP-1 which is little endian and it loads into Renoise fine.

Yeah, kinda makes sense that you can on Mac OSX as Taktik is probably running through and finding a decoder (Quicktime?)  I can't load AIFF little endian files afta8 as I don't have the decoder in Linux :)

 

 

Should be fairly straightforward right ;)

Well, err, yes?...err maybe :D  Bit of programming, got to format your string and get the length :)

 

There is afta8 just something tiny tiny I have just noticed with those 3 files....But I'm just going to have another look....