Import Pcm File Without A Header

Renoise Tool Development
1

Hey guys.

I’m currently working on an import plugin for REX loops. The actual converter works now, but I’m stuck on the LUA side of things. The converter I wrote takes a REX file and exports the individual slices as 32bit, 96kHz mono PCM files without a header. The files are perfectly valid (tested with SOX), but I can’t open them as they are. I know that I can read them as binary streams using “io.open”, but I can’t figure out how to create samples from the individual slices. Renoise seemingly wants to know the number of frames to create a buffer, but I have no idea how to get the number of frames from the files in question. I tried to create a small buffer, hoping it would simply grow if I feed it more data, but this doesn’t seem to work:

 local slice = io.open("slice_l0.raw", "rb")  
  
 renoise.song().selected_instrument:clear();  
 renoise.song().selected_instrument.samples[1].sample_buffer.create_sample_data(96000, 32, 1, 1);  
 renoise.song().selected_instrument.samples[1].sample_buffer.set_sample_data(0, 0, slice);

returns:

*** No matching overload found, candidates:  
*** bool create_sample_data(SampleBuffer&,long,long,long,long)  
*** stack traceback:  
*** [C]: in function 'create_sample_data'  
*** main.lua:72: in function 'import_rex_loop'  
*** main.lua:18: in function <17><br>```

<br>
<br>
And there are two more things I don't get. First of all, shouldn't "samples[]" start at "0"? If I try that, it complains about an "[i]attempt to index field '?'[/i]". Also, in set_sample_data, what is the "frame_index" supposed to be? Do I have to fill the data frame by frame? Seems... slightly inconvenient. At least for my purposes <img src="https://files.renoise.com/forum/emoticons/default/smile.gif" class="bbc_emoticon" alt=":)"><br>
<br>
<br>
EDIT: Bundling SOX and using that to convert the files to regular WAV files before opening them seems to work (using "[i]sample_buffer:load_from()[/i]"). I'd prefer a native solution, though - I don't really feel like bundling several MBs worth of binaries, especially if there's a native solution.</17>
2

Both are two issues related to the Lua language. To call methods, use “:” instead of “.”
-> sample_buffer:create_sample_data(96000, 32, 1, 1);

and Lua counts from 1, so 1 is the first sample in the list.

See Lua tips for more info.

Also see


for some sample buffer examples.

3

Both issues are clearly outlined in the thread you linked - now I feel stupid… I blame the alcohol. Thanks taktik! :)

4

Don’t worry, both issues created lots of headaches here as well when starting to use Lua.
Simply needs a while for everyone of us to get used to a new language…

5

It’s kinda odd, considering most other languages count from 0. And the lack of increment and decrement operators is weird, too. I think I’ll get used to it.

Anyway, it seems that simply dividing the filesize of the bitstream by 4 should tell me the number of frames. Now I have to figure out how to split the file in individual frames. I came up with this:

 local currentFrame = 1;  
 local currentSlice = 0;  
 local fileinfo = io.stat("slice_l0.raw");  
 local frames = fileinfo["size"] / 4;  
 local slice = assert(io.open("slice_l0.raw", "rb"));  
  
 renoise.song().selected_instrument:clear();  
 renoise.song().selected_instrument.samples[currentSlice + 1].sample_buffer:create_sample_data(96000, 32, 1, frames);  
  
 while true do  
 local frameData = slice:read(4);  
 renoise.song().selected_instrument.samples[currentSlice + 1].sample_buffer:set_sample_data(1, currentFrame, frameData);  
 currentFrame = currentFrame + 1;  
 if not frameData then break end  
 end

But it doesn’t work. The reason for that seems to be that frameData is nil, even though slice:read does seem to read something. If I put a simple rprint function with a counter inside the loop, it counts to the number of frames plus 1. I don’t really get how it can even loop, considering frameData is supposedly always nil…

EDIT: Seems like I also need to find a way to convert the bytes to frames? Sounds… not easy at all. Any idea how to approach this?

EDIT2: I also have to figure out a way to handle tempo matching as well. Syncing the individual samples to lines is probably not precise enough. Would be the best idea in theory though, as tempo changes wouldn’t affect the instrument itself. Needs to be tested. Also, the “Generate Drum Kit” function isn’t accessible using LUA, is it? Can’t find it in the documentation.

6

You’ll have to shuffle around bytes, cause a double floating point number is the only number that Lua uses out of the box.

Heres an example for reading 16bit signed/unsigned little endian:

[luabox]

local band = bit.band
local bor = bit.bor
local lshift = bit.lshift

– read_uint16

local function read_uint16(file)
local bytes = file:read(2)
if (not bytes or #bytes < 2) then
return nil
else
return bor(bytes:byte(1),
lshift(bytes:byte(2), 8))
end
end

– read_int16

local function read_int16(file)
local bytes = file:read(2)
if (not bytes or #bytes < 2) then
return nil
else
local unsigned_value = bor(bytes:byte(1),
lshift(bytes:byte(2), 8))

if (band(unsigned_value, 0x8000) ~= 0) then
return -0x10000 + unsigned_value
else
return unsigned_value
end
end
end

local selected_instrument = renoise.song().selected_instrument
selected_instrument:clear()

– for each slice:
local current_slice = 1

local file_name = “some_file.raw”
local file = io.open(file_name, “rb”)

if (file ~= nil) then
local num_frames = math.floor(io.stat(file_name).size / 2)
local sample_buffer = selected_instrument.samples[current_slice].sample_buffer

if (sample_buffer:create_sample_data(44100, 16, 1, num_frames)) then

sample_buffer:prepare_sample_data_changes()

for frame=1,num_frames do
– read a signed 16bit value
local sample_value = read_int16(file)
if (sample_value) then
– normalize 16bit values to -1,1
sample_value = sample_value / (2^14)
sample_buffer:set_sample_data(1, frame, sample_value)
else
– file io error. show an error and abort
break
end
end

sample_buffer:finalize_sample_data_changes()
else
– sample buffer allocation failed. show an error and abort
end
else
– file failed to open. show an error and abort
end
[/luabox]

See the Files/Bit snippets for more: https://code.google.com/p/xrnx/source/browse/trunk/Snippets/Files%26Bits.lua

Note: reading from the file in big blocks (like 1024 bytes) instead of word by word may be a lot faster. Not sure if that’s relevant here though…

7

Thanks again. With this code (or, more precisely, the dword snipped from the files & bits example), loading the sample worked. Feeding the buffer 32bit float as is (signed, of course), all I hear is noise, though. What format does this need to be in?

And performance definitely is a concern - 40 files per loop (20 slices, two channels) are not exactly uncommon. But I’d be happy to actually get it working for now. There are a couple other, more pressing issues. Tempo matching for example. I haven’t quite figured out how to approach this, yet. A simple time stretch seems to be the obvious choice, but I’d prefer something that survives tempo changes. Beatsync would be a lot more flexible, but I doubt the resolution is sufficient.

8

Try changing the endianess or signed/unsignedness then. Loading a float byte per byte will be a big pain though. This is not what the files&bits examples do. Is the raw format a 32bit float or int format?

Would of course be nice if sample_buffer:load_from_file() also would support “raw” files, then you don’t need to mess around with this in lua.

9

Surely coverting the rex timing data into notes and delay values (as midi import does now) would be the best option ?

Whenever I’ve imported a midi file it’s always sounded pricise.

10

According to SoX, it’s 32bit float. Signed, I guess - as far as I know, 32bit PCM float is usually signed and centered.

I thought about that. I don’t believe it would survive tempo changes, though. Something that works even if the samples are not played in a specific order would be nice as well. I need to experiment with a few different approaches.