Renoise 1.8 Xml Format Discussion

Hello.

This article is technical. It covers Renoise 1.8.0 b2, XML file format revision 2.

It’s a great idea that we at last have the XML format. It opens a wide variety of new possibilities. Unfortunately, some of the ideas in the current implementation make it hard to use all the goodies that XML gives us. However, since the format is still undocumented, I hope some things may be either solved or just properly justified.

Disclaimer: Keep in mind that all my comments are written in good faith. Dispite me working for a company that bases its solutions on XML, I don’t consider myself God. My remarks may be wrong, and if they are, please explain why. Also, as I am not native English, bits of my advice may seem harsh but that’s only because of my poor vocabulary.

Let me go through some of them and give some hints how I would deal with the issues.

Issues:

1. There are major inconsistencies in the list numbering area. I mean, handling of various serial data is done differently (and sometimes in a overly complicated way).

XML files are documents which means the order of elements in them is significant. That means we don’t need to specify numbers for things we present in series. But that is valid if we don’t skip any indices, ever. Plus, if we need access to specific indices in the XML at runtime, having an explicit index value is handy.

I’d propose to change all handling of series in a way that ensures these three things:

• for data that is always in the order and always fully specified, skip the indices. Example: Instruments/Instrument/VstiProperties/VstDeviceDoc/OptionalNodes/OptionalNode/ParameterN (where N is the parameter number). Skip the N. If it’s absolutely needed, do it as an attribute (so it can be queries using XPath tools in other software, maybe later you’ll gonna use it yourselves ).
• use attributes for indexing everywhere else (example below)
• for data that is rarely different from the default, assume the default (so when an element is not found in the document, 0 or another reasonable default depending on the context, is taken). Examples: Instruments/Instrument/SplitMap should be empty by default, assuming all Splits be zeroes. A single non-zero Split would be then presented as: Y where X and Y are numbers. Same goes for Tracks/Track/NoteNolumnStates (let’s assume all states as “Active” by default).

2. PatternPool/Patterns/Pattern/Tracks/Track/Lines/Line is raw text. Examples: 44:—…,—…,G#600… or 16:C-40340…. This is wrong because this raw text holds much information that should be available for XML parsers (line numbers, instrument numbers, sample numbers). This is required for XPath querying, for linking notes with the instruments within the document, etc. etc. Moreover, raw text representation is error-prone and not future proof.

I’d rather do elements like this: C-4. Some explanation: ‘id’ is the line number. As you can see, comment 1 also applies here. ‘col’ is the column number within the track (no need for —…,). ‘instr’, ‘sample’ are self-explanatory but notice that the presence of these attributes lets you join the information from a line with the instrument/sample within the document, without any other processing. Even though there is no explicit Instrument numbering in the format, XPath and other tools let you query for “the first” Instrument in the document, “the third” sample in the Instrument, etc. ‘pan’ is open for discussion. We could do 0% = totally LEFT, 100% = totally RIGHT. Or something else. You can think about surround panning (will Renoise still be stereophonic in 5 years? We don’t know ). ‘vol’ is self-explanatory.

Obviously, if you skip an attribute within the element, a default value is assumed. If you think that this approach would lead to bigger files, I can assure you that is hardly the case. Zip compression handles redundant text blocks very well. And the advantage of having proper data partition is big.

3. In XML it is usually considered a design defect if copious empty data structures must be presented to maintain document validity. It is often said that XML is used to present data, not the lack of it. So, using indexing on elements, you could rid of the empty PatternPool/Patterns/Pattern/Tracks/Track structures.

Questions:

1. Why the complexity of VSTi information? I mean the path to parameters is: Instrument/VstiProperties/VstDeviceDoc/OptionalNodes/OptionalNode/

2. Why are all atomic data presented as text in the nodes? For instance 0 could be just . That’s usually easier to parse. But this one’s really a thing of preference. The important thing is to keep this consistent whithin all document. When we present atomic data as text, we do so everywhere (and you seem so, that’s good).

3. What are Parameter Chunks for and can’t they be presented in another way than binary?

4. Isn’t Tracks/Track/NoteNolumnStates a typo? I mean “Nolumn”?

I didn’t test the format very much (I downloaded RNS1.8 only a couple of hours ago) and in general it looks very promising. If you noticed any other issues or have other questions, add them here.

If your songs would load with VST’s having a million of parameters and you have set plenty of them, songloading could take up to 6 minutes.
This also happened with MIDI parameters however this is solved by treating them differently.
But a lot of things have been done to speed up loading and processing.
If you have tips that can speed up processing (and thus loading) you could try and contact Taktik.
From various statements he made about his affinity to XML he is not that long into it.

If you know where and how to delimit the data, migrating that condensed data (16:C-40340) to a new structure in the future, either based on attributes, elements, or a hybrid, wouldn’t be too hard.
Sure, it’s bad XML. I guess it’s either faster, easier or both at relating the data to any internal structure or handling in Renoise.

About attributes, even though you can format attributes to be more readable, and even though they’re probably faster/easier to parse, it’s not an enveloped structure, you can’t have two or more identical attributes, you have to work around illegal characters, ++

Thanks for the feedback!

Indices are redundant, yes, and they are only used in VST devices for the following reason:
Take a look at other Filter devices: there each parameter has an unique identifier, that describes the parameter as its visible in Renoises GUI. VSTs also have those, but they can be missing, or not be unique (the VST standard allows this), so we are auto generating names for them by simply using “ParameterN” as name. Thats not nice, yes, and we should maybe use a more clever way to auto-generate those names, using the VSTs labels as a base…

See below (empty nodes - defaults)…

That wasn’t an easy decision: the current way of serializing Pattern contents is indeed in the first place not parser friendly, but a lot more readable for those who want to change something “by hand” in their text editor. Again you get what you see and know (it looks like this in the pattern). The goal wasn’t to make the document structure great XML, but easy to understand and parse (in that order).

I’m nevertheless unsure if this was the right decision. Would be great to hear some more comments from others. Im/Export performance improvements or something like this was not the problem.
Backwards compatibility is also no problem, using the current or your proposed way. It makes no difference (at least for us), if we have to rewrite the contents of one node or multiple nodes.

I don’t agree here. Explicit is better than implicit. That there is some structure, and this structure is empty, is an important information. Now you see what you get, and we don’t need to document how default values are defined for which document version, if someone wants to write a program which creates renoise xml data. Also as soon as the defaults change, you anyway have to fill up these values when upgrading the document. This easily gets very messy.

Thats ugly I know, but I could not avoid this thanks to an implementation detail: Structures which can be empty (which is the case for a VST instrument device which can be present or not), have to be wrapped into a OptionalNodes/OptionalNode/ block. Maybe I will find some when later a better solution for this. For now we have to live with that…

As you said: there is no rule for that, so I simply have chosen to avoid attributes (as they are limited in some aspects).

VSTs can save their data in two flavors (where the VST chooses which one should be used). One is to simply give the Host a raw, undefined block of binary data, the other that the host has to save each parameter manually (including all available banks). The first has to be written as CDATA, there is no way to avoid this. The second one used to be (in earlier version of the Renoise format) a bank/parameter block. This slowed down loaded so much with some plugs (there are plugs which have something like 16000 parameters in 128 banks), that we device to use a internal defined raw chunk for that instead.

Yeah, just a stupid typo. Will fix that for the next build.

Thanks again for the detailed discussion. We are of course interested in any kind of feedback or criticism. Why would we else open the file format?

Oh, I didn’t get that in the first place. For me looked just like a weird way of saying . The former is incorrect XML while the latter is. But then again, if these represent an unique identifier used by Renoise in the GUI, that’s fine. Although, maybe a better naming convention would help.

That’s incorrect. If you left the line notation that way because you want to let users edit these files by hand, you are getting into BIG trouble. What if somebody misses a dot writing these raw text data? What if they forget to leave other empty columns in the line of the track? If you had proper XML, understanding it would not be harder and any problems would be recognized by the parser level (hopefully even on the user’s side while editing in a good XML editor).

It’s also not true that XML notation and raw notation are equally future proof. Let’s see: 16:C-40340… OK. Having Renoise opened, I can tell which character stands for what. But, if Renoise version 2.x adds a new column (let’s name it Coolness Factor) then we have to break this line structure in the file format. Old Renoise format parsers (old Renoise versions, old Winamp Renoise plugins (1), etc.) will just stop working. Even for files that don’t use Coolness Factor. If you had something like: C-4, and added an attribute coolness=“50%”, it won’t break anything. Old parsers will ignore it, new parsers will use it. Well, right, the module will probably sound not as intended, but this way let’s old parsers load the module in the first place. Moreover, extending parsers to handle Coolness Factor is easier if we tell them: just parse another attribute. Changing the raw line structure is more tricky. Plus, even updated parsers must still parse older documents. Having raw text data, I can see the huge switch(doc_version) statements already!

But of course, it’s up to you to decide what will eventually be in the format.

(1) The reason I bolded Winamp plugs is that people rarely update them once they are installed. So better have something really future proof if we intend to provide something like that.

Why do you skip empty Lines then?

I’d say leaving the defaults out and adding explicit indexing makes it a lot easier to alter the data. Why? Because you just query for the right path within the document (say, Instruments/Instrument/SplitMap), check whether is present already, and if it isn’t, add it anywhere in the section. Since elements are indexed, they don’t even have to be in the order. And see what happens if we want to alter a document having all empty/default data explicitly told: we have to enumerate through all nodes and choose the right one to alter. And while that’s kind of easy to do by machine (although somewhat less performant), it’s very error prone as soon as you’d like to change your XML document by hand in an XML editor.

A pity we can’t have the data in XML (yeah, I’m an XML zealot allright ) but I understand the motivation behind it. 16k parameters, whoa!

— and a short reply to hcys —

Yes, that’s true. Attributes can’t be structured and there is a policy of having one attribute of a given name per element. But if structure or redundancy is needed, elements can contain other elements, instead of attributes. Some formats use also delimited values (i.e. house_number=“66234,66235,66236”). I don’t like the current raw representation of because it is the first thing to be changed in the format when Renoise will get more (or altered, or enhanced) functionality. Having Lines with attributes (or elements, as said above) is more extensible. I already mentioned above (in reply to taktik), why.
— end of reply —

And uh, devs… Please, keep in mind that some people already use Renoise 1.8 on a day-to-day basis and it doesn’t save legacy .rns modules so keep all the format changes documented, so we can edit them by hand when Beta 3 (4, 5…) and the Final gets released. Or even better, let Renoise load the old format stepping and silently alter it while saving to the new and better.

Sure, we would else get into really big troubles. All documents are versioned (see the “doc_version” attribute) and automatically upgraded/converted step wise when imported, so we are free to change the format when needed. Even developers don’t have to take care about future document changes, as soon as they use the correct/old version number in their document generating tools…

Just a quick note. Let me answer the rest later…

I agree that better XML compliancy would make things easier… I personally wouldn’t want to have to parse something like 16:C-4021A0E either… and I think something like C-4 would work much better. Unfortunately, I also realise it might make file sizes bigger, but if you’re compressing the songs anyway, then it wouldn’t make that much of a difference.

Also, with clipboard data such as the following:

it might be better represented as:

This is kind of going off on a tangent, but im curious as to the minimum requirements Renoise has for the xml in terms of loading it? Is it possible to simply have the pattern pool and sequencer data, or just the pattern pool data, and still load it as a module for editing?

Also, and this is from looking briefly over the xml (and being confused out of my mind), and wondering if there’s an explicit reason the data is contained in only one file, as opposed to multiple files for each type of data, and then concatenated during loading? I’m thinking the majority of 3rd party tools will be for explicit segments of the XML as opposed to the whole shebang. As is, the prospect of writing a parser for this thing in Flash (my platform of choice i’m afraid) scares me out of my mind. There’s just so much i would never even touch.

I’d like to have GlobalSongData.xml, RecordManager.xml, Instruments.xml,Tracks.xml etc. Is that so wrong? =)

Perhaps that could be an option, but personally, I like having it in one file.

Ya, but flash chokes on 900kbs of xml And it gets worse with song complexity. Oh well hehe

I completly agree with Ambivalence on this one. If you want to edit something by hand you rather do it directly in renoise. It is better to make the format more easy to parse and make it futureproof by following the elemental rules of xml instead of to make it easy to understand and edit by hand.

Those are points indeed since the format was also actually explicitly set up for devs to create third party tools and not for the ordinary user to debug broken songs or something similar in that category (more or less quoting Taktik here). Although the current format makes it pretty easy for the everyday user to fix or adjust things outside Renoise, specially if the song seems broken.

Hence why I stated it could be an option… that way you could do either

That’s a bad idea. Either we pack everything in one file or split the structure to many files. Having it optional would produce way too much hassle. But the problem persists in the XML world. An XML file is essentially a tree (with significant order of node appearance). A filesystem is also a tree (but without any more order than sorting by name/dates/etc.). The granularity of XML (in the means of: what to put in a single file and what to solve on the filesystem level) is a big problem. Usually XML files are split if the biggest chunks don’t have to preserve order, can be processed in a stand-alone manner (specifically: they make sense without opening other parts of the XML file), etc.

In our case the thing is, we don’t make any specific gain in splitting the files. Sure, it may produce smaller files and be somewhat easier to read by a human. On the other hand, zip files are compressed per file which means the overall level of compression would be lower if splitting files. Moreover, chunks of the file don’t make much sense without other parts. And they probably won’t be parsed without other parts, ever. So, no, I don’t think we’ll have the file split version.

Actually, it’s a great idea to develop a tool which would handle broken files, which could compact other files (throw out unused patterns, instruments, restructure patterns to minimize redundancy, etc.). I’ll think about it when the XML file format is ready. I can prepare a XML Schema and a DTD for it.

Speaking of which file format is better for fixing or adjusting… On the first sight, it may seem that the current one is more readable. But in fact it isn’t. It’s very error-prone to by hand adjustments. No parameter delimiting, no parameter description, the need to specify also the empty notes in a multi-note track, etc. Proper XML would in the end turn out to be much more friendly, not only to machines, but to men, too. And if you still want to see the notes like they are presented in Renoise, writing a parser that would present them that way is trivial, once you have a good XML spec. In Python or PHP you can do it in less than 50 lines of code.

Plus, having all in proper XML can lead to various formatting in the end. Just imagine exporting xrns to HTML, PDF, note notation, whatever. Everything customizable (with panning shown, hidden, measured in other units, etc.). It’s just a case of parser friendliness.

In fact that’s a very interesting problem. If we could produce minimalistic tunes, that would mean that various other tools (Audacity, REAPER, etc. come to mind) could do easy exporting to .xrns. And, think the other way round

But, let’s wait for taktik to first answer the things in my earlier post. You promised, taktik, you did

Thats indeed a strong point. Its a lot easier to break that string magic, than the “fully XMLed” notation, and evil to think that people could/should edit the XML by hand. Enough reason(s) to change that as soon as possible

Version compatibility anyway has to be solved different. As said in my previous post, all XML files in Renoise are versioned and automatically upgraded while loading. We are not trying to be backwards compatible: Thats something that never works, especially not in such complex documents that we currently have.
So if there is a new “coolness” column, you only have to take care of that when using/writing document versions which already have that column. If you write a document with a version number, which does not have that coolness column, Renoise will upgrade it automatically for you while loading. There again it really doesnt make a difference if we have that column inlined into the complex line string or in the attributes.
Ok, that means that people who write Renoise XML parsers have to upgrade their parsers with every document change as well, but I dont see a solution for this to avoid this… Better let it immediately fail, than letting it result in a pseudo/half working translation.

I am also not sure if using percentages for volume/panning and effect values are a wise descision: I see your point in abstracting the thing, beeing as compatible as possible, but volume pan effect values are not just values that got from 0 to something. There are commands coded into the hex code (like for example e3 in pan for retriggering) that are not very well represented in percentages. So maybe its even a good idea to save that values in hex strings, instead of ints???

Let me see how easy it will be to remove the full string representation, and then send a proposal for a new structure here…

We are skipping empty lines, because thats the internal representation of the lines (just for memory performance reasons). It was not easily possible to hide that implementation detail into the XML structure. I understand your point that this is not very consistent, but I really could not change that…

Didnt we (you) said that changing XML in a text editor is evil? Then its fine for me that you (your machine) have (has) to enumerate all the nodes
Also: As said in my previous post: I just dont like this “defaults get removed” magic, because it makes things more complicated (implicit) than they have to be…
And this makes things more complicated when writing/changing documents: You have to change/update all existing indices then. With no indices, you just insert a new node somewhere…

GlobalSongData, RecordManager, Instruments and Tracks are not self contained, they nearly all depend on each others. Thats why they are in one file. If we would present them as different files, you anyway would have to open all of them and fix, upgrade the links. Also if they are seperated into own files, one could think that you could just add/replace them (like hey - I want the patterndata of this song in that song) which might in the best case result in a crash / broken document…

Point taken. To reiterate, what is the absolute minimum requirements for a functional xrns?

Yippie!

Yes, point taken. The thing is I was more referring to third party parsers (like a Winamp plug-in for instance). But it’s true that it can be solved by versioning. And since we don’t plan to take over the mp3 player market, we don’t need to care about supporting legacy parsers.

I agree. It’s better to have it in hex strings. You got my idea right, it was to make things even more abstract. But I didn’t think it over enough to see the problem with embedded commands. So, hex values it is!

I pointed that out just to show that it works fine and so would other areas where we hold default data.

That’s true It’s only that I have some academic approach and when I see an opportunity to save computing complexity, I usually go for it.

OK, as I said - it’s up to you. This one was not really that important. And it’s true about the indexing updates. So your approach seems good.

Thanks for the detailed discussion. It’s good to see that the software you use is made by people open to arguments and with an objective point of view. Keep up the good work!

Perhaps a solution to sunjammer’s problem would be a community coded external utility that could split/rejoin the .xml documents

the other day i was thinking.

since the fileformat is open now, it is now possible to manipulate renoise songs, without renoise.

i was looking into embedded systems for microcontrollers awhile back.
an the other day i noticed some pretty nice MCU development boards.

a big array of buttons, big graphics lcds , sd card storage, tcp/ip stacks, etc…

add a dsp, a micro controller/processor an alot of memory (sampler),
an this is the start of a hardware tracker.
that would run using renoise XML structure.

so the more clear and concise the renoise XML structure is, the easier it will be for someone interested in writing code for doing something like this would be.

You would start a hardware tracker and end up with a simplified laptop. This is interesting but I can’t recall a single reason why a 13.3" MacBook would be worse than a hardware tracker. But it’s another topic.

Back to the point: when can we consider the XML format “stable”? Some of us have a Great Idea and we’d want to start implementing it right away