Oh wow, thanks everyone! I hadn’t seen this so I was writing my own definitions as I needed them.
Is the readme status up to date? I will see if I can contribute.
sure, what are you looking for?
within the Renoise Scripting Editor? or do i need an external code editing app?
It won’t work with the built in script editor. You need to use a Language Server Protocol enabled editor and connect it to the Lua language server. The easiest to setup is Visual Studio Code with the LuaLS extension installed.
Edit: you can see how I use the definitions in my exs24 tool’s repo.
The .luarc.json tells LuaLS how to load the definitions.
The definitions are setup as a Git submodule like this:
mkdir -p definitions
git submodule add https://github.com/renoise/definitions.git ./definitions/renoise
I am looking into generating the docs.
I am able to spit out JSON and Markdown with this command:
mkdir -p build
lua-language-server --doc=./ --doc_out_path=./build --logpath=./build
To turn it into HTML, I am thinking we could use Pandoc. That would let us easily generate a PDF too.
Pandoc lets you write custom readers in Lua. This is executed by Pandoc directly and there are libraries available for JSON etc. So it’s pretty straighforward to build, and it’s Lua which should be familiar to contributors 
In addition to reading in the API docs, we can read in markdown, images, etc. to build the full docs site. So we can port over the documentation txt files and the images for the view builder API.
I would love to add a single page view, navigation, and (client side) search. It’s taking me forever to find things with the current docs. And if we are using Pandoc I would like to spit out Dash / Zeal docs too 
If someone could look into the document generation this indeed would be great and help a lot. Thanks for considering that @mattya.
The problem with the JSON from the Lua language server seems to be that it contains far too much information in a rather unhelpfully structured way for the API docs, including all the standard Lua API symbols and so on. I am not sure how easy it would be to exclude these and structure the resulting PDF in a custom way, similar to the old docs:
https://files.renoise.com/xrnx/documentation/
How would you do the JSON processing, API doc filtering & sorting?
Like the Pandoc idea though. I think @unless has tested/considered using using mdBook here as well.
Hey, I already started restructuring and extending the txts from the manuals into markdown files. Currently using mdbook (it has a fast search built-in as well) to generate a html view but once everything is in markdown any other doc gen could work.
As for the definitions reference, it would be great to have some readable format! I got to excluding all the irrelevant parts from the JSON output but it’s still rather disconnected in its structure so some custom logic has to be written to parse everything into meaningful chunks of classes and such.
1 Like
Here is a shell script that you can call with a lua definition file and it will output a .json with the same name with any standard lua stuff removed.
#!/bin/bash
lua-language-server --doc="$1" --doc_out_path="."
jq --arg search $1 '[.[] | select(any(.defines[]; .file | contains($search)))]' doc.json > "$1.json"
rm doc.json doc.md
jq '.[] | .name' "$1.json"
echo "-> $1.json"
Essentially all renoise entries will contain the source file’s path while the standard stuff will not.
Nice, appreciate all the input!
The JSON output is unfortunate. I’m using a similar trick to @unless ATM where I filter out types that aren’t defined in the current working directory. Some of them would go away if we changed how the definitions are written (e.g. the class API has a function definition, which causes the functions it calls to be included), but others I can’t get rid of.
The structure is more problematic. We can get a pretty decent structure just by splitting on the namespace dots in the type name and inferring the hierarchy. But that won’t match the old docs, because for example renoise.Sample won’t be nested under renoise.Song. It might work better to use the file structure. That is what rust doc does. That still doesn’t match the old docs but it’s close: Song > Instrument > Sample.
The JSON processing, filtering, and sorting can all be done in the Pandoc Lua reader. I have the basics working - I’m able to document a single function signature.
That’s great you already started converting the txt files. I think we could go either way: put the API docs in mdbook or put merge the markdown files into pandoc. With the pandoc stuff the same code can render to markdown or HTML. So it could generate markdown that then gets inserted into mdbook.
Using mdbook would actually solve the structure issues, since we could manually organize the “entry points” to the API docs from there. This isn’t using mdbook but the Elixir ecosystem does a good job of combining guides and API docs in a similar format: Ecto — Ecto v3.11.2
Edit: This is how it would look if we merged the default Markdown output into mdbook:
With mdbook we can probably skip pandoc completely. A Rust preprocessor could parse the JSON and generate the markdown on it’s own.
2 Likes
I like the mdbook stuff. Especially if that means that we can merge the general, custom API descriptions (the old txt contents from here) and the API docs together.
Perhaps the file/folder structure of definitions/library at main · renoise/definitions · GitHub can be used as a template, programmatically for the API doc structure? Then the structure doesn’t need to be hardcoded.
2 Likes
Heh, I came to the conclusion that using the file structure made more sense after trying the namespacing approach. That’s roughly how rustdoc works too.
However with mdbook (at least by default) you need to list the pages in SUMMARY.md. So at least the “entrypoints” for the API would need to go there. For the example above I had this:
# Summary
[Introduction](./introduction.md)
# User Guide
- [Enable scripting](./enable-scripting.md)
# API Reference
- [Standard library](./standard.md)
However, it seems like a preprocessor could be used to inject the API docs. So we could have a directive like this:
{{#luadoc definitions/renoise.lua}}
…that would expand to the chapter list, and inject the pages.
The simplest thing to do by far is to only have the preprocessor build the definition markdown files, and write the links in SUMMARY.md manually. Then for each API reference we can optionally write prose and include the generated markdown like this:
# Some API
Some optional prose
{{#include definitions/somedefinition.lua}}
Edit: it is possible to add pages with a preprocessor, there’s an implementation here: mdbook-all-the-markdowns — Rust utility // Lib.rs
I started a mdbook preprocessor here: GitHub - matt-allan/mdbook-luacats: A mdbook preprocessor for LuaCATS API documentation
The mdbook part is just a stub ATM. The core logic is being built out separately, with a test binary that simply prints rendered markdown to stdout.
This is the output it currently generates for the bit.lua file: bit.md · GitHub
I will try and get this finished but I might not before I am unavailable for a month or two, so feel free to fork this if I am not responding.
2 Likes
Well I had more free time than anticipated and I spent a very long time trying to parse the LuaCats doc.json. I ended up making a new repository because I had to start from scratch: GitHub - matt-allan/mooncats: Mdbook support for LuaCats API docs.
It sort of works but the JSON docs are really not good. I don’t think it’s worth the effort. Parsing the definition files ourselves (similar to how the existing docs are built) is probably a better idea.
The doc representation is clearly optimized for IDE usage. Even after doing all the work to clean up the data so it can be used it’s incomplete. There is data we need that just isn’t there.
A few examples of the issues:
- Enum fields don’t specify the enum value or even type
- Types don’t specify optionality (e.g.
integer?) - Fields like
renoise.API_VERSION = 6.1just show up asnumber - The “view” (source code) for type aliases is truncated if it’s too long
I did get to see how it would look though. Here’s a screenshot:
(We could add custom CSS, that’s just the default styling for generated markdown)
Mirroring the directory structure seems to work well.
If I try this again I will probably try to write the preprocessor in Lua and use LPeg to parse the definitions. Not sure if I can sink any more time into this ATM.
Hey, thanks for all the efforts you’ve put into this!
I did experiment a bit last week as well, after you’ve first shared your repo and I came to some of the same conclusions.
However, there are a few things that make the situation a little less hopeless for us.
- Enum fields don’t specify the enum value or even type
Enums contain the entire lua code block of their definition which could be rendered as is in the final markdown. Their types are missing but AFAIK they are all just integers in Renoise anyways, and…
- Fields like
renoise.API_VERSION = 6.1just show up asnumber
Constant fields are missing their values but @taktik made a fair point previously about having just the identifier in the API docs is enough as it’s better if people use those instead of the actual value literals, so it’s enough if a user knows they can use renoise.API_VERSION, they don’t need to know the exact value (they can always check if they really want to know), the same thing is kinda true for the enums above as well.
- Types don’t specify optionality (e.g.
integer?)
Types do specify optionality but it’s a bit hidden and just present as a ? char at the end of types, instead of being a separate field or something more transparent. Also, depending on where you are reading the data from, the type might say integer or integer? In this regard I can totally agree that the JSON is super annoying to work with, everything is duplicated with subtle nuances and organized in unintuitive structures. 
- The “view” (source code) for type aliases is truncated if it’s too long
The larger string union types’ views do print as ...(+1) instead of the whole list just like in an IDE when you hover it (which is frustrating even in the editor imo), but the complete list of values can sometimes be read from the extends/types array which you aren’t parsing at all currently.
So overall, there is definitely some level of string parsing needed if we want to get the actual types properly like parsing unions from string|integer and nullables from integer? (things like fun() typed arguments and some table args where this gets trickier). Still, this might be less work than starting from scratch with parsing all lua docs strings (as the LPeg you mention only covers Lua in general if I’m not mistaken), but I can certainly understand your sentiment here.
On the other hand we can obviously cut some corners in terms of just the Renoise API docs generation since it isn’t using all of the features of LuaLS which other codebases might need.
That said, if you want this to be a general solution for any Lua API generation, I can see how this might not cut it for you. I guess another alternative could be to fork/patch/extend lua-ls itself with some not-so-oddly-formed JSON export (or straight up fix their markdown output for instant docs), outputting a nice JSON would be more widely useful in the general sense as anyone could render that however they wanted.
Anyway, thanks again for all the things you already did on this, it definitely moved things forward! I’ll try implementing some of the improvements I’ve mentioned above.
So I digged into the code of lua-ls a bit and it seems all we need to change is this config file to get rid of the ...(+n) truncation. Showing the alias’ name like this instead of expanding it could also be handy.
This can be simply patched on an built/installed instance of lua-ls.
It should be possible to pass in a config file or place a .luarc.json file into the working dir for lua-ls to pick it up but I can’t get this to work on the output doc for some reason, overwriting the template file does work.
1 Like
I pushed a branch using the Lua + Lpeg approach here:
If you want to run it you need to install lua 5.1, Luarocks, and Mdbook, then install lpeg, luafilesystem, and dkjson via Luarocks. After that mdbook serve.
It is able to parse most (all?) of the definitions into tokens. Rendering the markdown from there is only implemented for the main renoise file ATM.
Overall I like this approach. The definitions are already in the order we want them which is great. We can capture whatever we need to from the source files. It should be less code overall. There is no dependency on the language server or its doc format. It’s definitely more fun to work on too
To keep it simple I restricted the grammar to what absolutely needs to be in a definition file. Random bits of executable code (like in the class definition) would need to be deleted. There is some annotation syntax in use ATM that I don’t think is valid.
The main downside is maintaining the grammar. Lpeg is dense. It will need to be updated if new annotation kinds are added.
@unless I realize I made it sound like the issues were insurmountable. It was more that I felt pretty disappointed after doing all that work normalizing the JSON that I would need to parse source code anyway 
I suspect the doc format will change at some point because of how hard it is to use, at which point the code would have to be updated which won’t be fun. Fixing things upstream would definitely be ideal.
I think either approach could work well. I’m not super keen on the language-server docs approach at this point but it wouldn’t take much to get it over the finish line.
Thank you both for your massive efforts!
Unfortunately, both solutions require a lot of work, so I’m wondering if the simpler one might be just good enough for now to start with? Is there a preview of the docs as built from the JSON?
Otherwise I’d try setting up some github workflows to automatically build the docs using either approach. We’ll likely need this anyway.
Is there anything else I can do to help here?
This one is cool too but personally I’d vote for improving the first approach (I can do that on this week or so), I think that would be easier to maintain on the long run simply because Rust is less error-prone and more self-documenting than a custom parser and generator written in lua.
It’s a valid concern that the JSON format might change in the future and looking at the 4.0 branch of the lua-language-server, the parser code seems to be rewritten and the output structure probably differs as well, although I have no idea in what sense, or when the new version is planned for release. But if the rest of the doc generation is complete and isolated I don’t think it would be too hard to adapt it to a new JSON format, especially if that JSON was made easier to consume.
For a “first edition” we can skip parsing strings and just render the appropriate views for complex types from the JSON, for example by choosing the one which has the ? for optionals instead of the one that doesn’t, same for unions, a raw string like number|string is good enough to be rendered into the docs at first and we can make it so that doc generator actually understands that as a union later (which could be done with a smaller parser than the whole lua-ls annotation spec).
As for the order/organization, I think it would be better to not use the file structure at all and instead generate single pages per class in alphabetical order (with fields and methods also sorted) this would make
- the docs less prone to change in case files are split in the future, which otherwise might break links pointing to the doc site
- easier to navigate and search even without search functionality
- the markdown rendering simpler
- easier to create internal links between pages as you’d don’t need to know where a referenced class was actually defined, only its name
1 Like
Made some progress here:
@mattya: unless picked up your moocats rust app and continued there. He is currently away for a few weeks, so I picked up his work and continued tweaking things.
The result is not a generic luaCATs definition doc generator - for now, but Renoise specific, and embedded in the other Renoise XRNX docs.
I just saw that you were still working on that, but in a more general way.
A preview is available here:
https://renoise.github.io/xrnx/API/index.html
Sources are here:
There are tons of things that could or should be fixed and improved, but I’m stopping here for now. I think this is quite usable as it - for a start.
Note that the rest of the book also is WIP. Contributions are always welcome!
2 Likes