New "LuaCATS" definition for the Renoise API

Oh wow, thanks everyone! I hadn’t seen this so I was writing my own definitions as I needed them.

Is the readme status up to date? I will see if I can contribute.

sure, what are you looking for?

within the Renoise Scripting Editor? or do i need an external code editing app?

It won’t work with the built in script editor. You need to use a Language Server Protocol enabled editor and connect it to the Lua language server. The easiest to setup is Visual Studio Code with the LuaLS extension installed.

Edit: you can see how I use the definitions in my exs24 tool’s repo.

The .luarc.json tells LuaLS how to load the definitions.

The definitions are setup as a Git submodule like this:

mkdir -p definitions
git submodule add https://github.com/renoise/definitions.git ./definitions/renoise

I am looking into generating the docs.

I am able to spit out JSON and Markdown with this command:

mkdir -p build
lua-language-server --doc=./ --doc_out_path=./build --logpath=./build

To turn it into HTML, I am thinking we could use Pandoc. That would let us easily generate a PDF too.

Pandoc lets you write custom readers in Lua. This is executed by Pandoc directly and there are libraries available for JSON etc. So it’s pretty straighforward to build, and it’s Lua which should be familiar to contributors :smile:

In addition to reading in the API docs, we can read in markdown, images, etc. to build the full docs site. So we can port over the documentation txt files and the images for the view builder API.

I would love to add a single page view, navigation, and (client side) search. It’s taking me forever to find things with the current docs. And if we are using Pandoc I would like to spit out Dash / Zeal docs too :grin:

If someone could look into the document generation this indeed would be great and help a lot. Thanks for considering that @mattya.

The problem with the JSON from the Lua language server seems to be that it contains far too much information in a rather unhelpfully structured way for the API docs, including all the standard Lua API symbols and so on. I am not sure how easy it would be to exclude these and structure the resulting PDF in a custom way, similar to the old docs:
https://files.renoise.com/xrnx/documentation/

How would you do the JSON processing, API doc filtering & sorting?

Like the Pandoc idea though. I think @unless has tested/considered using using mdBook here as well.

Hey, I already started restructuring and extending the txts from the manuals into markdown files. Currently using mdbook (it has a fast search built-in as well) to generate a html view but once everything is in markdown any other doc gen could work.

As for the definitions reference, it would be great to have some readable format! I got to excluding all the irrelevant parts from the JSON output but it’s still rather disconnected in its structure so some custom logic has to be written to parse everything into meaningful chunks of classes and such.

1 Like

Here is a shell script that you can call with a lua definition file and it will output a .json with the same name with any standard lua stuff removed.

#!/bin/bash
lua-language-server --doc="$1" --doc_out_path="."
jq --arg search $1 '[.[] | select(any(.defines[]; .file | contains($search)))]' doc.json > "$1.json"
rm doc.json doc.md
jq '.[] | .name'  "$1.json"
echo "-> $1.json"

Essentially all renoise entries will contain the source file’s path while the standard stuff will not.

Nice, appreciate all the input!

The JSON output is unfortunate. I’m using a similar trick to @unless ATM where I filter out types that aren’t defined in the current working directory. Some of them would go away if we changed how the definitions are written (e.g. the class API has a function definition, which causes the functions it calls to be included), but others I can’t get rid of.

The structure is more problematic. We can get a pretty decent structure just by splitting on the namespace dots in the type name and inferring the hierarchy. But that won’t match the old docs, because for example renoise.Sample won’t be nested under renoise.Song. It might work better to use the file structure. That is what rust doc does. That still doesn’t match the old docs but it’s close: Song > Instrument > Sample.

The JSON processing, filtering, and sorting can all be done in the Pandoc Lua reader. I have the basics working - I’m able to document a single function signature.

That’s great you already started converting the txt files. I think we could go either way: put the API docs in mdbook or put merge the markdown files into pandoc. With the pandoc stuff the same code can render to markdown or HTML. So it could generate markdown that then gets inserted into mdbook.

Using mdbook would actually solve the structure issues, since we could manually organize the “entry points” to the API docs from there. This isn’t using mdbook but the Elixir ecosystem does a good job of combining guides and API docs in a similar format: Ecto — Ecto v3.11.2

Edit: This is how it would look if we merged the default Markdown output into mdbook:

With mdbook we can probably skip pandoc completely. A Rust preprocessor could parse the JSON and generate the markdown on it’s own.

2 Likes

I like the mdbook stuff. Especially if that means that we can merge the general, custom API descriptions (the old txt contents from here) and the API docs together.

Perhaps the file/folder structure of definitions/library at main · renoise/definitions · GitHub can be used as a template, programmatically for the API doc structure? Then the structure doesn’t need to be hardcoded.

2 Likes

Heh, I came to the conclusion that using the file structure made more sense after trying the namespacing approach. That’s roughly how rustdoc works too.

However with mdbook (at least by default) you need to list the pages in SUMMARY.md. So at least the “entrypoints” for the API would need to go there. For the example above I had this:

# Summary

[Introduction](./introduction.md)

# User Guide

- [Enable scripting](./enable-scripting.md)

# API Reference

- [Standard library](./standard.md)

However, it seems like a preprocessor could be used to inject the API docs. So we could have a directive like this:

{{#luadoc definitions/renoise.lua}}

…that would expand to the chapter list, and inject the pages.

The simplest thing to do by far is to only have the preprocessor build the definition markdown files, and write the links in SUMMARY.md manually. Then for each API reference we can optionally write prose and include the generated markdown like this:

# Some API

Some optional prose

{{#include definitions/somedefinition.lua}}

Edit: it is possible to add pages with a preprocessor, there’s an implementation here: mdbook-all-the-markdowns — Rust utility // Lib.rs

I started a mdbook preprocessor here: GitHub - matt-allan/mdbook-luacats: A mdbook preprocessor for LuaCATS API documentation

The mdbook part is just a stub ATM. The core logic is being built out separately, with a test binary that simply prints rendered markdown to stdout.

This is the output it currently generates for the bit.lua file: bit.md · GitHub

I will try and get this finished but I might not before I am unavailable for a month or two, so feel free to fork this if I am not responding.

2 Likes