Is there a way to create custom samples?

Either draw your own samples in Renoises sample editor using draw and combining loop settings with dsp/vst’s, tweaking instrument envelopes in the instrument setting or get some tools that generate wave shapes and go from there.

Tools like:

For l33t math synthesis get this tool: Overtune | Renoise

and check out Cas’s posts: