Linux: priority too high?

Help, Support & Bugs
1

HI,

I’ve been working on a realtime AV performance using Renoise, Processing & SuperCollider on Linux for couple of years now and have never been able to truly get rid of XRUNS, sometimes really badly. I tried everything and today I noticed something interesting.

I followed some of advices in how to setup realtime IRQ and scheduling priority using rtirq (via http://www.rncbc.org/drupal/node/252). There I found some nice diagnostic tools to see what priorities some processes and their threads have.

So essentially highest priority should have the soundcard (firewire in my case), just after rt clock (rtc), and then jack and finally renoise?

I now also realise that on linux Renoise uses FIFO and RoundRobin schedulers (no other audio program seem to use RR scheduler on Linux). While those renoise threads that use FIFO (FF) have a priority of 50, those that use RoundRobin (RR) have priority of 95! Which is above the priority of firewire kernel driver AND realtime clock(?). While I’m not an expert but I’m suspecting that this is what is creating quite heavy glitches in my Linux.

Could somebody please explain a bit if Round Robin scheduled threads with priority of 95 should or shouldn’t interfere with jack and soundcard? I guess it would be helpful in my quest to get rid of glitches on Linux.

here’s some indication on what’s happening:

$ sudo ps -eLo pri,rtprio,cls,pid,nice,cmd | grep -E 'firewire|renoise|jackd|scsynth|rtc|RTPRI' | sort -r
PRI RTPRIO CLS PID NI CMD
135 95 RR 2255 - renoise
135 95 RR 2255 - renoise
130 90 FF 114 - [irq/8-rtc0]
125 85 FF 200 - [irq/20-firewire]
106 66 FF 2583 - jackd -R -P 60 -d firewire -r 44100 -p 128 -n 3
105 65 FF 2583 - jackd -R -P 60 -d firewire -r 44100 -p 128 -n 3
104 64 FF 2583 - jackd -R -P 60 -d firewire -r 44100 -p 128 -n 3
100 60 FF 2583 - jackd -R -P 60 -d firewire -r 44100 -p 128 -n 3
 95 55 FF 2938 - scsynth -u 57110 -a 124 -i 4 -b 1026 -m 900000 -R 0 -C 0 -l 1
 95 55 FF 2255 - renoise
 90 50 FF 2938 - scsynth -u 57110 -a 124 -i 4 -b 1026 -m 900000 -R 0 -C 0 -l 1
 90 50 FF 2938 - scsynth -u 57110 -a 124 -i 4 -b 1026 -m 900000 -R 0 -C 0 -l 1
 90 50 FF 1443 - [irq/29-nvidia]
 41 1 FF 2583 - jackd -R -P 60 -d firewire -r 44100 -p 128 -n 3
 41 1 FF 2583 - jackd -R -P 60 -d firewire -r 44100 -p 128 -n 3

 39 - TS 196 -20 [firewire_ohci]
 39 - TS 192 -20 [firewire]
 19 - TS 2938 0 scsynth -u 57110 -a 124 -i 4 -b 1026 -m 900000 -R 0 -C 0 -l 1
 19 - TS 2938 0 scsynth -u 57110 -a 124 -i 4 -b 1026 -m 900000 -R 0 -C 0 -l 1
 19 - TS 2938 0 scsynth -u 57110 -a 124 -i 4 -b 1026 -m 900000 -R 0 -C 0 -l 1
 19 - TS 2938 0 scsynth -u 57110 -a 124 -i 4 -b 1026 -m 900000 -R 0 -C 0 -l 1
 19 - TS 2938 0 scsynth -u 57110 -a 124 -i 4 -b 1026 -m 900000 -R 0 -C 0 -l 1
 19 - TS 2938 0 scsynth -u 57110 -a 124 -i 4 -b 1026 -m 900000 -R 0 -C 0 -l 1
 19 - TS 2583 0 jackd -R -P 60 -d firewire -r 44100 -p 128 -n 3
 19 - TS 2583 0 jackd -R -P 60 -d firewire -r 44100 -p 128 -n 3
 19 - TS 2583 0 jackd -R -P 60 -d firewire -r 44100 -p 128 -n 3
 19 - TS 2583 0 jackd -R -P 60 -d firewire -r 44100 -p 128 -n 3
 19 - TS 2583 0 jackd -R -P 60 -d firewire -r 44100 -p 128 -n 3
 19 - TS 2583 0 jackd -R -P 60 -d firewire -r 44100 -p 128 -n 3
 19 - TS 2583 0 jackd -R -P 60 -d firewire -r 44100 -p 128 -n 3
 19 - TS 2255 0 renoise
 19 - TS 2255 0 renoise
 19 - TS 2255 0 renoise

RR = round robin, FF = first in first out, TS = OTHER - the default universal time-sharing scheduler policy used by most processes

$ sudo /etc/init.d/rtirq status
 PID CLS RTPRIO NI PRI %CPU STAT COMMAND 
 114 FF 90 - 130 0.0 S irq/8-rtc0 
 200 FF 85 - 125 1.0 S irq/20-firewire 
 491 FF 80 - 120 0.0 S irq/28-snd_hda_ 
 492 FF 80 - 120 0.0 S irq/17-snd_hda_ 
 110 FF 75 - 115 0.0 S irq/16-uhci_hcd 
 109 FF 74 - 114 0.0 S irq/18-uhci_hcd 
 108 FF 73 - 113 0.0 S irq/19-uhci_hcd 
 107 FF 72 - 112 0.0 S irq/23-uhci_hcd 
 113 FF 70 - 110 0.0 S irq/1-i8042 
 112 FF 69 - 109 0.0 S irq/12-i8042 
 38 FF 50 - 90 0.0 S irq/9-acpi 
 92 FF 50 - 90 0.0 S irq/14-ata_piix 
 93 FF 50 - 90 0.0 S irq/15-ata_piix 
 99 FF 50 - 90 0.0 S irq/19-ata_piix 
 106 FF 50 - 90 0.0 S irq/23-ehci_hcd 
 423 FF 50 - 90 0.0 S irq/7-parport0 
1144 FF 50 - 90 0.0 S irq/27-eth0 
1443 FF 50 - 90 0.3 S irq/29-nvidia 
 3 TS - 0 19 0.0 S ksoftirqd/0 
 14 TS - 0 19 0.0 S ksoftirqd/1 
 19 TS - 0 19 0.0 R ksoftirqd/2 
 24 TS - 0 19 0.0 S ksoftirqd/3

here above you can see there are two Renoise threads that have higher priority than RTC and firewire hardware. does that mean that if Renoise demands CPU cycles and there’s no time left for firewire HW this produces a glitch? or is RR algorhythm here the one that does not make this a priority?

from http://www.cyberciti.biz/faq/howto-set-real-time-scheduling-priority-process/ :

The scheduler is the kernel part that decides which runnable process will be executed by the CPU next. The Linux scheduler offers three different scheduling policies, one for normal processes and two for real-time applications.

  1. SCHED_OTHER – the default universal time-sharing scheduler policy used by most processes.
  2. SCHED_FIFO or SCHED_RR – intended for special time-critical applications that need precise control over the way in which runnable processes are selected for execution

So gathering from this RR is pretty realtime (or lowlatency), and could interfere with FF-scheduled threads if it has a higher priority?

I’m using a lowlatency kernel, no fancy window managers with any fancy 3D accelerations (Awesome WM) etc. (it is true however that Processing is using opengl, but that’s actually a separate X server on the same machine - however nvidia IRQ has a very low RT priority of 50, far from firewire and jackd, so it shouldn’t be a problem).

any ideas greatly appreciated!

2

Hi!

I think renoise isn’t the very most stable thing for sub 10ms latency. Or maybe 5-8ms with a hard rt kernel. And some xruns detected by jack don’t really seem to glitch the audio. IDK why, maybe the fixed prio 95 rr worker threads do have something to do with it, but maybe it’s allright and those threads handle getting the load done without interfering with other programs too much. And yes - on the one hand as a jack program you would expect renoise to do its processing in the jack threads, on the other hand this way it probably has its greater freedom for example for multicore scheduling. I’ve once tried (as the prio 95 has 4 prios above up to 99 possible) to put certain stuff (soundcard irq, jack, rtc, …) above the renoise rr threads, but it actually worsened performance and gave extremely more xruns iirc. Btw I think to recall that renoise realtime priorities are exclusive btw, there isn’t any “so and so much above and thus more priorised by a certain amount”, it is “above, even by just one, always gets served first until it passes control to all stuff below which will get served in order of priority”.

For another recent discussion on this topic see: https://forum.renoise.com/t/low-latency-and-xrun-optimizations-on-linux/45002

3

Same as OopsFly said: Under 10ms expect some trouble with Renoise (dunno about your other apps).

I don’t think either it’s so simple as ordering priorities from HW driver > Jack > applications - this doesn’t give the best performances - at least with USB.

There is something else going on, but unfortunately I’ve not seen any convincing explanation.

Out the blue, I’d recommend making a try after a cold boot without rtirq activated - with all priorities to defaut, and let Renoise climb to the roof alone - just for the sake of testing.

I have also to mention that since my last post in that other thread, I managed to achieve a reasonably xrun free configuration at 2ms latency (96K643 in and out so ~6ms roundtrip). no rtirq involved - I use irq threading but do not change priorities at all, and downgraded to jack1 (don’t ask me why but that did help).

Also I use Renoise just as a sequencer with some samples and almost no effects, so I’m not sure if that would be of any help for you. But I can run totally xrun free for an hour or so with a song loaded - yeah :wink:

Oh, and my rt kernel is built with “make localmodconfig” - that’s pretty radical, but I’d swear this helps a tiny bit as well.

If you want to push it further, there’s apparently a way to build jack2 in tracing mode (although I didn’t step there yet by lack of time):

https://www.google.cz/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0ahUKEwjM7rPNufPMAhWGMhoKHTnnCygQFggcMAA&url=http%3A%2F%2Fwww.grame.fr%2Fressources%2Fpublications%2FTiming.pdf&usg=AFQjCNESbNDBZN_65YrCB70-WdpSvQHREQ&sig2=IuOkICmKpPYj0X3MUD1Sqw

Cheers,

LX

4

o wow, nice, thanks guys, so many interesting pointers.

There’s a long period of work when I don’t have to dable with kernel/system tweaking, but then comes the point - usually connected to live performance, when things start to get really annoying.

Yes, it seems like cold boot and just running what you need helps a lot. At the moment a good improvement seems to be to set cpu scaling governor to performance. at the moment I’m running a FireWire Edirol FA-101 breakout box at 44100, 3 periods of 128 frames. it’s alright. it glitches sometimes, but seems like manageable.

But it’s too bad that Renoise is not capable of being more tight with regards to realtime and xruns. I will have to consider some other tools (SuperCollider?) for live A/V performance then. Otherwise I love Renoise for composition!

Do you have experience with any other apps within your configuration? Is - in your case - just Renoise that is the culprit?

Also, thanks for engaging in a discussion. Feels like I’m not the only Linux wierdo here…

5

You’re for sure not the only weirdo here - Linux or not :wink:

The more programs running concurrently, the harder to troubleshoot. You may find a hint about the cause of an xrun in the jack messages log. On my system this is usually Renoise, but it really depends on what you’re doing with it, as some effects seem to be particularly nasty. OopsFly started to build a list in the other thread - it should be somewhere around page 2, iirc.

Regarding the the CPU governor, it should always be set to performance - I didn’t mention it, but this is really a hard requirement for real time audio.

If you want to be sure not to forget anything, there’s a good checklist there:

http://wiki.linuxaudio.org/wiki/system_configuration#how_do_i_build_a_real-time_audio_workstation_on_linux

Then you can use the Quickscan script to verify your config:

https://github.com/raboof/realtimeconfigquickscan

6

got some interesting responses on linux audio users list:

http://thread.gmane.org/gmane.linux.audio.users/99939

From: Kjetil Matheussen:

This thread could be a watchdog thread, in which case a priority of 95

should be unproblematic.

The glitches and XRUNS you see are probably due to other reasons

than thread priorities. Renoise is probably doing some time consuming things

in the jack thread, or is the victim of priority inversion, or waiting for

a non-realtime thread doing some work.

7

Yes, I also think that could be the main problem for blocking sub 8ms latencies.

Yesterday I did a test with interesting results, however. Regarding the jack and alsa driver, and performance. I had a song that bombed cpu load above 90% at some stage. The same song, with about the same latency settings, happily played on with the alsa driver - without glitches also. Could it be the jack driver implementation just isn’t that perfect, and the 95 rr worker threads work much better with the alsa driver? But it is hard to tell, I don’t think with the alsa driver you have any option to see xruns?

8

But it is hard to tell, I don’t think with the alsa driver you have any option to see xruns?

Might be possible to see xruns with alsa oops → http://www.alsa-project.org/main/index.php/XRUN_Debug Don’t know, maybe some low latency kernels are compiled with that ‘xrun_debug’ proc support :slight_smile:

9

Thank you for this pointer! I’ll rebake my realtime kernel with that interface enabled once I have some time for these games - I guess it costs performance, so it isn’t present in the ubuntu lowlatency & my realtime kernel config.

10

Yes, I also think that could be the main problem for blocking sub 8ms latencies.

Yesterday I did a test with interesting results, however. Regarding the jack and alsa driver, and performance. I had a song that bombed cpu load above 90% at some stage. The same song, with about the same latency settings, happily played on with the alsa driver - without glitches also. Could it be the jack driver implementation just isn’t that perfect, and the 95 rr worker threads work much better with the alsa driver? But it is hard to tell, I don’t think with the alsa driver you have any option to see xruns?

my only thought about this is how low can latency get with pure ALSA?

i haven’t tries bigger latencies (longer periods, more period - like 5 x 1028 or something similar) with my test-case (Renoise+SuperCollider+Processing) - just for testing sake, since doing synced AV really needs good low latency.

11

I have no good experience with straight alsa, I usually get lower latencies with jack.

OopsFly: you sure you’re not outputing to dmix or sth like that?

12

Sorry, forget about that shit.

I’ve had the pulseaudio sink/source loaded :blush: . Unloading them I could drive the tune without problems or dropouts, and with similliar renoise cpu load values as with alsa, maybe even lower. Will do systematic testing for alsa/jack anyways, without the pulse module ofc. And config my system so I can load/unload the pulse/jack bridge only on demand and have an indicator of it being loaded (its so damn useful at times…). It really seems to rob jack stability “the big of”!

13

do you guys do hard RT kernel compilation?

like with PREEMPT/RT patch - https://rt.wiki.kernel.org/ ?

is there anyway to do kernel compilation on Ubuntu in a way that creates an deb package/s? i used to do that on debian, but i read somewhere years ago that ubuntu has a kind of architecture regarding kernels which does not allow custom kernels via apt-get/deb.

I have an nvidia module that needs to be compiled for the current kernel and i’m affraid this will take me a lot of time to figure out when i’ll get onto it.

14

Yes, the tool “make-kpkg” can make kernel .deb packages with custom version tags, you can install them with the “dpkg -i” tool and then boot via grub. You need the kernel sources, the corresponding version rt patch (obviously you’re bound to kernel versions that you have a rt patch for), and a good kernel configuration. I’d recommend finding a rt tuned detailed “.config” from close the kernel version you’re using & then import it with “make oldconfig”, tuning configs for rt is a black art by itself. tuning for a specific distro is so, too, I’ve had lots of experiments till I found something that worked with ubuntu, and that worked quite well. This seemed to work well with ubuntu 14.04: https://github.com/yomguy/Linux-realtime-config - it is not so important to find the exact kernel version for your distro, just like…all vital programs on your linux be compatible with it, so choose one close to or above to the version the original distro came with.

Google the details from here yourself, it’s been a while since I built my kernel. It takes quite a while until the whole kernel has been generated, even on a potent machine, you burn lots of electricity with it. Also with this kind of system modifications, it can always come like this kernel just won’t boot or perform well, and you need to find another config & compile again. I’ve also had kernel configs that didn’t boot with ubuntu (from arch or so), but the kpkg way leaves old kernels selectable in grub for safety. Using the proprietary AMD/Nvidia drivers, even with rt kernels, can also degrade rt performance, opensource graphics drivers should work best.

Without all the other tunings (irq priorities, cpu frequencies) it is not so effective, try to apply all other stuff first and measuring. I also remember: ubuntu 14.04 had a real bad open source nvidia driver, and updating the kernel to lowlatency versions from more recent ubuntu versions (are in the apt-get repos) already make a vital difference.

15

RT patch, yes and I build usually with “make localmodconfig” -> this builds a very light kernel with only the modules currently loaded on your machine.

Which also means that if for example your USB key is not connected at that time, you will have no USB storage at all, so careful with that :wink:

Unfortunately I cannot help you much with building for Ubuntu, but I think you’ll find all the resources you need here, in the distro specific section:

https://www.linuxmusicians.com/index.php

16

Oops, sorry, I think I mixed this up a bit. Once I had the kernel source tree (just the whole release from www.kernel.org) patched and configed, I think I built it with the kernel source intern “make deb-pkg”, and not “make-kpkg”. google this (maybe together with “ubuntu”), you can give it parameters for multicore machines and some tags to indentify your kernel in grub, and also omit the headers in case you don’t need them etc. It will generate some .deb packages in the directory above the kernel source tree that you can install with dpkg. Patching the nvidia (or amd) drivers to work with such a kernel is another task, I didn’t do it because those drivers are too aggressive and can degrade realtime performance. I think you’ll need an extra patch (from the community) for the nvidia drivers, complicating things and narrowing kernel versions you can use. If one day I decided to do graphics stuff with those drivers, I’d rather make a 2nd ubuntu installation for 3d and gaming.

The tip with the localmodconfig is great if you’re sure you won’t change/expand hardware, but I don’t know how an rt tuned config like the one I linked above could be integrated with this.