MIDI clock problems


I’ve started working on an AVR project which uses MIDI clock to keep stuff in sync. I’m using an USB to MIDI converter to send MIDI messages to the microcontroller. According to the MIDI specs the clock message is sent 24 times per quarter note. I’ve managed to write some code to read in the messages and count all incoming clock signals and flash a LED on every beat. Well… all good except the fact that not all clock signals seem to arrive. The blinking is a little bit slower than expected and if I do something on my PC (like start a youtube video) it slows down to 1/2 of the original speed. I’m pretty sure my code is not the problem here but I’m unable to eliminate this lagging.
I’m hoping some of you may have some knowledge to help me. Otherwise I feel like this project has doomed already…
Feel free to ask me for any additional information!

I have read one other post where midi timing in combination with USB wasn’t working well. Does it work better if you isolate the USB connection without using any other USB devices on the same USB channel?