Just like I wrote here, just talk out loud while using your headphones and if you can hear yourself the volume is quite ok. I only turn the volume up while mastering (up and down, every volume is required) and of course if I just want to listen to the music without any production actions.
Just thinking: If everybody uses references, one day everything will sound the same.
I have to admit that I never ever have used a reference track.