I agree that the current system isn’t perfect, and most of the points you raise are perfectly valid.
The whole building-block approach to modulation is a pretty unique one, and could be abstracted/optimized even more (as you suggest).
But, IMHO the most important thing for us to focus on right now isn’t that the modulation system involves some redundancy (it certainly does), but rather that we actively seek out and identify things that simply aren’t possible (a couple of examples: smooth movement, asynchroneous LFOs)