I think it depends how they are doing it.
Don't push my hair into my face and we are fine. I don't want my hair going into my mouth or eyes.
I don't mind if they hold my head or 'bounce' it because it lets me know the speed he likes or that he is getting into it.