The Queens Men arent named necessarily for their dedication to Queen Selyse. Instead, it refers to their conversion to the Rhllor faith. And while many might be concerned that their true loyalties lie with the Red Priestess, for the time being, they are just as willing to head to battle in service of Stannis Baratheon. In time, their true nature might come to light.