How do you know if the 'vous' person is for a group of people or to show a mark of respect to the person you're talking to?

The main different is mostly about the context in which the conversation takes place. If a question is asked using the 'vous' person and different voices answer the question, then you know that you are talking to a group of people, while if it is always the same person answering your question, it is one single person to whom you are showing respect (as that person isn't in your close relations).