It looks like it is adding up the star value * number of votes for it, then dividing that by five instead of the number of people who have rated the mission. So the more people who have rated your mission, the higher your rating, regardless of how many stars they gave it.
For example, a mission I'm looking at right now has a 13.4/5 average rating (and 5 stars). It has 2 5 stars, 3 4 stars, 9 3 stars, 6 2 stars, and 6 1 stars. That works out to be 10 + 12 + 27 + 12 + 6 or 69. 69/5 = 13.4, what it shows currently. What it should be is 69/28 = 2.5.
The more people rate your mission, regardless of how they rate it, the higher your "average" will be, because of it dividing by 5 rather than the number of people. So every 5 people rating you 1 star will boost your average rating by 1. And because you have that higher average rating, well, more people are going to play your mission =/