I had the same problem did the mission 5-6 times in group, solo, dropped it picked it up again and it would not complete, until i did it again and completed it by contacting starfleet while i was still on the planet then it showed
as completed and i could choose my reward. Never worked when contacted starfleet after i beamed up and/or left the system.
Now i got stuck at the Kuvagh'Mah misson on regulus...
Resolved here as well, after taking a 9 hour break (sleeeeeep). Of course, this also meant a fresh reboot and restart of the game.
Did not join an open instance team this time (perhaps the closed group setting only catches after a game restart? Would be odd though...), did it solo and it worked. Returned to sector space and called SF from there, mission complete.
I suspect this has a lot to do with serverload, where many players go into the game at the same time and the game decides "hey, people going to the same place, let's put them in the same group"... I strongly suspect that to be the cause of this and other issues where open instances are created for missions that really shouldn't be done using open instance teams...
I have also been having this issue, tried everything suggested here. Dropping the mission, making other one primary, relogging, contacting starfleet before leaving planet, restarting entire computer even. Nothing has worked so far.
Going to leave the mission open for now and try something else. Right now I am working off of a 5 day buddy trial, rather disapointed I ran into this on my first non-tutorial mission.
The problem is totally centered around grouping. I had to run the Beytan V mission 5 times, myself. After that I was so fed up I went back to my RA3 and ran the Hotep mission....would you believe I didn't get credit for it either. My teamate did.