Install the one that matches your OS ofc. Now when installed and running right click its tray icon and select 3D or Direct X 3d. Now select Settings.
Make sure AA/AAA/AF are all set to app controlled. I have never had a problem with AA and AAA forced with STO but some people do. So I would suggest setting it to App controlled for all.
Now heres the part that should help: From there select: Additional on the little Tabs.
Now look down to where it says: Geometry Instancing. If its unticked (Driver Defaults to Unticked) Tick it and press apply and ok. Now you can quit ATI tray tools if you like.
You never have to do this again as long as you don't install another driver.
Now fire up STO and see if the problems gone and I suspect it will.
I have never encountered any problem with Instancing on in any game. In fact some like STO and Gothic 3 rely on it. All it does is say you have a rock. Instead of loading 50 instances of it into your GFX memory it just loads the 1 because if said rock is the same in those 50 instances then whats the point of loading it in 50 times when it can be cloned from the 1 loaded rock Basically it.
Not all games use it but there absolutely no adverse effects from leaving it enabled. If I disable it in STO or Gothic 3 then those games are simply unplayable because of crazy artifacing.
Hope this helps and please try it. Only takes 5 mins tops to try.