UPDATE:

Thank you VainEldritch,
I updated my drivers to the very latest ones and now the card runs normally
I did not change any settings though, just updated the drivers... even though mine were just a few months old, maybe it was already too much

EDIT:
Eh the fan is still loud... not as much as before but still loud. I hope they fix it