I have also only "heard" there is a penalty. The information random players have told me was that only 2 similar type consoles stack, and after that there are penalties. I also don't know if it is true. I don't see why there would be any penalty, bc when you increase one aspect by using those spaces, you lose opportunities for other things your ship might need.
Given the limited amount to which anything is actually stackable, there really isn't a need for any sort of penalty. Things stack additively, so it doesn't run off exponentially. The only seeming exception is resists, and if you look at them as extending effective health against a given type of damage, they stack linearly (additively) too.
What I seem to recall, not ingame atm, is that if you stack two of the same consoles that the second losses about 30% effectiveness but does stack. So, if you have two +10's that you end up with a bonus of +17. Not sure that is the exact percentage or if it is consistent between console types.
The confusion stems from the fact that the resistance bonus listed on resist items isn't the percentage resistance that the item gives, but the percentage increase in the amount of damage the item will let you take.
For example, a +20% Phaser resist console will increase the amount of phaser damage it takes to destroy your ship's hull by 20%. Stacking 5 of them will increase the amount of phaser damage it takes to destroy your hull by 100%.