Certain gems preventing item from becoming "Exalted"

Hello, ive just gotten to the point where i can exalt items. For example i tried reaching Exalted status on my Festering Earth sword but because i have the chipped ruby gem slotted in it now giving it heat infusion, it seems to be preventing the sword itself from reaching “Exalted” status, i exalted all the other stats on the sword but when i tried exalting the gem nothing happens, i did it twice to make sure which caused me to waste some material.

And this also seems to apply to the armors ,if you slot for example the health regeneration in combat gem on a armor or the regen focus in combat gem it seems to prevent the items from reaching exalted status since these gems dont have a % stat to it, which really sucks because i cant just change the gem on the armors and i really dont want to change the gems on the weapons since they now give fire/frost/lightning infusion.
The other items i have that has stats/enchantment with a percentage stat im able to reach Exalted status on.