I get what you mean, I also have been thinking about it. I think Gem’s should not be powerful effects, but have rather specific functions. Very similar to what we have now, but more targeted and make it so enchantments cannot give those bonuses. That way gems become valuable for a specific function.
I explained it before on this post as a reply to @RomoloHero.
Elemental resistances, stamina, regen, focus regen, Armor, stuff like this that support a certain build. But because enchantments are pretty damn uninteresting at the moment, I think it feels like gems are kind of bland and feel a bit of a filler system. As if the devs were like “let’s have gems in the game, because… gems are cool right?”
I dunno it feels very rough-cut at the moment, but hey that’s just Early Access for you.