Gems offer bonuses to your ship or pet. There are 3 different categories for gems which are also separated by levels, the higher the level, the better the bonus.
It is possible to remove these bonuses and put them back into your storage, but depending on the level of the gem, it will cost gold or pearls. However, placing the gems on your ship doesn't cost anything.
If you have 4 gems of the same level (does not need to be the same type) you have the ability to forge them together to make stronger gems.
Forging 4 gems will result in 1 of the gems being upgraded. It is totally random which of the 4 gems gets upgraded (but you can use materials to help increase the chance of certain gems) and the other 3 gems will be completely destroyed.