The more stress a Crystal is put under while in the Ritual Altar, the more the Crystal will start to Fracture. This is a percentage and will slowly increase the more stress the Crystal is put under. Once the Crystal reaches 100% fracturing, it will be destroyed. Fracturing can be repaired by dropping the crystal into Liquid Starlight.
The more you empower the Crystal while in the Pedestal, the more chance it will start to fracture. Setting up a Pedestal and leaving it will cause your Crystal to eventually break if not looked after.