Collider shapes influence simulation even when not colliding.

If I enable or disable a collider, the rigidbody behaves completely different. Even if the collider is never colliding. I already found out that adding colliders changes Center-Of-Mass and the Inertia-Tensor. So I override the center of mass and inertia tensor in Awake(), and still the simulation is completely different, depending on whether the rigidbody has a collider or not. Is there something else that the collider does to the rigidbody, other than change its C.o.M. and Inertia Tensor? Thanks!