How to Make a rigidbody fall straight away?

I have a character in a game and when he goes over the edge the character is meant to fall to the bottom of a pit. It all works but there seems to be a slight delay before the character actually starts moving down using gravity. I can literally walk over the edge and back on to the ledge before he actually starts falling (moving down). i have tried putting the rigidbody(of the character) mass up high (3000) and putting the drag down to 0.001 but it really doesn't have much effect. I want the character to start falling straight away when he goes over the edge and not allow the player to bring the character back to the ledge again. Thanks