The German suffered the fracture in a crash on the first stage of the three-week race but rode on with a special cast before finishing 12th in Monday's ninth stage time trial.
"It's difficult for me to step out of the Tour and leave my team mates here to keep on fighting," Martin said in an Omega Pharma-Quick Step statement.
"But it's the right thing to do now, and I know that even the guys will understand.
"I have still three weeks before the Olympics. I know that it won't be easy, but I will do all my best to be in good shape."
The road cycling time trial at the Olympics is on August 1.