The split among Real fans over their support for coach Jose Mourinho, in the spotlight after his side slipped 11 points behind La Liga leaders Barcelona, was evident at the Bernabeu as some sections whistled when others chanted his name.
Qatar-owned Malaga, who are fourth in La Liga and have cruised into the last 16 of the Champions League, were clinging on desperately at the end of their last-32, second leg but had luck on their side.
Cacereno winger Chapi pounced on a loose ball after a free-kick to put the visitors ahead in the 37th minute and they sensed an upset towards the end of the second half.
Malaga defender Martin Demichelis diverted a goal-bound shot just wide of his own post and goalkeeper Carlos Kameni pushed another shot into the side-netting in the closing stages.
When the visitors did finally bundle the ball home again at a corner the 'goal' was ruled out for offside and Manuel Pellegrini's side hung on to go through.
"Without the correct attitude you cannot play football," Pellegrini, who famously suffered a 4-1 humiliation to third-tier Alcorcon as Real Madrid coach, told a news conference.
"If we had gone out it would have been fair enough."
Mourinho gave four B team players a start against third-tier Alcoyano, including an official debut for 21-year-old winger Denis Cheryshev, who made his first appearance for Fabio Capello's Russia in a friendly this month.
However, it was the established figures of Jose Callejon with two goals and substitute Angel Di Maria with one who struck late for a 3-0 victory as Real triumphed 7-1 on aggregate.
Mourinho sent out assistant Aitor Karanka to speak to the media after the game and answer questions about the whistles.
"You hear a lot of things in the dugout," he said. "People at the stadium have the right to express themselves.
"There will be people who are in favour and those who aren't but no one can doubt our professionalism."
Cordoba upset Real Sociedad by taking advantage of some poor defending to draw 2-2 at Anoeta and advance 4-2 on aggregate.
The Andalucian club's prize is likely to be a meeting with holders Barcelona, who host Alaves on Wednesday with a 3-0 first-leg advantage.
Las Palmas ousted Rayo 1-0 on aggregate after grabbing a goalless draw in Vallecas where the home side had their goalkeeper sent off for the second time in the tie.
Real Betis, spurred on by their surprise 1-0 defeat of league champions Real at the weekend, overturned a first leg deficit against Valladolid to progress 3-1 on aggregate.
Osasuna joined them with a 2-0 home win over second tier Sporting Gijon, giving them a 2-1 triumph.