Matt Tubbs fired the visitors ahead six minutes before the break, and after the Gulls had Chris Zebroski sent off for a second bookable offence, Tubbs and team-mate Craig McAllister had second-half penalties saved by Scott Bevan.
Crawley had defender Dean Howell sent off in the 83rd minute and almost paid the price late on when defender Kevin Nicholson crashed a shot against the crossbar, but they managed to hang on to claim a place in the last 16.
Crawley, heavily funded and on course for promotion to the Football League, are the first team from outside the professional divisions into the last 16 since Kidderminster Harriers in 1994.
"It was the least we deserved. We dominated for long periods," Crawley manager Steve Evans said.
"We said we could win, not many people gave us a hope here but we did."
Forgotten men Stuart Parnaby and Kevin Phillips were the heroes as Birmingham carved out another cup day to remember by coming from two down to beat local rivals Coventry 3-2 and reach the fifth round of the FA Cup.
The Blues were on a high after booking their place in the Carling Cup final in midweek but they were brought back down to earth as Marlon King and Richard Wood put the visitors two up with 26 minutes gone.
David Bentley reduced the deficit before half-time with a stunning strike from 25 yards for his first Birmingham goal and, after a sustained spell of pressure, two goals in six minutes from Parnaby and Phillips midway through the second half completed the turnaround.
Sheffield Wednesday came from behind to seal a place in the FA Cup fifth round as three second-half goals saw them beat Hereford 4-1.
Sloppy defending from the hosts allowed Stuart Fleetwood to give the Bulls a ninth-minute lead, but Darren Potter struck Wednesday's equaliser six minutes later with a firm drive from just outside the area.
Clinton Morrison made it 2-1 to Wednesday from the penalty spot in the 70th minute after he had been hauled down in the area by Joe Heath, who was sent-off for the offence, and a fine individual effort from Jermaine Johnson put the home side further ahead seven minutes later when he latched on to Weaver's long ball and finished expertly.
Two minutes later, Morrison found the net from the spot again after Michael Townsend was ruled to have handled inside area.
Rene Gilmartin's error from Ashley Barnes' first-half strike was enough to hand League One leaders Brighton a surprise 1-0 victory at Championship outfit Watford in their FA Cup fourth-round clash.
In a game of few clear cut chances, Gilmartin should have easily saved Barnes' 20-yard effort, which came straight at him, but the Hornets goalkeeper spilled the ball into the net.
Mikele Leigertwood's first goal for Reading had put the Championship side in control but Darius Charles appeared to have earned the hosts a replay with a 78th-minute goal.
But Long grabbed his ninth goal in as many games with just three minutes left to put the Royals into round five.
Jimmy Smith headed Russell Slade's men ahead in the 35th minute, but the Swans levelled on the stroke of half-time through Cedric Van der Gun.
Yet the real drama was still to come as Tate diverted a cross from substitute Paul-Jose M'Poku into his own net to send the League One club into the next round.
Eddie Howe's first home game in charge of Burnley ended in victory as the Clarets reached the FA Cup fifth round with a 3-1 win over Burton Albion.
Howe, whose first two matches have been Championship away fixtures at Scunthorpe and Portsmouth, received a rousing reception when he was introduced to the Burnley fans before kick-off.
Chris Eagles' double and a late goal from substitute Martin Paterson on his return from injury prolonged Howe's honeymoon period at Turf Moor, but not before Calvin Zola had marked his Burton debut with a goal to make it 2-1 and set up a tense finale.