Tries from Henry Trinder, Ugo Monye and James Simpson-Daniel had put England in charge before the Barbarians mounted a comeback either side of the interval.
England thought they had escaped with the win with a late try from the Exeter flanker Tom Johnson.
But Visser's second try, scored with the last play of the game after a piece of Fijian magic from Joe Tekori, sealed the Barbarians' win.
The England squad was being coached by Stuart Lancaster but Johnson was running the rule over a handful of fringe World Cup squad contenders.
Monye limped off with an ankle injury while Simpson-Daniel, having enhanced his credentials with a sparkling try, blotted his copybook by allowing Visser to score his first.
In fact, the best performance from an England wing came from Paul Sackey, who was playing for the Barbarians in the socks of his new club, Stade Francais.
England also had Harlequins prop Joe Marler, a contender for the World Cup training squad, carried off on a stretcher in the second half as the Barbarians surged into the lead.
Luke Narraway captained the inexperienced England line-up and was impressive in the first-half, having a hand in the tries scored by Trinder and Simpson-Daniel, but tailed off.
Matt Banahan enjoyed a confident trial run at inside centre - his third different position in an England jersey this season - and he seems certain to travel to New Zealand as a utility back.
The Barbarians squad had a combined total of 539 caps and, as usual in this fixture, they took some time to get going.
England were faster out of the blocks and Gloucester centre Trinder gave them the lead after excellent build-up work from Banahan and fly-half Charlie Hodgson.
Baby hit back for the Barbarians before Monye chipped ahead and then snatched the turnover from full-back Nicolas Jeanjean to score the second.
After Hodgson had landed a penalty - a decision greeted by boos from the 38,680 Twickenham crowd - Simpson-Daniel left Ruan Pienaar in his wake to score England's third.
England finished the half with 14 men after Sale flanker Carl Fearns was sin-binned for killing the ball as the Barbarians' French fly-half Michalak drove for the line.
Michalak then picked off a pass from England scrum-half Paul Hodgson and fended off a weak tackle from Hodgson to score just before the interval.
Visser, the Edinburgh wing, left Simpson-Daniel clutching at thin air as he collected a cross-kick from Michalak and raced clear to score.
Sackey showed good strength to drive Jeanjean forward before some delightful hands from Tekori sent Smith over in the corner as the Barbarians drew level.
Hodgson slotted a second penalty for England, who then took the lead with Johnson's try.
But Tekori's offload sent Van Niekerk clear and his deft pass sent Visser over for the match-winning try.