Arsenal midfielder Steph Houghton curled home the only goal of the game with an excellent free-kick midway through the second half to secure all three points at the Millennium Stadium.
"It was something special and I'm sure my family are very proud. I don't think it will sink in for a while. I'm just glad it got us the win," said Houghton, who missed the 2007 world and 2009 European championships through injury.
It was a solid start from Hope Powell's side – who started with nine English players and two Scots – but they had to survive a few scares before securing all three points.
Indeed, New Zealand really should have levelled matters in the 72nd minute when a mix-up from Ifeoma Dieke and Alex Scott saw Sarah Gregorius clean through on goal but the New Zealand striker produced a poor low shot and Karen Bardsley was appeal to save easily.
Britain looked very nervous in the opening stages of the game but settled into things and were the better team in the closing stages of the half with Anita Asante come close with three headers, the first of which came off the post.
Eniola Aluko also had a close range shot saved by Jenny Bindon in a crowded box in first-half stoppage-time.
Team GB actually created fewer chances in the second-half but Houghton's goal brought relief. From just outside the D and in a central position she took full advantage of a fragmented New Zealand wall and poor positioning from Bindon to curl the ball into the middle right corner of the visiting keeper's net.
Britain then survived Gregorius's poor effort while Bardsley also had to tip over a shot from Amber Hearn in the 84th minute.
Nevertheless, Britain held on and are now in a great position to qualify from their group with the top two and best two third-placed teams all qualifying out of just three groups.
Team GB return to Cardiff to take on Cameroon on Saturday in their second group match before facing Brazil at Wembley three days later.
In the day's other games, reigning Olympic champions the United States, bidding for a third successive gold, came from behind to beat France 4-2 while world champions Japan defeated Canada 2-1.
Brazil, among the favourites for gold, thrashed Cameroon 5-0 and Sweden, another possible medallist, thumped South Africa 4-1.
A major diplomatic incident marred the first day of competition when the North Korea women's football team walked off after the South Korea flag was mistakenly displayed before their match against Colombia. The North Koreans went on to win 2-0 after kick-off was delayed for more than an hour.
The US had an unfortunate start to their match with France in the earlier match at Hampden but at least that was only because of what happened on the field.
France raced into a 2-0 lead against the title holders after only 14 minutes before the U.S. rallied to take all three points with Alex Morgan scoring twice, Abby Wambach once and substitute Carli Lloyd grabbing the other.
Brazil, playing in the second match at Cardiff, were highly impressive in demolishing Cameroon with Marta, World Player of the Year five times, scoring twice including a penalty.
Cristiane, who came on as a substitute at the start of the second half, also scored