The 2012 Wimbledon finalist and his older brother won the first set 7-5 but then couldn't find a way past the Austrian pair, losing the final two sets 7-6 (8-6) 7-5.
Jamie was unable to dominate on his serve and Melzer and Peya broke him three times in the final set to seal a place in the second round.
"It was tough, we played well," said Andy.
"We were a break up twice in the third set. We had enough chances to win the match, each time we got the break we got broken straight back so it was tough.
"They (the crowd) did a great job. They were really loud from the first point to the last. They helped a lot, you don't always get that for doubles matches and it was a really nice atmosphere to play in."