Korver made a 3 in his 79th consecutive game to surpass Dennis Scott and tie Michael Adams for second place on the NBA career list. After Korver hit a 3 on a fast break late in the fourth, Teague's fastbreak layup made it 97-88 with 1:41 remaining.
The Hawks have won nine straight over the Magic at Philips Arena.
Atlanta took the game's first double-digit lead at 57-47 on Millsap's jumper with 9:30 left in the third. DeMarre Carroll followed with a 3 and Millsap hit a layup for a 15-point lead that caused Orlando coach Jacque Vaughn to call a timeout.
"Guys were just getting in position where they were open," Teague said. "I was hitting them, and they were making shots. Paul and Kyle had their pick and pop all night, and they were making them. When I was getting in the lane, I was doing my best to draw defenders and kick it to them."
Carroll's reverse dunk a couple of minutes later put the Hawks ahead by 17 for the game's biggest lead, but the Magic went on a 23-7 run that ended with Andrew Nicholson's hook shot in the final minute of the third and gave Orlando a 71-70 lead.
The Magic, who dropped to 0-3 on the road this season, got back in contention defensively by shutting down the passing lanes and forcing Atlanta to wind down the shot clock on multiple possessions late in the third and early in the fourth.
"We fought our way back into the game," Vaughn said. "That unit that was on the floor gave us a lot of energy on the second night of a back-to-back and got us back into the game, take the lead. Great resolve by that group over there -- the Hawks -- making shots."
E'Twaun Moore and Afflalo hit consecutive 3s midway through the final period to cut the lead to 79-77, and Hawks coach Mike Budenholzer called a timeout.
Afflalo 3's put the Magic up 88-87 with 3:50 remaining, but they didn't lead again.
"There's just no room for error at that point," Afflalo said. "We just couldn't close the game off tonight."
Teague, secure in his role after Atlanta re-signed him to a multiyear contract last summer, is showing the kind of leadership that Budenholzer wants from his point guard.
"I think he's really starting to understand what we're saying," Budenholzer said after his team improved to 3-3. "I think for the last week to 10 days that he's had a lot of opportunities and a lot of responsibility for the way we play. He's doing a lot of good things. I think he's going to get better and better. "
Dana Barros set the NBA mark by hitting a 3 in 89 straight games from Dec. 23, 1994-Jan. 12, 1996, during stints with Philadelphia and Boston. ... Adams made a 3 in 79 straight games during the 1987-88 and 1988-89 seasons with Denver. ... The Hawks had the biggest lead of the first half at eight on a pair of baskets by Scott in the first minute of the first quarter.
Copyright by STATS LLC and The Associated Press
ATLANTA, GA - NOVEMBER 9: Cartier Martin #20 of the Atlanta Hawks shoots the ball against the Orlando Magic on...