Calliste said it felt like Oregon was down by 50 points.
"We just knew we had to get stops and rebounds," he said.
"That's probably the most exciting game I've ever played in," said Damyean Dotson, who finished with 15 points for Oregon.
All-America candidate Tyler Haws led the Cougars (8-5) with 32 points, and Kyle Collinsworth added 15 points, five rebounds and eight assists.
"This one hurts. We felt like we had it for most of the game," Haws said.
Oregon entered the contest third in the nation, averaging 89.3 points per game, and the Cougars came in ranked fifth at 88.6. The game lived up to expectations with a tight score and players regularly diving for loose balls.
Even the crowd noise was contested, with Duck fans responding to drown out BYU chants.
Haws scored on his first four shots to give the Cougars a slim early lead. Calliste hit a 3-pointer to tie the game at 18, but Haws responded with two straight layups, getting fouled and making the free throw on the latter, a breakaway with 12:54 left in the half.
Oregon tied the game again at 28, but the Cougars went on a 9-2 run, fueled by three 3-pointers, to take a 32-40 lead at the 6:11 mark in the first.
BYU went to halftime up 41-47, and the lead grew to as many as 10 points in the second half.
The Ducks trailed 77-83 when Young made a 3-pointer with 1:58 left.
Johnathan Loyd made two free throws with 1:12 left to cut the margin to 82-84. Mike Moser blocked BYU's Matt Carlino on the next possession, and Moser made a jumper for the Ducks with less than a minute left.
After a Cougar miss, Elgin Cook was fouled on a shot in the paint with 1.8 seconds left and the Ducks had a chance to take their first lead since the first minute of the game. But Cook missed both free throws and regulation ended with the score tied 84-84.
In overtime, with the score tied at 94, Moser drew a charge by Frank Bartley IV with 46.5 seconds left.
Dotson scored on a drive on the Ducks' next possession.
After Dotson scored on a drive, Collinsworth missed a free throw, and Young scored on another layup on a fast break to give the Ducks a 98-95 lead.
Oregon has started 11-0 only four times in school history, most recently in 2006-07 when the Ducks advanced to the Elite Eight. That team went 13-0 to start the season.
Young came into the game leading the Ducks with 19.3 points per game.
Oregon has seven transfers on its squad this season, including Calliste, Young and Moser, who was averaging nearly 15 points and 7.4 rebounds before the game.
The Cougars were coming off a 64-81 loss to Utah in the Holy War on Dec. 14.
BYU begins conference play its next game at Loyola Marymount on Dec. 28.
"I feel like we've learned a lot in the preseason, and if we can learn from it and stick together, we'll be fine in conference," Haws said.
Haws came into the game leading the Cougars with 22.3 points per game. He has scored in double figures in 26 consecutive games.
Collinsworth returned to BYU this season after serving the last two years on a church mission in Russia. He was averaging 12.7 points, 8.3 rebounds and 5.1 assists entering the contest.