Patterson was a highly-touted recruit in 2012 coming out of Hutchinson Community College. Rated as a 5-star prospect by 247Sports.com and the top JUCO player in the country, Patterson received an athletic scholarship to attend the University of Tennessee, where he played for the Tennessee Volunteers football team in 2012. He made an immediate impact with his athletic ability and offensive versatility. Primarily used as a wide receiver, he also excelled as a return specialist and all-purpose running back. He scored at least one touchdown by reception, rush, kick return and punt return, with 10 total touchdowns in 12 games. In 12 starts, games, Patterson racked up an school-record 1,858 all-purpose yards (set by Reggie Cobb in 1987) to rank first in the SEC, tied for 18th in NCAA.