Originally from Tampa, Florida, Adderley moved to New York in 1955. His nickname derived from "cannibal", a title imposed on him by high school colleagues as a tribute to his voracious appetite.
Cannonball moved to Tallahassee, Florida, when his parents obtained teaching positions at Florida A&M University. Both Cannonball and brother Nat played with Ray Charles when Charles lived in Tallahassee during the early 1940s.Adderley moved to Broward County, Florida, in 1948 after finishing his music studies at Tallahassee and became the band director at Dillard High School in Fort Lauderdale, Florida, a position which he held until 1950. Cannonball was a local legend in Southeast Florida until he moved to New York City in 1955.
One of his known addresses in New York Corona, Queens. He left Florida originally to seek graduate studies at New York conservatories, but one night in 1955 he brought his saxophone with him to the Cafe Bohemia. Asked to sit in with Oscar Pettiford in place of his band's regular saxophonist, who was late for the gig, the "buzz" on the New York Jazz scene after Adderley's performance announced him as the heir to the mantle of Charlie Parker.
Adderley formed his own group with his brother Nat after signing onto the Savoy jazz label in 1957. He was noticed by Miles Davis, and it was because of his blues-rooted alto saxophone that Davis asked him to play with his group. He joined the Davis band in October of 1957, three months prior to the return of John Coltrane to the group. Adderley played on the seminal Davis records Milestones and Kind of Blue. This period also overlapped with pianist Bill Evans' time with the sextet, an association that led to recording Portrait of Cannonball and Know What I Mean?.
His interest as an educator carried over to his recordings. In 1961, Cannonball narrated The Child's Introduction to Jazz, released on Riverside Records.