Ives was born in 1909 near Hunt City, an unincorporated town in Jasper County, Illinois near Newton, Illinois; the son of Levi "Frank" Ives (1880–1947) and Cordelia "Dellie" White (1882–1954). He had six siblings: Audry, Artie, Clarence, Argola, Lillburn, and Norma. His father was at first a farmer and then a contractor for the county and others. One day Ives was singing in the garden with his mother, and his uncle overheard them. He invited his nephew to sing at the old soldiers' reunion in Hunt City. The boy performed a rendition of the folk ballad "Barbara Allen" and impressed both his uncle and the audience.
Ives had a long-standing relationship with the Boy Scouts of America. He was a Lone Scout before that group merged with the Boy Scouts of America in 1924. The collection of his papers at the New York Library for the Performing Arts includes a photograph of Ives being "inducted" into the Boy Scouts in 1966. Ives received the organization's Silver Buffalo Award, its highest honor.The certificate for the award is hanging on the wall of the Scouting Museum in Valley Forge, Pennsylvania. Ives often performed at the quadrennial Boy Scouts of America jamboree, including the 1981 jamboree at Fort A.P. Hill in Virginia, where he shared the stage with the Oak Ridge Boys. There is a 1977 sound recording of Ives being interviewed by Boy Scouts at the National Jamboree at Moraine State Park, Pennsylvania; on this tape he also sings and talks about Scouting, teaching, etc. Ives is also the narrator of a 28-minute film about the 1977 National Jamboree. In the film, which was produced by the Boy Scouts of America, Ives "shows the many ways in which Scouting provides opportunities for young people to develop character and expand their horizons."
From 1927 to 1929, Ives attended Eastern Illinois State Teachers College (now Eastern Illinois University) in Charleston, Illinois, where he played football. During his junior year, he was sitting in English class, listening to a lecture on Beowulf, when he suddenly realized he was wasting his time. As he walked out the door, the professor made a snide remark, and Ives slammed the door behind him. Sixty years later, the school named a building after its most famous dropout. Ives was also involved in Freemasonry from 1927 onward.
On July 23, 1929 in Richmond, Indiana, Ives did a trial recording of "Behind the Clouds" for the Starr Piano Company's Gennett label, but the recording was rejected and destroyed a few weeks later. In later years, Ives did not recall having made the Record.[