Miller was born in Waco, Texas, on October 12, 1919, to Henrietta and Connery Miller. He was the third of four sons and grew up in a strong and loving household. He enjoyed playing with his brothers but was also a considerate child. He often helped around the house, cooking meals and doing laundry, as well as working the fields. Miller was a good student and a fullback on the football team at Waco's A.J. Moore High School. They called him the "Raging Bull" because of his size (5 ft 9 in, over 200 lb - 1,75 m, over 90 kg).
He worked on his father's farm until enlisting in the United States Navy as Mess Attendant, Third Class in September 1939. Following training at the Naval Training Station, Norfolk, Virginia, Miller was assigned to the ammunition ship USS Pyro where he served as a Mess Attendant, and on January 2, 1940 was transferred to USS West Virginia, where he became the ship's heavyweight boxing champion. In July of that year he had temporary duty aboard USS Nevada at Secondary Battery Gunnery School. He returned to the USS West Virginia on August 3, 1941.