Born in Illinois, Rawlins practiced law there after being admitted to the bar in 1854. In 1861, at the outbreak of the Civil War, Rawlins met Ulysses S. Grant, who was raising a regiment from Galena to answer President Abraham Lincoln's call for troops. He initially served as a volunteer aide-de-camp, but at Grant's request, Rawlins joined the United States Army as a captain and assistant adjutant general under Grant's command. Rawlins remained with Grant throughout the war, in roles of increasing responsibility and rank, including Chief of Staff of the Army of the Tennessee and of the Military Division of the Mississippi. He was known for his great attention to detail, as well as being a stickler for proper protocol. He was promoted to brigadier general on August 11, 1863. When Grant was promoted to general in chief of all the Union armies, Rawlins became Chief of Staff of the General Headquarters of the United States Army. He was promoted to brevet major general on February 24, 1865, to brigadier general in the regular armyMarch 3, and brevet major general in the regular army on April 9.
Rawlins remained with Grant even after the general was elected President, serving as Grant's first Secretary of War. However, Rawlins had contracted tuberculosis, and his failing health caused his term in office to be brief (March 11 – September 6, 1869). His doctors recommended that Rawlins go to Arizona, where the dry desert climate would allow him to live longer. Rawlins refused, wishing to stay at Grant's side as his Secretary of War. He died in Washington and was buried in Congressional Cemetery, but his remains were later relocated to Arlington National Cemetery.
Rawlins devoted his efforts to maintaining Grant's public image during the war. Grant was known before the war for trouble with alcoholism, but it was revealed, in a letter from Rawlins to Grant (which Grant never saw), that Grant maintained his sobriety during his command of the Army. In this letter, made public in 1891—several years after Grant's death—Rawlins wrote, "I find you where the wine bottle has been emptied, in company with those who drink, and urge you not to do likewise." Rawlins noted that this advice was "heeded, and all went well", thus proving that Grant was not impaired by drink when his decision-making was critical.
There was speculation that by the time Rawlins died, he and Grant had grown distant and that Grant no longer needed Rawlins's constant fussing over his image. When Rawlins died, only his temporary successor as Secretary of War, General William Tecumseh Sherman, was at his bedside. In his memoirs, written shortly before his death, Grant only mentioned Rawlins twice, and essentially ignored their professional and personal relationship. Surviving members of Grant's former staff were outraged at the fact that Grant would snub someone who had been as loyal to him—literally to the death—as Rawlins had been. The most likely explanation for this is given by historian E.B. Long, who wrote, "It might be that Grant did not wish to praise Rawlins too profusely because of the current reports picturing Rawlins as the protector of Grant from his own bad habits."