Spencer, returned on the Marlborough interest at Woodstock, entered the House as a member of the Bedford connexion and a supporter of the ministry. Appointed a lord of Trade in April 1770 he was not a very regular attender either at the Board or in the House, and is only known to have spoken twice in debate; even in his private circle he had a reputation for silence. In January 1781 he suddenly decided to join the Opposition, perhaps as a consequence of his friendship with Fox, though Fox denied having made any attempt to persuade him. His brother, Marlborough, not consulted beforehand, wrote to William Eden: ‘This is a very unpleasant business of Bob’s going so decidedly into Opposition ... As the case stands at present, it is actually my bringing in a member for Opposition.’ Spencer, having lost his official salary, and in financial difficulties which Selwyn described as bankruptcy, was admitted to a twelfth share in Fox’s faro bank at Brooks’s, with a fee of five or six guineas an hour for dealing.
He remained firmly attached to Fox. In May 1782 Fox made a special point of his appointment as vice-treasurer of Ireland, scouting the idea that he should share it with his brother, Lord Charles Spencer, who was Shelburne’s candidate: ‘Besides Lord Robert’s own claims, which are surely strong upon us, it is only reasonable and perhaps politic that it should appear to the world that I should have some weight in the disposition of appointments.’
Spencer died 23 June 1831.