Aubrey deserted Pitt over the Regency and forfeited the office he had accepted only as a prerequisite for a peerage, which was refused him. Denying that he was going into opposition or making the Regency a pretext for avenging his disappointment, he reminded Pitt, 3 Mar. 1789, ‘that I formerly adhered to your adversity and that I now retire from your prosperity’. His protest went unanswered. Having fought Pitt’s battle, as he conceived, in the contest for Buckinghamshire in 1784, he now found his prospects for the county bleak and by October 1789 gave up.
Aubrey’s silent metamorphosis (he never spoke in the House after 1790) was now complete. On 13 Dec. 1792 he voted for Fox’s amendment to the address and was omitted from the list of Portland Whigs. The Foxites listed him among those who could be expected to make a further contribution to the payment of Fox’s debts in 1793.
In 1796 Aubrey found another patron prepared to return Whigs and remained the Crespigny family’s guest until 1812. On 10 Jan. 1797 he was a steward at the Whig Club meeting.
Aubrey solicited a peerage from Fox when his friends came to power in 1806, but was put off with a promise of future satisfaction. He voted for the repeal of the Additional Force Act, 30 Apr. 1806, and was listed among ‘staunch friends’ of the aboliton of the slave trade. On Fox’s death, Lord Grenville was prepared to apply for a peerage for him at the next creation, if he had Fox’s promise. In the ensuing inquiry into this matter, the Duke of Devonshire and Richard Fitzpatrick took Aubrey’s part.
Aubrey did not waver in his opposition in the next Parliament and was in the minorities in favour of scrutiny of places and pensions, 7 July 1807, and of peace by mediation, 29 Feb. 1808. But his health affected his attendance for the rest of that session and subsequently he seldom saw a session out. It is true that he returned to vote favourably for Irish Catholic claims, 30 May 1808; but in 1809 no vote is known after that against ministers on the Duke of York (17 Mar.) and, after acting as a ‘thick and thin’ Whig from January to March 1810, he left the House ill before he could vote against Burdett’s committal to the Tower, 5 Apr., and did not reappear that session. He rallied to opposition on the Regency, but his only other known vote in 1811 was for Folkestone’s motion of 28 Mar. That month he could not be rallied to an extra-parliamentary meeting to promote constitutional reform. At that time he was expected to retire from the House.
Aubrey was the Duke of Norfolk’s Member for Steyning from 1812 and later for Horsham. He voted with opposition on the vice-chancellor bill, 11 Feb. 1813, on the joint paymaster general’s salary, 8 Mar., and in the majority for sinecure regulation, 29 Mar. He supported Catholic relief on 2 Mar. and 13 May, but was absent with leave on the crucial division of 24 May. One vote, against the blockade of Norway, is known in the session of 1814. In 1815 he voted against the property tax, 19 Apr., against the transfer of Genoa, 28 Apr., for Whitbread’s peace motion, 28 Apr., against the civil list, 8 May, and against the Duke of Cumberland’s establishment bill, 30 June. No vote survives in 1816, when he took a month’s leave for illness on 27 Feb and another for bereavement on 25 Apr. From 1817 he resumed fairly regular attendance in opposition and voted for parliamentary reform on Burdett’s motion, 20 May. He voted for inquiry into popular education, 3 June 1818. He signed the requisition to Tierney to lead the Whigs in the House. In the ensuing Parliament he voted for criminal law reform, 2 Mar. 1819, and paired in favour of burgh reform, 1 Apr. As in 1817-18, he was an opponent of repressive measures and on 16 Dec. 1819 voted for consideration of Robert Owen’s plan to alleviate popular distress.
Aubrey died Father of the House, 14 Mar. 1826. Lipscomb wrote of him:
He was not a frequent speaker, but possessed a sound judgment and considerable penetration, intermixed, however, with an inflexibility of temper which approached to obstinacy. In the domestic relations of life, and in the incorruptible integrity of magisterial duties, he set forth an excellent example, was a good landlord, a zealous friend, and a gentleman of highly polished manners.
An obituary, too, described him as ‘not easily influenced, and rarely diverted from his measures’.
