His eldest son died in service in 1756 so it was the second son, Alexander, who became third baronet in 1758, and on his death in January 1772 his cousin Francis (the Admiral’s son) became fourth baronet aged 19. In this portrait Francis is at shown about aged six, which suggests his midshipman’s uniform is either one made for him by his proud father, or simply painted in by Reynolds. He and his sisters were the children of his father’s marriage to Frances Lascelles, daughter of a plantation owner of Barbados. She had been left a widow in 1747 by the death of her first husband, Edward Lascelles. They had six children, three survived becoming the Admiral’s step-children. In 1795, on the death of his cousin, the eldest, Edward, inherited huge wealth including Harewood House, Yorkshire. In 1812 he became first Earl of Harewood.

As fourth baronet, Sir Francis Holburne married in 1786 and had two sons and three daughters. The elder son, also Francis, died of wounds when serving under Wellington at Bayonne in 1814, so again it was the younger son Thomas William (usually called William, 1793–1874) who in 1820 became fifth baronet, inheriting considerable family property, though not on the scale of his Lascelles cousins.