The family name of the Earls of Rosebery is Primrose, a name associated with lands in Fife owned by Dunfermline Abbey. The earliest known member of the family, Henry, was born in 1490; his grandson James became Clerk to the Privy Council under James the VI and I and died in 1640.

James' son, Archibald Primrose, succeeded his father as Clerk to the Privy Council during the perilous period leading up to the Civil War. Praised by Charles I for his "fidelity, judgement and discretion", he fought on the Royalist side, lost his estates and, at one point, was under sentence of death. His capable and unswerving service to the exiled Stuart kings earned him a knighthood at the Restoration and appointment as Lord Clerk Register of Scotland. In 1662, he bought the Barony of Barnbougle and Dalmeny and moved into the 13th-century tower house on the shore.