Hij had een relatie met Caroline Sackville
Joseph Damer, 1st Earl of Dorchester (12 March 1718 - 1798) was a wealthy landowner particularly associated with the reshaping of Milton Abbey and the creation of the village of Milton Abbas in Dorset, south-west England.
Born into a wealthy family (his great-uncle was a money-lender in Ireland), Damer was educated at Trinity College, Dublin in 1734-5, and became Member of Parliament (MP) for Weymouth in 1741 at the age of 21. He later represented Bramber in Sussex (1747) and Dorchester (1754). Damer was created Baron Milton of Shrone Hill, Tipperary, Ireland on 30 May 1753 and Baron Milton of Milton Abbey on 10 May 1762.
He married Lady Caroline Sackville, daughter of the 1st Duke of Dorset on 27 July 1742. Ten years later, he purchased Milton Abbey and embarked on an ambitious project to reshape the surrounding valley.
He replaced some existing buildings at the Abbey with a mansion house (designed initially by architect John Vardy, then by Sir William Chambers, and completed by James Wyatt) for his own use. Landscape gardener Capability Brown was commissioned to remodel the surrounding grounds.
As a wealthy landowner Damer also set about the systematic removal of the neighbouring small town of Middleton and its residents. By 1780, most of the residents had been relocated to a new purpose-designed and built model village, Milton Abbas, approximately half a mile south-east of the Abbey; the town's school was moved to Blandford Forum, 7 miles (11 km) away. The original town was razed to the ground and landscaped, most of the site disappearing beneath a new ornamental lake.
When his wife died on 24 March 1775 at the age of 57 he commissioned the Italian sculptor Agostino Carlini to create a magnificent tomb to her memory in the Abbey Church.
Earlier, in 1751, Damer also commissioned Vardy to build him a London residence on Park Lane. After Damer became the first Earl of Dorchester and Viscount Milton in 1792, this mansion became The Dorchester. Although Vardy's mansion was replaced by an Italianate building during the mid-19th century, the name lives on: today it is a world-famous luxury hotel: The Dorchester.
He had three sons. The eldest, John, born in 1744, married the sculptor Anne Seymour Conway in 1767. She separated from him after seven years. Deep in debt, John Damer shot himself in 1776. The second son, George, born 1746, succeeded his father as Earl of Dorchester.
SOURCE: Wikipedia: https://en.wikipedia.org/wiki/Joseph_Damer,_1st_Earl_of_Dorchester