The many illustrious characters who visited Barnes while he resided at the Rectory render it a place of great historical significance. Thomas Hardy is viewed as the most noteworthy.   Hardy and Barnes met while Hardy worked as an apprentice architect in Dorchester. Hardy would often seek the older Barnes for advice at his school next door. While there was a 40-year age gap between the two, Hardy formed a deep bond with Barnes and built his own home, Maxgate, near the Old Came Rectory. He was a regular visitor there until his death. Tennyson and Palgrave were also received at the Rectory, pleasantly surprised by such a cosmopolitan Victorian gentleman living in this quaint village.

Although Barnes was well known for his writing and poetry, he was also a natural linguist who spoke 70 languages and kept diaries in Italian, Spanish and Welsh. He was revered by the local community  given his determination to write his poetry in the local dialect. He was also a gifted mathematician and inventor; his intellectual curiosity and determination as an educator and student knew no bounds. Well after his death, the Rectory continued to be a meeting point for scholars, poets, writers and members of the clergy.

Siegfried Sassoon rented the Rectory in the early 1920s with fellow poet Edmund Blunden, and while there T.E. Lawrence (aka Lawrence of Arabia) came to stay. The three of them went to see Hardy, by then a much-respected elderly man. Hardy was able to regale the three with tales about William Barnes, his friend and mentor. Portraits of Barnes, Tennyson, Hardy and T.E. Lawrence, painted by the gifted portrait artist, Warren Prosperi, can be viewed at the Rectory, adding gravitas to the walls.