They weren't first generation UEL.
They weren't from the British Isles (those folks arrived some 20 years later.)
They were from the new United States, coming to Upper Canada for land and new prospects.
(Or perhaps, I've read, some folks didn't care for that new republic so much after all. )
Some were second generation UEL, migrating from their family's original land grant.
And they all lived at Grover's Tavern.
That is, they all lived in the area around the tavern operated by John Grover (arrived 1798) until 1832, when the growing village took the name of Grafton.
It appears that Grover's hometown of Grafton, Massachusetts inspired the change. (Facts from the self-guided walking tour guide produced by Heritage Alnwick/Haldimand.