Mary Goodman will remember him as her dentist, he put in her gold tooth in 1960.
"Still here 52 years later and we will miss him," Goodman said.
Holley became Portsmouth's first black mayor in 1984. Around a table at Victory Center Senior Apartments, those who saw him rise to power are feeling his loss as they're reminded of the joy of him swearing into office.
"It made me feel good and proud," Henrietta Harrell said. "It was something, the first thing that had happened in Portsmouth that was really, really exciting."
Despite all his accolades, Holley's time in office was not all smooth sailing.
He was recalled in 1987, and again in 2010 at the age of 83.
"He was kind of like a mentor to us, and then things like that happen and it didn't feel too good," Harrell said.
Now only the fond memories are coming back as they grieve and miss the man who meant so much to them.