He has faced criticism over his role in the Post Office scandal.
Yvonne Tracey worked at the post office for more than 30 years and said his attitude “must be challenged”.
Sir Ed, MP for Kingston and Surbiton, initially refused to meet campaigner Alan Bates when he was a minister.
He has now said he was “deeply misled” by Post Office executives.