They say the building at Kings Lane has fallen into disrepair and attracted anti-social behaviour, causing fears over public safety.
Runnymede Borough Council says knocking it down will address these concerns while longer-term plans are developed.
These could include new leisure and community facilities for the area.
