When John Adams moved from Philadelphia to the White House, the building was still under construction and there was no sign of a garden.