The planning inspector said the proposal would "overwhelm" the manor house the hotel is based in.