Victoria's decision to walk away from 2026 hosting duties could be a "death knell" for the Games.