It is completely possible to start your Linux desktop without a display manager, or even a GUI for that matter.