How would you use LLMs and vision-language models in autonomous driving, and could an LLM actually drive the car given hallucination and real-time latency limits?

Not in the control loop. Control runs many times a second and must be verifiable; an LLM's latency varies, and a hallucinated object or rule has no safe failure mode.

Use them off the hot path: auto-labelling and mining rare scenarios from fleet logs, generating simulation scenarios, explaining decisions, and as a slow planner suggesting high-level intent that a fast, verified planner accepts or rejects, as DriveVLM proposes. Anything a VLM contributes online sits behind a deterministic safety layer and is validated in simulation first.