But, it has really helped me and I haven't felt that it is a dangerous distraction.
If you know the code and you're just listening to the words then it's probably going to be no more distracting than listening to voice radio. However, if you're trying to learn the code (or learn to read it faster) you're likely to have to concentrate on it a bit more, and that'll be a bit more distracting. Road conditions and familiarity will also affect the distraction level you find tolerable.
If I'm on a motorway in light traffic (or very slow, or moderate traffic on a familiar route) then I can try listening to morse from a known source. Even so I'll probably only catch a third of it...