I've made a critical semver mistake. Can it be fixed?

Because I was just learning Rust and semver, I published many versions of a crate that have the wrong numbers. I realized my crate should be at 0.x.y, not 3.x.y. If I yank all versions and re-publish as 0.x.y, will I get in trouble when a future version "collides" with an older version of the same number?

This is the version list. I'm planning to yank everything but the 0.\* versions