Why is u32/i32 faster than u8?

I solved day 5's advent of code problem with a rust program.

There I mainly used >!Vec::sort\_by, a HashSet::<(i32, i32)> plus some recursive depth first search!< to solve it.

At first I didn't bother checking how big/small the numbers is in my input so I just used an i32 to store them.

After getting the right answer I was curious to see how much faster the code would be with the "right" datatype so I changed the numbers to u8. Then the code ran \~50% slower! I tried with i8, i16, u16 all with about the same performance as using u8. Changing to u32 didn't make a difference compared to i32, and using u64/i64 made it 73% slower.

I don't know the x86\_64 instructions, but I've assumed the cost of widening the types would be free on a superscalar architecture even if it needed special instructions to do so. What else could cause the slowdown? Worse hashing performance?

\---- Edit after comments:

The code is here: https://pastebin.com/y0xVWHfs

It uses these crates:
aoc-runner = "0.3.0"
aoc-runner-derive = "0.3.0"

but one could just include the input file and add a main that calls input\_generator with it, followed by calling solve\_part2 with the result from input\_generator.

I've benchmarked it with cargo-aoc bench, it compiles the code in release mode when benchmarking. It only benchmarks the solve\_part2 function, not the input\_generator call.