My go at the one billion row challenge: past and current roadblocks, and what I tried to make it faster than the reference
Hi everyone,
so I have been working on this solution on and off for a few weeks (months?). At first I couldn't get faster than 12 minutes. I got discouraged and stopped for a while.
Then I found out about target-cpu=native and got to around 2 minutes. Why is that not the default? Seems weird you have to check the "make my code run fast" box in a language that prides itself in being safe AND fast.
Mutlithreading barely made it faster because only two threads where running at the same time and the second one only started when the first was halfdone.
Then moving from BufReader::read\_lines to reading Bytes made it run in around 8 seconds. Somehow that change made my threads run in parallel. Don't know why though.
I did some other smaller things and managed to get it to 5.8s (although, again, I don't know why because after tinkering a little but then reverting back to that same code I can't get below 6.2s and it's mostly at 6.7s)
I also realised how much nicer channels are to work with over Arc-Mutex. (I think) my code looks way cleaner that way.
Now my question is: can I make it faster still without resorting to libraries? My solution is still \~1.6x slower than the reference.
I tried using &str and &\[u8\] instead of String as Hashmap keys but then I get borrowing issues I'm not sure how to fix. And I don't know how to create a wrapper around a byte-slice, especially not a more efficient one than the std::String. Looking at the fastest java solution I saw that they used some bitmask magic to parse the lines but I rather look at other ways to make my code faster first, especially since the reference implementation manages to be faster without that.
Also, are Hashbrown and FxHashmap really faster than the standard hashmap? It doesn't change much for me. Maybe I'm missing something there. But it doesn't really matter since I don't want to use libraries for now anyway and most of the time seems to be spent reading and parsing the file.