Lingua 1.5.0 - The most accurate natural language detection library for Rust, now with support for detecting multiple languages in mixed-language text
Hello dear Rustaceans,
it took me over a year but now there is finally a new release of my natural language detection library named Lingua.
https://github.com/pemistahl/lingua-rs
Its task is simple: It tells you which language some text is written in. This is very useful as a preprocessing step for linguistic data in natural language processing applications such as text classification and spell checking.
It is still the most accurate library for this task in Rust's ecosystem. Check out the detailed benchmarks in the README. Here is a little foretaste:
In this new release, I've introduced an experimental feature that most other language detectors do not have. It is able to detect multiple languages in mixed-language text. Let me give you an example:
use lingua::DetectionResult;
use lingua::Language::{English, French, German};
use lingua::LanguageDetectorBuilder;
fn main() {
let languages = vec![English, French, German];
let detector = LanguageDetectorBuilder::from_languages(&languages).build();
let sentence = "Parlez-vous français? \\
Ich spreche Französisch nur ein bisschen. \\
A little bit is better than nothing.";
let results: Vec<DetectionResult> = detector.detect_multiple_languages_of(sentence);
assert_eq!(results.len(), 3);
if let [first, second, third] = &results[..] {
assert_eq!(first.language(), French);
assert_eq!(
&sentence[first.start_index()..first.end_index()],
"Parlez-vous français? "
);
assert_eq!(second.language(), German);
assert_eq!(
&sentence[second.start_index()..second.end_index()],
"Ich spreche Französisch nur ein bisschen. "
);
assert_eq!(third.language(), English);
assert_eq!(
&sentence[third.start_index()..third.end_index()],
"A little bit is better than nothing."
);
}
}
Take a look at the release notes for an overview about all new features:
https://github.com/pemistahl/lingua-rs/releases/tag/v1.5.0
Lingua is also available for Python, Go and the JVM. The JVM version does not yet have all the new features but they will be added to it as well, eventually.
If you are into natural language processing, then please check it out and let me know what you think about the new multi-language detection. Thank you very much.