Lindera 0.38.0 is released
I’ve released version 0.38.0 of Lindera, a multilingual (Japanese, Korean, and Chinese) morphological analyzer developed in Rust. Morphological analyzers are commonly used in natural language processing, and one well-known example is MeCab, written in C++. Through consistent improvements, Lindera has now achieved speeds comparable to MeCab for simple word segmentation tasks.
Lindera was originally designed with search engine integration in mind. It’s more than just a morphological analyzer—it also includes various features like character filters and token filters tailored to Japanese. With this release, you can now configure the tokenizer (including character and token filters) using YAML, making it even more user-friendly.
https://github.com/lindera/lindera
To make it easier to use with Python—essentially the de facto standard for AI and machine learning products—I’ve provided a Python binding that you can install via pip. Plus, since the dictionary is included, you don’t need to install one separately.
https://github.com/lindera/lindera-py
I’ve also started offering a tokenizer for SQLite’s FTS5 extension as dynamic linked libraries for various platforms. This allows you to perform full-text search with YAML-configurable tokenizers, providing an experience similar to Elasticsearch or Solr. Additionally, by combining it with sqlite-vec, you can enable hybrid search—vector search and keyword search—within a single application.
https://github.com/lindera/lindera-sqlite
For Tantivy, a Rust-based full-text search library, Lindera provides a tokenizer as well. This brings advanced tokenizer configurations—similar to Apache Lucene—to Tantivy, which has been somewhat lacking in Japanese language support.
https://github.com/lindera/lindera-tantivy
Lindera is being used by startups like Quickwit (https://quickwit.io/), Meilisearch (https://www.meilisearch.com/), Qdrant (https://qdrant.tech/), and ParadeDB (https://www.paradedb.com/). Some of these services are even running Lindera in their production environments (Qdrant requires specific build options, though).
Since this is an open-source project, I’d love for you to give it a try if it interests you! And of course, if you’d like to contribute to development, that would make me super happy as a maintainer.