test add tokenizer

2026-08-28 00:54:20 +07:00
1 parent 7e8c38a80f
commit 9dfae88805
@@ -11,3 +11,61 @@ docker run -it -p 5500:5500 synesthesiam/opentts:ko
 ```

 It's required to have Docker running.
+
+## Full Text Search
+
+This is done with [kuromoji-ko](https://www.npmjs.com/package/kuromoji-ko).
+
+```js
+import { MeCab } from "kuromoji-ko";
+
+const mecab = await MeCab.create({
+  engine: "ko",
+  dictPath: "./node_modules/kuromoji-ko/dict",
+});
+```
+
+### Tagset
+
+From <https://www.sketchengine.eu/hannanum-korean-tagset-simplified/>.
+
+| ORIGINAL          | SIMPLIFIED (in Sketch Engine) | DESCRIPTION    | Example               |
+| ----------------- | ----------------------------- | -------------- | --------------------- |
+| NOMINALS          |                               |                |                       |
+| NC                | **N**                         | noun           |                       |
+| NQ                | **NP**                        | proper noun    |                       |
+| NB                | **N**                         | noun           |                       |
+| NP                | **Pron**                      | pronoun        |                       |
+| NN                | **Num**                       | numeral        |                       |
+| PREDICATES        |                               |                |                       |
+| PV                | **V**                         | verb           |                       |
+| PA                | **Adj**                       | adjective      |                       |
+| PX                | **VAux**                      | auxiliary verb |                       |
+| MODIFIERS         |                               |                |                       |
+| MM                | **Det**                       | determiner     |                       |
+| MA                | **Adv**                       | adverb         |                       |
+| INTERJECTION      |                               |                |                       |
+| II                | **Interj**                    | interjection   |                       |
+| RELATIONAL SUFFIX |                               |                |                       |
+| JC                | **Suff**                      |                |                       |
+| JX                | **Suff**                      |                |                       |
+| JP                | **Suff**                      |                |                       |
+| ENDING            |                               |                |                       |
+| EP                | **Suff**                      |                |                       |
+| EC                | **Suff**                      |                |                       |
+| ET                | **Suff**                      |                |                       |
+| EF                | **Suff**                      |                |                       |
+| AFFIXES           |                               |                |                       |
+| XP                | **Pref**                      |                | 접두사                |
+| XS                | **Suff**                      |                |                       |
+| SYMBOLS           |                               |                |                       |
+| SP                | **Sym**                       |                | 쉼표                  |
+| SL                | **Sym**                       |                | 여는 따옴표 및 묶음표 |
+| SD                | **Sym**                       |                | 이음표                |
+| SU                | **Sym**                       |                | 단위 기호             |
+| SF                | **Sym**                       |                | 마침표                |
+| SR                | **Sym**                       |                | 닫는 따옴표 및 묶음표 |
+| SE                | **Sym**                       |                | 줄임표                |
+| SY                | **Sym**                       |                | 기타 기호             |
+| FOREIGN WORDS     |                               |                |                       |
+| F                 | **X**                         |                |                       |