70365
I got involved with tokenization (mecab, kuromoji) and got confused as to what
a "word" is in the Japanese language.

**Scenario 1:**

"たべません" analyzed by a tokenizer, outputs the following tokens:

```

    たべ  動詞,自立,*,*   たべる
    ませ  助動詞,*,*,*   ます
    ん   助動詞,*,*,*   ん
    
```

While the result outputs 3 token, I think generally it would be considered a
single word. In this case the word "たべません" would consist of a verb and 2
auxilary verbs (助動詞) attached to it.

**Scenario 2:**

"たべています" analyzed by a tokenizer, outputs the following tokens:

```

    たべ  動詞,自立,*,*   たべる
    て   助詞,接続助詞,*,* て
    い   動詞,非自立,*,*  いる
    ます  助動詞,*,*,*   ます
    
```

I think usually "たべています" would be considered as two seperate words "たべて" and
"います". In this case the word "たべて" consists of a verb and a particle (助詞) and
"います" consists of a dependend (非自立) verb with an auxilary verb attached to it.

**Scenario 3:**

"たべてて" (a contraction of "たべていて") analyzed by a tokenizer, outputs the
following tokens:

```

    たべ  動詞,自立,*,*   たべる
    て   動詞,非自立,*,*  てる
    て   助詞,接続助詞,*,* て
    
```

While I think that "たべていて" would be seperated as "たべて" and "いて", I have no
clue what to do with "たべてて". Following the logic from scenario 2, I would say
we should split at the てる as it is a dependent verb, just like the "います" was
in scenario 2. On the other hand it seems inconsistent to me, that we would
end up with "たべ" by itself...

**Scenario 4:**

"たべながら" analyzed by a tokenizer, outputs the following tokens:

```

    たべ  動詞,自立,*,*   たべる タベ  タベ
    ながら 助詞,接続助詞,*,* ながら ナガラ ナガラ
    
```

I think usually "たべながら"would be considered as two seperate words "たべ" and
"ながら". However this would contradict the logic in scenario 1, since "ながら" is
classified as particle (助詞),just like "て".

**Conclusion and Problem**

As pointed out in scenario 3 and 4, I can't come up with a good and consistent
way to split "words" in Japanese.

I want to programmatically process big amounts of text with meaningful
results. If I consider every token a single word, the data would end up beeing
confusing and unpresentable. On the other hand if I include every particle and
helper verb into a single word, I would end up with far too many unique words.

I am hoping you can show me a middle ground between both extremes.

Is my intuition bad as far as the suggested "splits" go? Am I missing some
important rule here? Are the tokenizer results debatable? Or is there simply
no hope?
CC BY-SA 4.0
2019-07-19T18:37:11.690
69590
2019-08-27T03:39:19.367
34367
question
grammar words particles syntax auxiliaries
Are there any consistent rules as to how to distinguish words in Japanese?
What constitutes a “word” is very different depending on different contexts.
For example, studies based on intonation show that most English speakers
consider “high school” to be a single word, despite it being written as two in
the orthography. In English splitting words is usually done based on the
spelling, which makes the problem a lot easier. In Japanese, there are many
different possible ways to split text into words, none of which are
necessarily the most correct. To complicate things even more, in Japanese
particles are usually considered clitics that attach to the word before them
by native speakers, and thus are written with no space before them; a Japanese
person asked to write Japanese with spaces would be more likely to write 魚を
than 魚 を.

So, if you want a consistent way to split Japanese into words that makes sense
to everyone (or just most people), you aren't going to find it. However, if
you're programmatically splitting this because you're going to process it in
some further computation, the important thing is that you pick a consistent
definition of what constitutes a word and stick to it. The easiest solution to
work with in my mind would be to consider verb and adjective conjugations part
of the word that comes before them (and I would personally consider something
like the iru in -teiru to be part of the same word, but this is really up for
debate), and to consider particles as separate words, as this is probably
easier to work with. If you're looking to transform sentences in ways which
are grammatically meaningful (I'm still not entirely sure what it is that
you're trying to do with these split words), simply splitting into words might
not be sufficient and you might need to get into syntax trees, but hopefully
you won't need that for your use case as it is a pretty enormous task to write
software that can analyze natural language that way.
CC BY-SA 4.0
2019-07-27T18:51:52.893
69737
2019-07-27T18:51:52.893
34822
69590
answer
You are seeing the difference between 日本語文法 taught at Japanese-as-a-second-
language (JSL) school and 国語文法/学校文法 used by native Japanese speakers. The
outputs of tokenizers made by Japanese developers naturally follow the latter.
For instance, 食べません is generally considered a single (conjugated) word by JSL
learners, but it's generally considered as having three words in Japan.
Japanese monolingual dictionaries like 広辞苑 treat て/で, ます, ん and so on as
"words", not part of a conjugation. There is no such a thing as te-form in
国語文法.

  * [Auxiliary verbs in Japanese](https://japanese.stackexchange.com/q/63351/5010)
  * [Does "te-form" of a verb always include て/で? Why?](https://japanese.stackexchange.com/q/36310/5010)  
(This question was made by me, a native Japanese speaker, when I was ignorant
of the difference)

  * [Random Japanese / 活用](https://grapefruitcake.tumblr.com/post/145712193913/%E6%B4%BB%E7%94%A8-conjugation)
  * [Isn't a tag "auxiliary verbs" or "jyodoushi" necessary?](https://japanese.meta.stackexchange.com/q/1858/5010)

* * *

Regarding 食べてて, it's a contracted form of 食べていて, which is _four_ words
according to 国語文法:

  * 食べ: [連用形](https://japanese.stackexchange.com/a/65953/5010) of the verb 食べる
  * て: 接続助詞
  * い: 連用形 of the verb いる (used as a subsidiary verb to express progressive aspect)
  * て: 接続助詞

The contracted form 食べてて no longer includes い, but it is semantically
important, and a tokenizer cannot ignore it just because it's visually
nonexistent. To address this issue, the tokenizer introduced a "utility" or
"quasi-auxiliary" verb `てる`, which is essentially a single-word shortened
version of two-word `て + いる`. With this, 食べてて can be cleanly tokenized without
losing or adding something:

  * 食べ: 連用形 of the verb 食べる
  * て: 連用形 of the "utility verb" てる
  * て: 接続助詞

I think similar "utility verbs" are used to tokenize other contractions like
(食べ)とく, (食べ)ちゃう and (食べ)たげる. (I tentatively called them "utility verbs", but
there may be a better name.) Note that てる, とく, ちゃう, たげる and so on belong to no
traditional word class, just as you cannot classify English contractions like
_d'you_ , _it'll_ and _aren't_.
CC BY-SA 4.0
2019-08-27T02:42:56.607
70365
2019-08-27T03:39:19.367
2019-08-27T03:39:19.367
5010
5010
69590
answer