11689
Unlike some languages (English, French, ...), written Japanese sentences don't
have spaces between words.

I know that it is the same in Chinese for example, but the fact that the
Chinese language only use one alphabet (hanzi) makes it easier to split words,
most of the time it is 1 or 2 characters for each word.

I also know that a lot of it comes from knowing vocabulary/grammar to be able
to know where each word begins and ends, but I was wondering if there were any
tips that can be used to be able to differentiate them?
CC BY-SA 4.0
2013-04-11T05:48:21.423
11687
2019-10-29T12:54:04.527
2019-08-23T06:23:55.220
1628
3331
question
syntax parsing
How to separate words in a Japanese sentence?
Great question! This is something that I have struggled with (especially early
on... when trying to read any large section of text...) but believe me, it
does get easier with time and practice.

It may already be something you are using, but there is at least one thing
that has helped me, and that is to look for
[particles](http://en.wikipedia.org/wiki/Japanese_particles) in a given
sentence. This doesn't always help... but it _can_ provide boundaries for
where many words (or phrases) begin and end.

Other than that, with time and practice it will become more evident where
words begin and end as your vocabulary is further enriched!
CC BY-SA 3.0
2013-04-11T05:56:44.060
11688
2013-04-11T05:56:44.060
1188
11687
answer
Certainly vocabulary helps, but you can get quite far by considering the
structure of a sentence.

  * Nouns are usually written in kanji (or katakana) and are _practically always_ followed by a particle (を, が, は, から, etc.) (unless they are followed by a copula で, だ, です, etc.)
  * The stem of verbs (including adjectival verbs, or "i-adjectives") is usually written in 漢字, with the ending (the conjugated part) written in kana (called 送り仮名).
  * Adverbs are often written in kana
  * Adjectival nouns (AN) ("na-adjectives") are usually written in kanji and usually followed by な.

With that information, you can usually parse a sentence without knowing any
vocabulary (but you need to know a list of particles and knowing the possible
conjugation for verbs also helps).

For example,

> 住宅地域における本機の使用は有害な電波妨害を引き起こすことがあり、その場合ユーザーは自己負担で電波妨害の問題を解決しなければなりません。

becomes

> (N) 住宅地域  
>  (P) に  
>  (V) おける  
>  (N) 本機  
>  (P) の  
>  (N) 使用  
>  (P) は  
>  (AN) 有害な  
>  (N) 電波妨害  
>  (P) を  
>  (V) 引き起こす  
>  **(N)** こと  
>  (P) が  
>  (V) あり、  
>  **(D)** その  
>  **(N)** 場合  
>  (N) ユーザー  
>  (P) は  
>  (N) 自己負担  
>  (P) で  
>  (N) 電波妨害  
>  (P) の  
>  (N) 問題  
>  (P) を  
>  (N) 解決  
>  (V) しなければなりません。

where for the bold lines, I needed to know that

  * こと is a nominalizer, i.e. some sort of auxiliary noun, and thus is not written in 漢字
  * その is a determiner and not an adverb
  * 場合 is also an auxiliary noun and doesn't take a particle here.

But as you can see, you get quite far with the handful of rules above.
CC BY-SA 3.0
2013-04-11T05:58:02.437
11689
2013-04-22T18:34:37.283
2013-04-22T18:34:37.283
1628
1628
11687
answer
Separating words in a sentence, at least when done by computer, is called
segmentation (分かち書き) or tokenization (トークン化). When using an IME to input
Japanese, when you hit the space bar to convert the kana to kanji, the IME has
to segment whatever it is that you typed, then use a dictionary to replace the
kana with its kanji. As you've probably learned by now, the IME is not 100%
accurate. As for general tokenization rules, it depends on what you're
tokenizing and what you define as a word or token.

For example, [Kuromoji](http://www.atilika.org/), an open source morphological
parser, will segment Earthliŋ's sentence above this way:

> 住宅 | 地域 | における | 本 | 機 | の | 使用 | は | 有害 | な | 電波 | 妨害 | を | 引き起こす | こと | が
> | あり | 、 | その | 場合 | ユーザー | は | 自己 | 負担 | で | 電波 | 妨害 | の | 問題 | を | 解決 |し |
> なけれ | ば |なり | ませ | ん | 。

Even though Earthliŋ parsed the last word しなければなりません as a single token,
Kuromoji parsed it as 6 tokens. How you parse depends on what information you
want to extract. Kuromoji uses a dictionary to aid the parsing, but another
parser, [TinySegmenter](http://chasen.org/~taku/software/TinySegmenter/), is
pattern based and also does a very good job:

> 住宅 | 地域 | に | おける | 本機 | の | 使用 | は | 有害 | な | 電波 | 妨害 | を | 引き起こす | こと | が
> | あり | 、 | その | 場合 | ユーザー | は | 自己 | 負担 | で | 電波 | 妨害 | の | 問題 | を | 解決 | し
> | なけれ | ば | なり | ませ | ん | 。

As you can see, there are a few differences between Kuromoji and TinySegmenter
(both of these you can use right in the browser). Although no person can
tokenize the sentence in the manner these small programs do, it should come
naturally and unconsciously as you learn Japanese.

If short on time however, a very crude way to tokenize is to just group
characters contiguously by script (hiragana, katakana, kanji):

> 住宅地域 | における | 本機 | の | 使用 | は | 有害 | な | 電波妨害 | を | 引 | き | 起 | こすことがあり | 、
> | その | 場合 | ユーザー | は | 自己負担 | で | 電波妨害 | の | 問題 | を | 解決 | しなければなりません | 。 |

The way Earthliŋ parsed the sentence above was in fact still vocabulary
intensive. For example, in order to segment `引き起こすことが` into `引き起こす | こと | が`
(and not, say, `引き起 | こすこ | とが`) he probably first started with the fact that
こと is a nominalizer and so is a token. Next, delimiting the token こと then
yields all 3 tokens `引き起こす | こと | が`. If he didn't know beforehand that こと is
a nominalizer he would not have known that it was a token and so he would not
have been able to choose between the parsings `引き起こす | こと | が` (correct) and
`引き起 | こすこ | とが` (incorrect).

Well, I'm just guessing. Obviously I don't actually know what went on inside
the head of Earthliŋ, but I wanted to draw attention to the fact that when it
comes to segmenting, having a vocabulary (or a dictionary if you're a
computer) results in different segmentation strategies.
CC BY-SA 4.0
2013-05-06T17:01:47.470
11834
2019-10-29T12:54:04.527
2019-10-29T12:54:04.527
22352
1454
11687
answer