tokenization-tool

Tokenization Tool

Tokenization is the process of breaking up a string into tokens which usually correspond to words. This is a common task in natural language processing (NLP).

Demo

Text

The text to tokenize.

Tokenization result

[
  "Switzerland",
  ",",
  "officially",
  "the",
  "Swiss",
  "Confederation",
  ",",
  "is",
  "a",
  "landlocked",
  "country",
  "located",
  "at",
  "the",
  "confluence",
  "of",
  "Western",
  ",",
  "Central",
  "and",
  "Southern",
  "Europe",
  ".",
  "It",
  "is",
  "bordered",
  "by",
  "Italy",
  "to",
  "the",
  "south",
  ",",
  "France",
  "to",
  "the",
  "west",
  ",",
  "Germany",
  "to",
  "the",
  "north",
  "and",
  "Austria",
  "and",
  "Liechtenstein",
  "to",
  "the",
  "east",
  "."
]1
[
2
  "Switzerland",
3
  ",",
4
  "officially",
5
  "the",
6
  "Swiss",
7
  "Confederation",
8
  ",",
9
  "is",
10
  "a",
11
  "landlocked",
12
  "country",
13
  "located",
14
  "at",
15
  "the",
16
  "confluence",
17
  "of",
18
  "Western",
19
  ",",
20
  "Central",
21
  "and",
22
  "Southern",
23
  "Europe",
24
  ".",
25
  "It",
26
  "is",
27
  "bordered",
28
  "by",
29
  "Italy",
30
  "to",
31
  "the",
32
  "south",
33
  ",",
34
  "France",
35
  "to",
36
  "the",
37
  "west",
38
  ",",
39
  "Germany",
40
  "to",
41
  "the",
42
  "north",
43
  "and",
44
  "Austria",
45
  "and",
46
  "Liechtenstein",
47
  "to",
48
  "the",
49
  "east",
50
  "."
51
]

The produced tokens.

Made by Anton Vasetenkov.

If you want to say hi, you can reach me on LinkedIn or via email. If you really-really like my work, you can support me by buying me a coffee.