Extract Tokens

Synopsis

This operator extracts the tokens of a document.

Description

Documents can be splitted into 'parts', which are called tokens. The most common way to do this is to use the Tokenize operator. This operator takes these tokens and converts them into an ExampleSet or a collection of documents, where each document contains one token.

Input

doc

The document you want to export the tokens from.

Output

exa

Extracted tokens as ExampleSet.

col

Collection of documents. Each document has one token.

ori

Original document.