Skip to main content

Process Documents

Synopsis​

Generates word vectors from a text object.

Description​

This operator uses one single TextObject as input for generating a term vector. The resulting exampleset will hence consist of only one single example. This makes this operator especially useful for applying a model on one single text. But since the SingleTextInputOperator even provides a parameter for specifying the text, this one is more appropriate if used by a program, where a TextObject might simply be constructed and passed to the process.

Input​

word list​

The word list port.

documents​

The documents port.

Output​

example set​

The example set port.

word list​

The word list port.

Parameters​

Create word vector​

If checked, the tokens of a document will be used to generate a vector numerically representing the document.

Vector creation​

Select the schema for creating the word vector.

Add meta information​

If checked, available meta information of the text like filename, date is added as attribute.

Keep text​

If checked, the input text will be stored as a special String attribute with the role text.

Prune method​

Specifies if to frequent or to infrequent words should be ignored for word list building and how the frequencies are specified.

Prune below percent​

Ignore words that appear in less than this percentage of all documents.

Prune above percent​

Ignore words that appear in more than this percentage of all documents.

Prune below absolute​

Ignore words that appear in less than that many documents.

Prune above absolute​

Ignore words that appear in more than that many documents.

Prune below rank​

Words are ordered by frequency and words with a frequency less than the frequency of the rank given by this percentage will be pruned.

Prune above rank​

Words are ordered by frequency and words with a frequency higher than the frequency of the rank given by this percentage will be pruned.

Datamanagement​

Determines, how the data is represented internally.

Parallelize vector creation​

Determines whether the execution of Vector Creation should be parallelized.