Skip to main content

Stem Tokens using ExampleSet

Synopsis​

Replaces terms by pattern matching rules. This operator uses an ExampleSet to stem a list of words inside a "Process Documents" operator.

Description​

This operator can be used in your "Process Documents" operator and allows to provide a custom list of tokens to be filtered out. It is like the Stem (Dictionary) operator, except the input here is an ExampleSet rather than a file.

It reduces terms to a base form using an external ExampleSet with replacement rules. The ExampleSet must contain a rule per line: targetExpression:pattern1 pattern2 ... where targetExpression is the term to which the input terms are reduced, if it matches any of the patterns. patternX is a simple string or a regular expression. A simple example would be a mapping like: weekday : .*day Please keep in mind, that very short words are filtered out in the default setting of the TextInput operators.

Input​

doc​

The documents input port.

exa​

The ExampleSet with the tokens.

Output​

doc​

The resulting document.

Parameters​

Attribute​

The name of the attribute that should be used for stemming.