Skip to main content

Streaming Optimization

Synopsis​

This Operator allows to design streaming analytic processes in it's subprocess and only provide a collection of connections to streaming platforms on which the workflow can be deployed - the INFORE optimizer is then used to decide on an optimized placement of the operators in the workflow at the different platforms.

Description​

Upon execution of the Streaming Optimization operator the designed workflow (in the subprocess of the operator) is provided, along with the other required configurations, to the INFORE optimizer. The optimizer then performs an optimization of the workflow based on the provided information.

The received optimized workflow is then used to update the inner subprocess (create the corresponding Streaming Nest operators, place the operators inside the Nests according to the optimization, restore splitted connections). After the inner subprocess is updated, it is executed, which causes the deployment of the streaming workflows on the different platforms.

The connection information to the Optimizer Service has to be provided. The operator also provides the option to write the different configurations (workflow, network, dictionary, request) and the response of the optimizer to disk. If this option is selected, the input for the update of the inner subprocess (which is normally the response of the INFORE optimizer) is also read from disk and the updated workflow is written to disk as well. All file locations can be controlled by corresponding parameters. This allows for easy manipulation of the interaction with the optimizer, to test things.

The operator also provides the option to perform a dummy optimization without an actual execution of the optimized workflow.

Input​

kafka connection​

The connection to the Kafka cluster which shall be used to connect streaming jobs deployed on different platforms.

streaming site​

This port is a port extender, which means if a port is connected a new streaming site port is created.

Each streaming site port expects a collection of connections to streaming platforms available at the corresponding computation site. The name of this streaming site can be provided at the streaming sites names parameter. The INFORE optimizer expects all streaming platforms of one streaming site to be physically near each other and take this into consideration when the inner subprocess is updated.

The optimized workflow will be deployed (depending on the result of the optimization) on a combination of all available streaming platforms across all provided streaming sites.

After the optimization is performed, for every streaming platform in all streaming sites, an inner port is created in the subprocess of the Streaming Optimization operator and is connected to the corresponding Streaming Nest operator.

Parameters​

Restore original process​

This button allows to restore the original logical process, after an optimization of the inner subprocess was already performed.

Write and read from json​

If selected the configuration json files send to the INFORE optimizer are written to disk. Also the response of the optimizer which is used to update the inner subprocess is read from disk. This allows for easy testing of the interaction with the INFORE optimizer

Network (write)​

Path which is used to write the Network JSON configuration file to disk, which is used by the INFORE optimizer.

Only enabled if write and read from json is selected.

Dictionary (write)​

Path which is used to write the Dictionary JSON configuration file to disk, which is used by the INFORE optimizer.

Only enabled if write and read from json is selected.

Request (write)​

Path which is used to write the Request JSON configuration file to disk, which is used by the INFORE optimizer.

Only enabled if write and read from json is selected.

Workflow (write)​

Path which is used to write the Workflow JSON configuration file to disk, which is used by the INFORE optimizer.

Only enabled if write and read from json is selected.

Optimizer response (write)​

Path which is used to write the response JSON string of the INFORE optimizer to disk.

Only enabled if write and read from json is selected.

Optimizer response (read)​

File containing the JSON response of the INFORE optimizer. This is used to update the inner subprocess instead of the actual response of the optimizer

Only enabled if write and read from json is selected.

Optimized workflow (write)​

Path which is used to write the JSON representation of the updated inner subprocess to disk.

Only enabled if write and read from json is selected.

Host​

The host of the optimizer.

Port​

The host of the optimizer.

User​

The user name used to log into the optimizer.

Password​

The password used to log into the optimizer.

Optimizer version​

The version of the optimizer to be used.

  • 1.0: Version 1.0 of the INFORE optimizer.

Optimizer algorithm​

The algorithm of the optimizer to be used.

  • a*: A_Star algorithm, fast but with large memory footprint.
  • exhaustive: The exhaustive algorithm, slow but optimal.
  • greedy: The greedy algorithm, fast but approximate solutions.
  • automatic: Automatic decision (recommended).

Network name​

The unique name of the dictionary used by the optimizer.

Dictionary name​

The unique name of the dictionary used by the optimizer.

Workflow name​

The unique name of the workflow used by the optimizer.

Streaming sites names​

Names of the available streaming sites.

Dummy optimization​

If selected, the operator only performs the optimization, without actually deploying the optimized workflow.