Prompt
Synopsis
This operator can be used to send a custom prompt to an LLM like Chat-GPT4.
Description
The operator is used for submitting used-defined prompts to a remote LLM.
Input
Connection
The LLM connection which contains the LLM provider to use, together with the relevant credentials. This port is mandatory, and a valid LLM connection must be provided.
Chat
OPTIONAL. A previous conversation history with an LLM (created by the Create Chat operator) can be added here, which will be submitted to the LLM before the new prompt to pass a conversation history along. If this optional port is connected, the LLM will be given the historical context of the conversation. Note: The history is submitted as-is, no RAG or any other replacements will take place before submission.
Prompt
OPTIONAL. A preconfigured prompt (created by the Create Prompt operator) that will be used to submit the task to the LLM, together with its config. In other words, no prompt and no settings need to be configured in this operator if a prompt object is passed here. If this port is connected, the respective settings of the operator will vanish and the prompt and settings contained within the prompt object will be used instead. Advanced: It is also possible to use the output of a record output port of any of the Prompt Engineering operators, which will re-use the prompt that was used to generate the record.
RAG
OPTIONAL. These optional input ports are used to provide the LLM with reference data (the R in RAG). To use the RAG data at specific places within the prompt/system prompt, you can use special placeholders: a set of 2 curly brackets around the ref name together with the RAG port number the data is supplied at, e.g. {{RAG 1}}. Reference numbering starts at 1, referencing the first RAG port. If no placeholders are used, each RAG data is appended as a simple additional message to the LLM as-is. Note: For now, the implementation is very simplistic: The entire objects at these ports will be converted to a textual (e.g. JSON-based) representation and submitted as-is as part of the LLM query. Files are read and submitted in the same way as if you were to open a file on your computer with a text editor and using whatever is shown as part of the LLM query.
Output
Result
The data table containing the prompt result.
Record
A complete representation of the prompt that was used to query the LLM, including all input prompt messages, the response, as well as config and meta information.
Parameters
Provider
The provider of the model. Availability depends on the provided LLM connection.
Model
The model to submit the prompt to. Availability depends on the LLM provider specified in the provided connection.
Prompt
The custom prompt with optional rag references (via {{RAG x}} placeholders) that will be filled in before being sent to the LLM. Rag references are referencing the rag input data at the specified port (starting at 1), and they consist of a set of 2 curly brackets around the ref name, e.g. {{RAG 1}}.
System Prompt
An optional system prompt, with can also optionally contain rag references (via {{RAG x}} placeholders); can be left empty. The system prompt is an initialization prompt which can be sent to the LLM, and its instructions are normally prioritized over the normal prompt. It is often used to let the LLM use a certain persona or style in answering such as "You are a factual chatbot which prefers short answers."
Override Prompt Config
This parameter becomes available in case a Prompt object is provided at the prompt input port. By default, and with this override not being selected, the configuration is taken from the Prompt object. If this override is active, the configuration is taken from the parameters of this operator instead. This allows to create a prompt with one LLM, but use it with a different LLM.
Max Output Tokens
The maximum amount of output tokens that should be returned by the LLM. Note that different LLMs may have different maximum output token capabilities. A token is usually between 2-4 characters, but the exact amount depends on the model, the language, and the content.
Response Format
The response format of the LLM. Some models may not be able to deliver all available options.
Temperature
Controls the randomness used in the answers. Lower values will lead to less random answers. A temperature of 0 normally represents a fully deterministic model behavior.
Top P
Controls diversity via nucleus sampling. A value of 0.5 means that half of all likelihood-weighted options would be considered. A value of 1.0 means all options are considered.
Frequency Penalty
How much to penalize new tokens based on their frequency in the answer so far. Can be used to reduce repetitiveness.
Presence Penalty
How much to penalize new tokens based on their presence in the answer so far. Increases the model's likeliness to talk about new topics.
Use Seed
If selected, the seed can be specified to have more deterministic results.
Seed
Some LLMs (not all support this functionality!) will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result. Determinism is not guaranteed. Behavior depends on the selected model.