Prompt with Data
Synopsis
This fully customizable operator can be used to submit complex prompts to an LLM running for each row of a provided data set.
Description
The operator can be used to submit complex prompts to an LLM, enriched with both the data of each row from the input data, as well as optionally rag data references for grounding the LLM and providing additional context. The prompt is applied row-wise on each row, each time replacing placeholders with the actual value from any number of columns within the input data.
Input
Connection
The LLM connection which contains the LLM provider to use, together with the relevant credentials. This port is mandatory, and a valid LLM connection must be provided.
Data
This table input port expects the data which contains the column(s) to enrich the custom prompt for the LLM, so one prompt per row can be submitted to the LLM.
Prompt
OPTIONAL. A preconfigured prompt (created by the Create Prompt operator) that will be used to submit the task to the LLM, together with its config. In other words, no prompt and no settings need to be configured in this operator if a prompt object is passed here. If this port is connected, the respective settings of the operator will vanish and the prompt and settings contained within the prompt object will be used instead. Advanced: It is also possible to use the output of a record output port of any of the Prompt Engineering operators, which will re-use the prompt that was used to generate the record.
RAG
OPTIONAL. These optional input ports are used to provide the LLM with reference data (the R in RAG). To use the RAG data at specific places within the prompt/system prompt, you can use special placeholders: a set of 2 curly brackets around the ref name together with the RAG port number the data is supplied at, e.g. {{RAG 1}}. Reference numbering starts at 1, referencing the first RAG port. If no placeholders are used, each RAG data is appended as a simple additional message to the LLM as-is. Note: For now, the implementation is very simplistic: The entire objects at these ports will be converted to a textual (e.g. JSON-based) representation and submitted as-is as part of the LLM query. Files are read and submitted in the same way as if you were to open a file on your computer with a text editor and using whatever is shown as part of the LLM query.
Output
Result
The data table containing the new target column with the custom prompt result.
Record
A complete representation of ONE of the prompts that was used to query the LLM, including all input prompt messages, the response, as well as config and meta information. Since in this operator an individual prompt is generated for each row, and the prompts are submitted in parallel, the prompt available here is not necessarily the prompt for the first row, but it is the prompt for one of the first rows.
Parameters
Prompt
The custom prompt with optional row-wise replaced input, RAG, and expected result references, as well as global RAG references (via {{RAG x}} placeholders) that will be filled in before being sent to the LLM. Column references are referencing the name of a column in the input data, and they consist of a set of 2 square brackets around the column name, e.g. [[Column 1]]. Column RAG references are referencing the name of a column in the input data for context references, and they consist of a set of a square bracket followed by a curly bracket around the column name, e.g. [{Column 1}]. Column expected result references are referencing the name of a column in the input data for expected result references, and they consist of a set of a square bracket followed by a normal bracket around the column name, e.g. [(Column 1)]. In the end, all the three different column references above are simply replaced with the respective column values for each row, the distinction between them is to support Prompt Studio and allows for discerning what part of the prompt placeholders served which purpose. Last but not least, there is also a global reference, which is taken from the operator input ports in full, and not row-wise. These global RAG references are referencing the rag input data at the specified port (starting at 1), and they consist of a set of 2 curly brackets around the ref name, e.g. {{RAG 1}}.
Input Refs
The column input references that occur in the prompt via [[COL_REF]] and will get replaced by the actual data within each row of the input data before the prompt is submitted.
RAG Refs
The column RAG references that occur in the prompt via [{COL_REF}] and will get replaced by the actual data within each row of the input data before the prompt is submitted.
Expected Refs
The column expected result references that occur in the prompt via [(COL_REF)] and will get replaced by the actual data within each row of the input data before the prompt is submitted.
System Prompt
An optional system prompt, with can also optionally contain global RAG references (via {{RAG x}} placeholders); can be left empty. The system prompt is an initialization prompt which can be sent to the LLM, and its instructions are normally prioritized over the normal prompt. It is often used to let the LLM use a certain persona or style in answering such as "You are a factual chatbot which prefers short answers."
Target Column
The name of the new column which should contain the prompt result.
Provider
The provider of the model. Availability depends on the provided LLM connection.
Model
The model to submit the prompt to. Availability depends on the LLM provider specified in the provided connection.
Override Prompt Config
This parameter becomes available in case a Prompt object is provided at the prompt input port. By default, and with this override not being selected, the configuration is taken from the Prompt object. If this override is active, the configuration is taken from the parameters of this operator instead. This allows to create a prompt with one LLM, but use it with a different LLM.
Max Output Tokens
The maximum amount of output tokens that should be returned by the LLM. Note that different LLMs may have different maximum output token capabilities. A token is usually between 2-4 characters, but the exact amount depends on the model, the language, and the content.
Response Format
The response format of the LLM. Some models may not be able to deliver all available options.
Temperature
Controls the randomness used in the answers. Lower values will lead to less random answers. A temperature of 0 normally represents a fully deterministic model behavior.
Top P
Controls diversity via nucleus sampling. A value of 0.5 means that half of all likelihood-weighted options would be considered. A value of 1.0 means all options are considered.
Frequency Penalty
How much to penalize new tokens based on their frequency in the answer so far. Can be used to reduce repetitiveness.
Presence Penalty
How much to penalize new tokens based on their presence in the answer so far. Increases the model's likeliness to talk about new topics.
Use Seed
If selected, the seed can be specified to have more deterministic results.
Seed
Some LLMs (not all support this functionality!) will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result. Determinism is not guaranteed. Behavior depends on the selected model.
Enable Parallel Execution
This parameter enables the parallel execution of this operator and will cause multiple concurrent queries to the LLM. Please disable the parallel execution if this is not desirable.