Related topics: Architecture Overview, Structured Outputs

​
Structured LLM

The StructuredLLM class in Starfish is designed to simplify the interaction with Large Language Models (LLMs) by providing structured outputs. It allows developers to define schemas for the expected output, making it easier to work with LLM responses in a type-safe manner. This class handles prompt rendering, LLM API calls, and response parsing, supporting both synchronous and asynchronous execution. src/starfish/llm/structured_llm.py:23-40

StructuredLLM offers flexibility by supporting different LLM providers, dynamic prompts using Jinja2 templates, and automatic parsing of responses into structured data formats such as JSON schemas or Pydantic models. This approach ensures that the data returned by the LLM is consistent and easily usable within the application. src/starfish/llm/structured_llm.py:23-40

​
Architecture and Components

The StructuredLLM class comprises several key components that work together to provide structured LLM interactions. These include prompt management, LLM proxy calls, and response parsing.

​
Prompt Management

Prompt management is handled by the PromptManager class, which is responsible for rendering dynamic prompts using Jinja2 templates. The prompt can include variables that are passed during runtime, allowing for customized and context-aware LLM interactions. src/starfish/llm/structured_llm.py:103-107

The PromptManager uses Jinja2 templates, which are rendered with the provided variables. It also handles the conversion of Python f-string-like syntax to Jinja2 syntax. src/starfish/llm/structured_llm.py:35-39, src/starfish/llm/prompt/prompt_loader.py:4

​
LLM Proxy Calls

The StructuredLLM class uses a proxy to call the LLM API. Currently, it supports LiteLLM as the default proxy, which allows for integration with various LLM providers. The call_chat_model function is used to make the API call and retrieve the raw response. src/starfish/llm/structured_llm.py:108, src/starfish/llm/proxy/litellm_adapter.py

​
Response Parsing

The response parsing component is responsible for converting the raw LLM response into a structured data format. The StructuredLLM class supports both JSON schemas and Pydantic models for defining the output structure. The JSONParser and PydanticParser classes are used to parse the response accordingly. src/starfish/llm/structured_llm.py:109-112, src/starfish/llm/parser/json_parser.py, src/starfish/llm/parser/pydantic_parser.py

​
Key Classes and Functions

Here’s the markdown definition for the StructuredLLM class:

​
StructuredLLM Class

A builder for LLM-powered functions that can be called with custom parameters. [src/starfish/llm/structured_llm.py:22-]

​
Key Features

  • Handles Jinja template rendering with dynamic parameters
  • Manages LLM API calls
  • Parses responses according to provided schema
  • Provides both async (run, __call__) and sync (run_sync) execution methods
  • Supports automatic conversion of Python f-string syntax to Jinja2 templates

​
Initialization Parameters

ParameterTypeDescription
model_namestrName of the LLM model (e.g., ‘openai/gpt-4o-mini’)
promptstrTemplate string in Jinja2 or Python f-string format
model_kwargsOptional[Dict[str, Any]]Additional arguments to pass to the LLM
output_schemaOptional[Union[List[Dict[str, Any]], Dict[str, Any], type]]Schema for response parsing (JSON or Pydantic model)
prompt_templateOptional[str]Name of partial prompt template to wrap around the main prompt
strict_parsingboolWhether to raise errors on parsing failures (default: False)
type_checkboolWhether to check field types against schema (default: False)

​
Methods

MethodDescription
__call__Async convenience wrapper for run()
_prepare_prompt_inputsPrepares prompt inputs, including schema instructions
render_promptRenders the prompt template with provided parameters
render_prompt_printableReturns a printable version of the rendered prompt
runMain async method to execute the LLM with provided parameters
run_syncSynchronous version of run() (requires nest_asyncio in Jupyter)

​
Example Usage

llm = StructuredLLM(
    model_name="openai/gpt-4",
    prompt="Translate this to French: {text}",
    output_schema={"translation": str}
)

# Async usage
response = await llm(text="Hello world")

# Sync usage
response = llm.run_sync(text="Hello world")

​
Notes

  • The prompt parameter accepts both Jinja2 templates and Python f-string-like syntax
  • Single braces {variable} are automatically converted to Jinja2 syntax {{ variable }}
  • Use process_list_input=True in run methods to process list arguments
                                                                                                                 |
Here's the markdown definition for the `LLMResponse` class based on the code:

### `LLMResponse` Class

The `LLMResponse` class is a generic container for LLM responses, storing both the raw response and parsed data. [src/starfish/llm/structured_llm.py:14-20]()

#### Fields

| Field       | Type        | Description                                                                 |
|-------------|-------------|-----------------------------------------------------------------------------|
| `raw`       | `Any`       | The raw response from the LLM API                                           |
| `data`      | `Optional[T]` | The parsed data according to the output schema (if provided)               |

#### Methods

| Method        | Description                                                                 |
|---------------|-----------------------------------------------------------------------------|
| `__init__`    | Initializes the response with raw data and optional parsed data             |
| `__repr__`    | Returns a string representation showing the types of raw and parsed data    |

#### Example Usage

```python
response = LLMResponse(raw_response="Some raw text", parsed_data={"key": "value"})
print(response)  # LLMResponse(raw=<class 'str'>, data=<class 'dict'>)

​
Type Parameters

ParameterDescription
TThe type of the parsed data (e.g., dict, Pydantic model, or custom type)

This definition captures the essential aspects of the `LLMResponse` class, including its fields, methods, and type parameters, along with an example usage.