---
layout: docs
title: OpenAI API + Examples
nav_order: 1
parent: Local Server (Server Mode)
---

# 📑 Table of Contents

- **[📡 How Does the OpenAI Standard Work?](#-how-does-the-openai-standard-work)**
- **[📚 Developer Support](#-developer-support)**
- **[🚀 Quick Test: Use OpenAI SDK with FastFlowLM in Python](#-quick-test-use-openai-sdk-with-fastflowlm-in-python)**
- **[📌 Notes](#-notes)**
- **[🧪 More Examples](#-more-examples)**
  - [Example: Multi-turn Chat (Conversation History)](#-example-multi-turn-chat-conversation-history)
  - [Example: Streamed Output (Real-Time Response)](#-example-streamed-output-real-time-response)
  - [Example: Use a File as the Prompt](#-example-use-a-file-as-the-prompt)
  - [Example: Batch Requests (Multiple Prompts)](#-example-batch-requests-multiple-prompts)
  - [Example: Use Temperature, Top-p, and Presence Penalty](#-example-use-temperature-top-p-and-presence-penalty)
  - [Example: Multi-Image Input](#-example-multi-image-input)   
  - [Example: Multi-Modal Input](#️-example-multi-modal-input)   

---

# 📡 How Does the OpenAI Standard Work?

The **OpenAI API standard** defines a simple and powerful way for applications to communicate with language models — whether running in the cloud or locally.

It uses a **multi-role chat format**, which includes three types of messages:

| **Role**      | **Description** |
|---------------|-----------------|
| 🛠 **System**     | Used to give high-level instructions to the model. For example: "You are a helpful assistant," or "You are a coding tutor that only explains using examples." It sets the tone, rules, or available tools for the model. |
| 👤 **User**       | Messages from the application to the model. These often come from the end-user (e.g., a typed prompt like: “Explain black holes.”). |
| 🤖 **Assistant**  | Responses generated by the model and returned to the application. These are answers to the user prompts. |

This structure makes it easy to build multi-turn conversations with consistent behavior.

---

## 📚 Developer Support

OpenAI provides [official libraries](https://platform.openai.com/docs/libraries/python-library#install-an-official-sdk) in multiple programming languages to help developers follow the standard easily: **Python**, **JavaScript**, **.NET**, **Java**, and **Go**.

These libraries make it easy to send prompts, receive completions, and integrate with local or cloud-based OpenAI-compatible servers.

FLM follows the OpenAI API format in **Server Mode**.  

The following endpoints are actively supported and maintained:

- `v1/models`
- `v1/chat/completions`
- `v1/audio/transcriptions`
- `v1/embeddings`

---

# 🚀 Quick Test: Use OpenAI SDK with FastFlowLM in Python

You can try this instantly in any Python environment — including Jupyter Notebook. Follow the steps below by copying each block into a notebook cell.

---

## ✅ Step 0: Start FastFlowLM in Server Mode

Open PowerShell or terminal and launch the model server:

```
flm serve llama3.2:1b
```

> 🧠 This loads the model and starts the FastFlowLM OpenAI-compatible API at `http://127.0.0.1:52625/v1`.

---

## ✅ Step 1: Install the OpenAI Python SDK

```python
pip install --upgrade openai
```

---

## ✅ Step 2: Send a Chat Request to FastFlowLM

```python
# Quick Start
from openai import OpenAI

# Connect to local FastFlowLM server
client = OpenAI(
    base_url="http://127.0.0.1:52625/v1",  # FastFlowLM's local API endpoint
    api_key="flm"  # Dummy key (FastFlowLM doesn’t require authentication)
)

# Send a chat-style prompt using OpenAI API format
response = client.chat.completions.create(
    model="llama3.2:1b",  # Replace with any model you've launched with `flm serve`
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Why is the sky blue?"}
    ]
)

# Show the model's response
print(response.choices[0].message.content)

# cleanup
del response, client
import gc
gc.collect()
```

---

## 📌 Notes

- 🧠 You can replace `"llama3.2:1b"` with any other model available via `flm run` or `flm pull`.
- 🖥 Make sure the FastFlowLM server is running in the background (`flm serve ...`).
- 🔒 No real API key is needed — just pass `"flm"` as a placeholder.
- ⚡ FastFlowLM runs fully offline and is optimized for AMD Ryzen™ AI NPUs.

> ✅ This setup is perfect for quick offline LLM testing using standard OpenAI tooling.

---

# 🧪 More Examples

🚀 Ah — that was easy, right?  
Now let’s kick things up a notch with some awesome next-level examples!

---

## 💬 Example: Multi-turn Chat (Conversation History)

Use this pattern when you want the model to remember previous turns in the conversation:

```python
# Multi-turn conversation
from openai import OpenAI

messages = [
    {"role": "system", "content": "You are a creative writing assistant."},
    {"role": "user", "content": "Write the beginning of a fantasy story."},
]

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="flm")
response = client.chat.completions.create(model="llama3.2:1b", messages=messages)
print(response.choices[0].message.content)

# Add the assistant response and continue the conversation
messages.append({"role": "assistant", "content": response.choices[0].message.content})
messages.append({"role": "user", "content": "Continue the story with a twist."})
response = client.chat.completions.create(model="llama3.2:1b", messages=messages)

print(response.choices[0].message.content)

# cleanup
del response, client
import gc
gc.collect()
```

> ⚠️ The OpenAI API (and FastFlowLM server mode) is **stateless** — you must resend the full conversation each time. No KV cache is kept between turns.  
> 🌀 This means all previous messages are reprocessed (**prefill**), which adds latency for long chats.  
> ⚡ **FastFlowLM’s CLI mode** uses a **real KV cache**, making multi-turn responses much faster — especially with long conversations.  
> 🧠 FastFlowLM is optimized for **long sequences** with large KV caches, ideal for 32k–256k context windows.  
<!-- > 🔧 We’re working on adding **stateful KV cache** to server mode. Stay tuned!   -->

---

## 🔁 Example: Streamed Output (Real-Time Response)

Display the model’s output as it generates, token-by-token:

```python
# Streaming
from openai import OpenAI
import gc, sys

client = OpenAI(base_url="http://127.0.0.1:52625/v1/", api_key="flm")

stream = client.chat.completions.create(
    model="llama3.2:1b",
    messages=[
        {"role": "system", "content": "You are a fast, concise assistant."},
        {"role": "user", "content": "Summarize the plot of Hamlet in 3 sentences."}
    ],
    stream=True
)

print("=== STREAMING BEGIN ===")
try:
    for chunk in stream:
        content = chunk.choices[0].delta.content
        if content:
            print(content, end="", flush=True)
    print("\n=== STREAMING END ===")
except Exception as e:
    sys.stderr.write(f"\n[STREAM ERROR] {e}\n")

#cleanup
del stream, client
gc.collect()
```

---

## 📄 Example: Use a File as the Prompt

You can load a full `.txt` file as a prompt — useful for long documents or testing large context windows.

👉 [Download the sample prompt](https://github.com/FastFlowLM/FastFlowLM/blob/main/assets/alice_in_wonderland.txt)  

Download to Downloads folder. This contains over 38k token, thus may take longer to prompt. FastFlowLM supports full context length (32k–128k), making it ideal for processing long documents like this

```python
# Use a text file to prompt
from openai import OpenAI

with open("C:\\Users\\<username>\\Downloads\\alice_in_wonderland.txt", "r", encoding="utf-8") as f:
    user_prompt = f.read()

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="flm")

response = client.chat.completions.create(
    model="llama3.2:1b",
    messages=[
        {"role": "system", "content": "You are a story rewriter."},
        {"role": "user", "content": user_prompt}
    ]
)

print(response.choices[0].message.content)

# cleanup
del response, client
import gc
gc.collect()
```

---

## 📊 Example: Batch Requests (Multiple Prompts)

Loop over a list of prompts and generate answers — useful for eval or bulk testing.

```python
# Batched prompts
from openai import OpenAI

prompts = [
    "Summarize the causes of World War I.",
    "Describe how a transistor works.",
    "What are the key themes in ‘To Kill a Mockingbird’?",
]

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="flm")

for prompt in prompts:
    response = client.chat.completions.create(
        model="llama3.2:1b",
        messages=[
            {"role": "system", "content": "You are a concise academic tutor."},
            {"role": "user", "content": prompt}
        ]
    )
    print(f"\n📝 Prompt: {prompt}\n🔍 Answer: {response.choices[0].message.content}")

# cleanup
del response, client
import gc
gc.collect()
```

---

## 🧬 Example: Use Temperature, Top-p, and Presence Penalty

Control randomness and creativity — for brainstorming or open-ended tasks.

```python
# Change hyper parameters
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="flm")

response = client.chat.completions.create(
    model="llama3.2:1b",
    messages=[
        {"role": "system", "content": "You are a creative brainstorming partner."},
        {"role": "user", "content": "Give me 5 startup ideas that combine AI and education."}
    ],
    temperature=0.9,      # More randomness
    top_p=0.95,           # Nucleus sampling
    presence_penalty=0.5, # Encourage novelty
)

print(response.choices[0].message.content)

# cleanup
del response, client
import gc
gc.collect()
```

---

## 📷 Example: Multi-Image Input

Send multiple images together with a text prompt.  
The model will analyze them, compare, and respond with reasoning.

```python
import base64
from openai import OpenAI

# Paths to your local images
image_path_0 = r"C:\Users\info\OneDrive\Desktop\FLM\image_test\image0.jpg"
image_path_1 = r"C:\Users\info\OneDrive\Desktop\FLM\image_test\image1.png"

# Read and encode the images as Base64 strings (required for API input)
with open(image_path_0, "rb") as image_file:
    image_0 = base64.b64encode(image_file.read()).decode("utf-8")
with open(image_path_1, "rb") as image_file:
    image_1 = base64.b64encode(image_file.read()).decode("utf-8")

# Connect to your local FLM/OpenAI-compatible endpoint
client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="dummykey")

# Create a chat completion request with text + two images
response = client.chat.completions.create(
    model="gemma3:4b",  # Vision-capable model
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe the images. Which one you like better and why?"},
                
                {"type": "image_url", "image_url": {"url": f"data:image/jpg;base64,{image_0}"}}, # standard OpenAI-compatible API
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_1}"}}, 
            ],
        }
    ],
)

# Print the model's answer
print(response.choices[0].message.content)

# cleanup
del response, client
import gc
gc.collect()
```

---

## 🎙️ Example: Multi-Modal Input

Send an image and an audio file together with a text prompt to multimodal interface of `Gemma 4`.

```python
import base64
import gc
import sys
from openai import OpenAI

audio_path = r"C:\Users\info\OneDrive\Desktop\FLM\audio_test\audio.wav"
image_path = r"C:\Users\info\OneDrive\Desktop\FLM\image_test\image.png"


# Read the audio and image files and encode them as Base64 for API input
with open(audio_path, "rb") as audio_file:
    audio = base64.b64encode(audio_file.read()).decode("utf-8")
with open(image_path, "rb") as image_file:
    image = base64.b64encode(image_file.read()).decode("utf-8")


client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="dummykey")

stream = client.chat.completions.create(
    model="gemma4-it:e2b",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Finish two tasks: 1. Summarize the audio. 2. Describe the image."},
                {
                    "type": "input_audio",  
                    "input_audio": {"data": audio},
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{image}"},
                },
            ],
        }
    ],
    stream=True
)

print("=== STREAMING BEGIN ===")
try:
    for chunk in stream:
        content = chunk.choices[0].delta.content
        if content:
            print(content, end="", flush=True)
    print("\n=== STREAMING END ===")
except Exception as e:
    sys.stderr.write(f"\n[STREAM ERROR] {e}\n")

# cleanup
del stream, client
gc.collect()
```

---