Using VLMs requires correct model hyperparameters

For example

(vllm-env) PS C:\users\ravib\venv\Qwen> ollama pull qwen2.5vl:7b

pulling manifest

pulling a99b7f834d75: 100% ▕█████████████████████████████████████████████████████████ ▏ 6.0 GB/6.0 GB   62 MB/s      0s

verifying sha256 digest

writing manifest

success

(vllm-env) PS C:\users\ravib\venv\Qwen> ollama run qwen2.5vl:7b

 

>>> How many red cars are in the image at "C:\Users\ravib\Downloads\frames\frames\frame11.jpg"

Added image 'C:\Users\ravib\Downloads\frames\frames\frame11.jpg'

There are two red cars visible in the image. One is parked on the left side of the street, and the other is parked

on the right side of the street.

 

>>> /bye

This is the correct answer.

Now with a python script:

import requests

import json

import base64

import os

 

url = "http://localhost:8848/v1/chat/completions"

 

# 1. Path to your local image

image_path = r"C:\Users\ravib\Downloads\frames\frames\frame11.jpg"

 

# 2. Open the image file and encode it to Base64

if os.path.exists(image_path):

    with open(image_path, "rb") as image_file:

        base64_image = base64.b64encode(image_file.read()).decode('utf-8')

else:

    print(f"Error: File not found at {image_path}")

    exit()

 

# 3. Construct the payload with the Base64 data URI format

payload = {

    "model": "qwen2.5vl:7b",

    "messages": [

        {

            "role": "user",

            "content": [

                {

                    "type": "text",

                    "text": "How many red cars are in the image?"

                },

                {

                    "type": "image_url",

                    "image_url": {

                        "url": f"data:image/jpeg;base64,{base64_image}"

                    }

                }

            ]

        }

    ]

}

 

headers = {

    "Content-Type": "application/json"

}

 

response = requests.post(url, headers=headers, data=json.dumps(payload))

 

print("Status:", response.status_code)

print("Response:")

print(response.text)

 

## Response

# Status: 200

# Response:

#{"id":"chatcmpl-429","object":"chat.completion","created":1788656816,"model":"qwen2.5vl:7b","system_fingerprint":"fp_ollama","choices":[{"index":0,"message":{"role":"assistant","content":"There is one red car on the street in the image."},"finish_reason":"stop"}],"usage":{"prompt_tokens":1226,"prompt_tokens_details":{"cached_tokens":22},"completion_tokens":13,"total_tokens":1239}}

 

This is the incorrect answer.

Now, the fix that works:

import base64

from PIL import Image

import io

import ollama

 

# 1. Path to your image file

image_path = r"C:\Users\ravib\Downloads\frames\frames\frame11.jpg"

 

# 2. Open image with Pillow to ensure it passes cleanly as standard bytes

with Image.open(image_path) as img:

    if img.mode in ("RGBA", "P"):

        img = img.convert("RGB")

   

    # Optional: Keep resolution native but ensure metadata handles boundaries well

    buffer = io.BytesIO()

    img.save(buffer, format="JPEG", quality=100)

    image_bytes = buffer.getvalue()

 

# 3. Direct programmatic call to the Ollama server on your port

# (Point the client explicitly to your active instance)

client = ollama.Client(host="http://localhost:8848")

 

response = client.chat(

    model="qwen2.5vl:7b",

    messages=[

        {

            "role": "user",

            "content": "Look closely at the entire scene, including any cars parked down the sides of the street. How many red cars are visible in this image?",

            "images": [image_bytes]  # Pass raw image bytes, avoiding Base64 URI scaling issues

        }

    ],

    options={

        "temperature": 0.0,          # Eliminates random sampling path bias

        "num_predict": 100,          # Gives the model breathing room to elaborate

        "num_ctx": 8192              # Expand context window to avoid aggressive clipping

    }

)

 

# Print the final result cleanly

print("Response:")

print(response['message']['content'])

Response:

In the image, there are two red cars visible. One is parked on the left side of the street near the tall building, and the other is parked further down the street on the right side.

Latest models of Qwen recommend:

·        Thinking Mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0

·        Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0