# Run context binaries (.bin/.dlc)

Some models from [AI Hub](https://qc-ai-test.gitbook.io/qc-ai-test-docs/running-building-ai-models/ai-hub) (and from internal Qualcomm releases) are released as context binaries (`.bin` files). Context binaries contain the model, plus hardware optimizations; and can be ran with Qualcomm tools that directly use the Qualcomm® AI Runtime SDK. Examples of this are [Genie](https://qc-ai-test.gitbook.io/qc-ai-test-docs/running-building-ai-models/genie) (to run LLMs) and [VoiceAI ASR](https://qc-ai-test.gitbook.io/qc-ai-test-docs/running-building-ai-models/whisper) (to run voice transcription); but you can also run context binaries directly from Python using QAI AppBuilder.

{% hint style="info" %}
**Not portable:** Context binaries are not portable. They are tied to both the AI Engine Direct SDK version and your hardware target.
{% endhint %}

## Finding supported models

Models in context binary format can be found in a few places:

* [Qualcomm AI Hub](https://aihub.qualcomm.com/models) (note that these come in `.dlc` format - you'll need to convert them to `.bin` files, see below):
  1. Under 'Chipset', select:
     * RB3 Gen 2 Vision Kit: 'Qualcomm QCS6490 (Proxy)'
     * RUBIK Pi 3: 'Qualcomm QCS6490 (Proxy)'
     * IQ-9075 EVK: 'Qualcomm QCS9075 (Proxy)'
  2. Under 'Runtime', select "Qualcomm® AI Runtime".
* [Aplux model zoo](https://aiot.aidlux.com/en/models):
  1. Under 'Chipset', select:
     * RB3 Gen 2 Vision Kit: 'Qualcomm QCS6490'
     * RUBIK Pi 3: 'Qualcomm QCS6490'
     * IQ-9075 EVK: 'Qualcomm QCS9075'

Note that the NPU only supports quantized models. Floating point models (or layers) will be automatically moved back to the CPU.

## From .dlc -> .bin

If your model comes in `.dlc` format (a portable serialized format); you'll first need to convert them to context binaries (`.bin` files). You convert these files using `qnn-context-binary-generator` (part of the AI Runtime SDK). Open the terminal on your development board, or an ssh session to your development board, and:

1. Install the [AI Runtime SDK - Community Edition](https://softwarecenter.qualcomm.com/catalog/item/Qualcomm_AI_Runtime_Community):

   ```bash
   # Install the SDK
   wget -qO- https://cdn.edgeimpulse.com/qc-ai-docs/device-setup/install_ai_runtime_sdk.sh | bash

   # Use the SDK in your current session
   source ~/.bash_profile
   ```
2. Convert the `.dlc` file into a `.bin` file:

   ```bash
   qnn-context-binary-generator \
       --dlc_path=./inception_v3-inception-v3-w8a8.dlc \
       --output_dir=output/ \
       --backend=/usr/lib/libQnnHtp.so \
       --model=/usr/lib/libQnnModelDlc.so \
       --binary_file=inception_v3-inception-v3-w8a8.bin
   ```

You should now have `output/inception_v3-inception-v3-w8a8.bin`. Note that this file is not portable. It's tied to both the AI Engine Direct SDK version and your hardware target.

### Troubleshooting

If conversion fails (e.g. with `Failed to create dlc handle with code 1002 for dlc file`), there might be a discrepancy between the QNN version that created the `.dlc` file; and the QNN version on your development board. Run:

```bash
# Find QNN version for DLC file
strings ./inception_v3-inception-v3-w8a8.dlc | egrep -i 'qnn|qairt|version' | head
#         "tool": "qairt-converter",
#         "converterVersion": "2.37.0.250724175447_124859",

# Find QNN version on your development board
strings /usr/lib/libQnnHtp.so | grep AISW_VERSION
# AISW_VERSION: 2.35.0
```

Here this file was created by QAIRT 2.37.0; but your development board runs 2.35.0. Ask the person that gave you the `.dlc` file for a version tied to QNN 2.35.0 instead.

## Example: Inception-v3 (Python)

Here's how you can run an image classification model (downloaded from [Aplux model zoo](https://aiot.aidlux.com/en/models/detail/9?name=inception\&precisionShow=1\&soc=2)) on the NPU using QAI AppBuilder. Open the terminal on your development board, or an ssh session to your development board, and:

1. Build the AppBuilder wheel with QNN bindings:

   ```bash
   # Build dependency
   sudo apt update && sudo apt install -y yq

   # Clone the repository
   git clone https://github.com/quic/ai-engine-direct-helper
   cd ai-engine-direct-helper
   git submodule update --init --recursive
   git checkout v2.38.0

   # Create a new venv
   python3.12 -m venv .venv
   source .venv/bin/activate

   # Build the wheel
   pip3 install setuptools
   python setup.py bdist_wheel

   # Deactivate the venv
   deactivate

   export APPBUILDER_WHEEL=$PWD/dist/qai_appbuilder-2.38.0-cp312-cp312-linux_aarch64.whl
   ```
2. Now create a new folder for the application:

   ```bash
   mkdir -p ~/context-binary-demo
   cd ~/context-binary-demo

   # Create a new venv
   python3.12 -m venv .venv
   source .venv/bin/activate

   # Install the QAI AppBuilder plus some other dependencies
   pip3 install $APPBUILDER_WHEEL
   pip3 install numpy==2.3.3 Pillow==11.3.0
   ```
3. Create a new file `context_demo.py` and add:

   ```python
   import os, urllib.request, time, numpy as np
   from qai_appbuilder import (QNNContext, Runtime, LogLevel, ProfilingLevel, PerfProfile, QNNConfig)
   from PIL import Image

   def download_file_if_not_exists(path, url):
       if not os.path.exists(path):
           os.makedirs(os.path.dirname(path), exist_ok=True)
           print(f"Downloading {path} from {url}...")
           urllib.request.urlretrieve(url, path)
       return path

   # Path to your model/label/test image (will be download automatically)
   MODEL_PATH = download_file_if_not_exists('models/inception_v3_w8a8.qcs6490.qnn216.ctx.bin', 'https://cdn.edgeimpulse.com/qc-ai-docs/models/inception_v3_w8a8.qcs6490.qnn216.ctx.bin')
   LABELS_PATH = download_file_if_not_exists('models/inception_v3_labels.txt', 'https://cdn.edgeimpulse.com/qc-ai-docs/models/inception_v3_labels.txt')
   IMAGE_PATH = download_file_if_not_exists('images/samoyed-square.jpg', 'https://cdn.edgeimpulse.com/qc-ai-docs/example-images/samoyed-square.jpg')

   # Parse labels
   with open(LABELS_PATH, 'r') as f:
       labels = [line for line in f.read().splitlines() if line.strip()]

   # Set up the QNN config (/usr/lib => where all QNN libraries are installed)
   QNNConfig.Config('/usr/lib', Runtime.HTP, LogLevel.WARN, ProfilingLevel.BASIC)

   # Create a new context (name, path to .bin file)
   ctx = QNNContext(os.path.basename(MODEL_PATH), MODEL_PATH)

   # Load and preprocess image, input is scaled 0..1 (f32), no need to quantize yourself
   def load_image(path, input_shape):
       # Expected input shape: [1, height, width, channels]
       _, height, width, channels = input_shape

       # expects unquantized input 0..1
       img = Image.open(path).convert("RGB").resize((width, height))
       img_np = np.array(img, dtype=np.float32)
       img_np = img_np / 255
       # add batch dim
       img_np = np.expand_dims(img_np, axis=0)
       return img_np

   # As far as I know you cannot find the input shape from the .bin file...
   # Resolution found at https://aiot.aidlux.com/en/models/detail/9?name=inception&precisionShow=1&soc=2
   input_data = load_image(IMAGE_PATH, (1, 299, 299, 3))

   # Run inference once to warmup
   f_output = ctx.Inference(input_data)[0]

   # Then run 10x
   start = time.perf_counter()
   for i in range(0, 10):
       f_output = ctx.Inference(input_data)[0]
   end = time.perf_counter()

   # Image classification models in AI Hub miss a Softmax() layer at the end of the model, so add it manually
   def softmax(x, axis=-1):
       # subtract max for numerical stability
       x_max = np.max(x, axis=axis, keepdims=True)
       e_x = np.exp(x - x_max)
       return e_x / np.sum(e_x, axis=axis, keepdims=True)

   # show top-5 predictions
   scores = softmax(f_output)
   top_k = scores.argsort()[-5:][::-1]
   print("\nTop-5 predictions:")
   for i in top_k:
       print(f"Class {labels[i]}: score={scores[i]}")

   print('')
   print(f'Inference took (on average): {((end - start) * 1000) / 10:.4g}ms. per image')
   ```
4. Run the example:

   ```bash
   python3 context_demo.py

   # Top-5 predictions:
   # Class Samoyed: score=0.9563993215560913
   # Class Arctic fox: score=0.0022275811061263084
   # Class Pomeranian: score=0.001584612182341516
   # Class chow: score=0.0008018668740987778
   # Class keeshond: score=0.0006536662694998085
   #
   # Inference took (on average): 12.27ms. per image
   ```

Great! You now have ran a model in context binary format on the NPU.
