Host your AI Model Client-Side with Transformers.js and ONNX

Note: this guide was written for version 3.8.1 of Transformers.js

What is the @huggingface/transformers library

Transformers.js is a library from HuggingFace, “designed to be functionally equivalent to Hugging Face’s transformers library”. You can install it using npm:

npm i @huggingface/transformers

How @xenova/transformers differs from @huggingface/transformers

You might have also seen @xenova/transformers in the wild - it’s an older, deprecated version of the library (“Transformers.js v2” as opposed to “Transformers.js v3”).

It lacks WebGPU support and expects an older model format (for example, tokenizer merges need to be in the array-of-strings, not the array-of-arrays-of-strings format).

Does @huggingface/transformers allow you to host your app client-side

Transformers.js allows you to host the entire application including the required WASM files and the model client-side.

However, since this library was made with external API calls in mind, it doesn’t come with an official guide for that and you’re guaranteed to run into a few quirks. This guide will save you having to go through the debugging process yourself.

If you only want a working example, skip to Build a simple client-side Transformers.js app section. It contains the shell commands and the code you need.

Note that ONNX Runtime only exposes its InferenceSession API in the Web. You can’t modify the model file (train it, quantize it, etc.) due to JavaScript’s file safety.

How to convert your PyTorch model into ONNX format

You can use the Optimum Python library to convert your models to the ONNX format.

pip install "optimum-onnx[onnxruntime]"

Let’s download the official GPT-2 model from HuggingFace and export it to ONNX at the same time:

optimum-cli export onnx --model gpt2 gpt2_onnx

You can also use optimum-cli to export local models; in this case, you will have to specify the task argument.

optimum-cli export onnx --model source_folder --task text-generation destination_folder

The task argument is also useful if you want to use your ONNX model for something different than specified on HuggingFace (for example, you may want to use feature-extraction to see the hidden states of the model).

ONNX export warnings

You will probably run into a few warnings when exporting your model to ONNX; it’s normal and expected due to differences between PyTorch and ONNX operators/runtimes.

By default, ONNX tries to ensure an absolute tolerance (atol) of epsilon equal to 1e-05 (so each logit probability is guaranteed not to differ from the original logit probability by more than 1e-05).

Important note: Transformers.js expects the model.onnx file to be within an onnx subfolder. After exporting the model, you should create an onnx subfolder within your destination_folder and move your model there.

cd destination_folder
mkdir onnx
mv model.onnx onnx/model.onnx

How to quantize your model with ONNX

The memory limit for ONNX WASM Runtime is 4GB as it uses 32-bit addressing (as written in their documentation). So, you might want to quantize your model to a more reasonable size.

Generally, you’re better off quantizing your model BEFORE exporting it to ONNX (you get more flexibility and a more mature tool). But for the record, this is how to do it with ONNX:

python
from onnxruntime.quantization import quantize_dynamic, QuantType
quantize_dynamic(model_input="gpt2_to_onnx/model.onnx", model_output="model.int8.onnx", weight_type=QuantType.QInt8)

You can also use optimum-cli quantize but it has the drawback of having to specify a config file or a specific architecture to optimize for (which in theory might cause issues in WASM).

How to configure Transformers.js and its ONNX backend

To use local models, you have to enable the allowLocalModels option. If you’re only using local models, you might want to disable allowRemoteModels to prevent remote download as a fallback option.

import { env } from '@huggingface/transformers'
env.allowRemoteModels = false;
env.allowLocalModels = true;

You can also set the localModelPath variable (“/models/” by default).

env.localModelPath = "/not_models/"

Put your model folder in your public folder so your model files are accessible at:

site.com/your_path/model_name/file.json
site.com/your_path/model_name/onnx/model.onnx

By default, ONNX backend gets its WASM from HuggingFace’s Content Delivery Network. If you want to give it to the client yourself, you can download the precompiled WASM files here: https://cdn.jsdelivr.net/npm/@huggingface/transformers@3.8.1/dist/ - you need the .wasm and the .mjs file.

After downloading the WASM files, you have to set ONNX’s wasmPaths:

env.backends.onnx.wasm.wasmPaths = "/wasm/"

or

env.backends.onnx.wasm.wasmPaths = {
   "mjs": '/wasm/ort-wasm-simd-threaded.jsep.mjs',
   "wasm": '/wasm/ort-wasm-simd-threaded.jsep.wasm'
}

How to solve the Vite cannot import from public folder error

Vite won’t let ONNX import files as modules from the public folder - in this case just move wasm into your source folder and let it import the files properly.

import mjs from './wasm/ort-wasm-simd-threaded.jsep.mjs?url'
import wasm from './wasm/ort-wasm-simd-threaded.jsep.wasm?url'

env.backends.onnx.wasm.wasmPaths = {
	"mjs": mjs,
	"wasm": wasm
}

How env.localModelPath works in Transformers.js

If you provide a name in the pipeline that is not a valid HuggingFace model id (like a path starting with / or .), the variable localModelPath is ignored. So in this example:

env.localModelPath = "/not_models/"

const tokenizer = await AutoTokenizer.from_pretrained("gpt2")
const input_tokens = await tokenizer("strawberry").input_ids.data
console.log(`How many tokens are there in "strawberry"?\n${input_tokens.length} tokens! Here's a list: ${input_tokens.join(", ")}!`)

const generator = await pipeline("text-generation", "/models/gpt2", {dtype: "fp32", device: "webgpu"});
const result = await generator("Once upon a time,", {do_sample: true, temperature: 2.0});
console.log(result[0].generated_text)

The tokenizer will look for the file in /not_models/gpt2 but the generator will look for the file in /models/gpt2 as opposed to /not_models/models/gpt2.

Here is the library snippet responsible:

>>> transformers/src/utils/hub.js:110

const validModelId = isValidHfModelId(path_or_repo_id);

const localPath = validModelId ? pathJoin(env.localModelPath, requestURL) : requestURL;

Build a simple client-side Transformers.js app

Step 1. Install the Transformers.js library

npm init
npm i @huggingface/transformers

Step 2. Download an AI model, export it to ONNX format and structure the folder the right way

pip install "optimum-onnx[onnxruntime]"
mkdir -p models
optimum-cli export onnx --model gpt2 models/gpt2
mkdir models/gpt2/onnx
mv models/gpt2/model.onnx models/gpt2/onnx

Step 3. Get our WASM files

mkdir wasm
curl https://cdn.jsdelivr.net/npm/@huggingface/transformers@3.8.1/dist/ort-wasm-simd-threaded.jsep.mjs -o wasm/ort-wasm-simd-threaded.jsep.mjs
curl https://cdn.jsdelivr.net/npm/@huggingface/transformers@3.8.1/dist/ort-wasm-simd-threaded.jsep.wasm -o wasm/ort-wasm-simd-threaded.jsep.wasm

Step 4. Create index.html…

>>> index.html

<!doctype html>
<html>
<head>
    <title>Ask my model anything!</title>
    <style>
    body {
        max-width: 600px;
        margin: 40px auto;
    }
    .box {
        border: 1px solid #7a6272;
        padding: 10px;
        margin: 20px;
        display: flex;
    }
    textarea {
        flex: 1;
        resize: none;
    }
</style>
</head>
<body>
    <div class="box">
        <p>Chat history:</p>
        <div id="chat-history"></div>
    </div>
    <div class="box">
        <textarea id="user-input"></textarea>
        <button id="submit-button">Let's chat!</button>
    </div>
    <script type="module" src="/src/main.js"></script>
</body>
</html>

Step 5. and create src/main.js

>>> src/main.js

import { pipeline, env } from "../node_modules/@huggingface/transformers/dist/transformers.min.js";

const chat_history = document.getElementById("chat-history");
const input = document.getElementById("user-input");
const button = document.getElementById("submit-button");

button.disabled = true;

env.allowLocalModels = true;
env.backends.onnx.wasm.wasmPaths = "/wasm/"

const generator = await pipeline("text-generation", "gpt2", { dtype: "fp32" });

button.disabled = false;

async function talkToAI() {
	const question = input.value;
	if (!question) return;
	input.value = '';
	button.disabled = true
	
	const user_item = document.createElement("div");
	const user_tag = document.createElement("b");
	const user_message = document.createElement("span");
	user_tag.innerText = "User: ";
	user_message.innerText = question;
	user_item.appendChild(user_tag);
	user_item.appendChild(user_message);
	chat_history.appendChild(user_item);
	
	const result = await generator(`User: ${question}\nBot: `, { do_sample: true, return_full_text: false });
	const result_text = result[0].generated_text.split("\n")[0];
	
	const bot_item = document.createElement("div");
	const bot_tag = document.createElement("b");
	const bot_message = document.createElement("span");
	bot_tag.innerText = "Bot: ";
	bot_message.innerText = result_text;
	bot_item.appendChild(bot_tag);
	bot_item.appendChild(bot_message);
	chat_history.appendChild(bot_item);
	
	button.disabled = false;
}

button.addEventListener("click", talkToAI);

Step 6. Finally, serve your website with a simple http.server

python -m http.server 8000

Your website should work whether you’re online or offline!

You can remove legacy files like merges.txt, vocab.json and special_tokens_map.json from the model folder. Your file structure should look like this:

├── node_modules
├── index.html
├── models
│   └── gpt2
│       ├── config.json
│       ├── generation_config.json
│       ├── onnx
│       │   └── model.onnx
│       ├── tokenizer.json
│       ├── tokenizer_config.json
├── package-lock.json
├── package.json
├── src
│   └── main.js
└── wasm
    ├── ort-wasm-simd-threaded.jsep.mjs
    └── ort-wasm-simd-threaded.jsep.wasm