# Run pytorch inference on nanovm

**URL:** <https://forums.nanovms.com/t/run-pytorch-inference-on-nanovm/83>\
**Category:** Uncategorized\
**Created:** [July 17, 2025, 3:02pm UTC](https://forums.nanovms.com/t/run-pytorch-inference-on-nanovm/83 "2025-07-17T15:02:46Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![sara](https://avatars.discourse-cdn.com/v4/letter/s/a5b964/32.png) [@sara](https://forums.nanovms.com/u/sara)\
**Post date:** [July 17, 2025, 3:02pm UTC](https://forums.nanovms.com/t/run-pytorch-inference-on-nanovm/83/1 "2025-07-17T15:02:46Z")

</div>

I’m trying to run pytorch inference on nanovm (that requires another python packages such transformers..),

I’m trying to run from my python environment- venv, with the needed packages,

my config file is:

```auto
{
  "Env": {
    "HOME": "/",
    "PYTHONDONTWRITEBYTECODE": "1",
    "USER": "root",
    "TRANSFORMERS_CACHE": "/model-cache",
    "HF_HOME": "/model-cache",
    "TOKENIZERS_PARALLELISM": "false",
    "OMP_NUM_THREADS": "1",
    "MKL_NUM_THREADS": "1",
    "PYTORCH_NO_CUDA_MEMORY_CACHING": "1",
    "SSL_CERT_FILE": "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem",
    "REQUESTS_CA_BUNDLE": "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem"
  },
  "MapDirs": {
    "./venv_3_8/*": "/venv_3_8"
    },
  "Args": [
    "/venv_3_8/bin/python3.8",
    "./llama3_inference.py"
    ]
}

```

The command is:  
`ops pkg load eyberg/pytorch:2.1.1 -c config.json --nanos-version=0.1.52 `  
I have venv\_3\_8 directory in my app directory which I run the ops command from,

But I get an error:  
SyntaxError: Non-UTF-8 code starting with ‘\xe2’ in file /venv\_3\_8/bin/python3.8 on line 2, but no encoding declared; see [PEP 263 – Defining Python Source Code Encodings | peps.python.org](https://python.org/dev/peps/pep-0263/) for details

It seems that the nanoVM does not recognize the python exec file…

how can I solve it and run correctly from my venv with the needed packages?

(the python inference works well in the local environment (without nanoVM))

thanks

---

<div class="post-metadata">

**Author:** ![eyberg](https://avatars.discourse-cdn.com/v4/letter/e/ea666f/32.png) [@eyberg](https://forums.nanovms.com/u/eyberg)\
**Post date:** [July 17, 2025, 3:23pm UTC](https://forums.nanovms.com/t/run-pytorch-inference-on-nanovm/83/2 "2025-07-17T15:23:00Z")

</div>

Hi Sara - thanks for moving this over.

As I mentioned in that other thread the ‘\xe2’ error you are getting is a multibyte issue in on of your source files that it’s not recognizing. Sometimes this can happen because of copy/paste.

Is it possible to see a copy of your llama3\_inference.py or something reproducible - perhaps a github repo with the code showing this?

---

<div class="post-metadata">

**Author:** ![sara](https://avatars.discourse-cdn.com/v4/letter/s/a5b964/32.png) [@sara](https://forums.nanovms.com/u/sara)\
**Post date:** [July 20, 2025, 7:21am UTC](https://forums.nanovms.com/t/run-pytorch-inference-on-nanovm/83/3 "2025-07-20T07:21:22Z")

</div>

```auto
# llama3_inference.py

import sys

import torch

import warnings

from transformers import AutoTokenizer, AutoModelForCausalLM

# Suppress warnings for cleaner output
warnings.filterwarnings("ignore")

def main():
    print("Starting TinyLlama inference on NanoVM...")
    
    # Model configuration
    model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
    
    try:
        print("Loading tokenizer...")
        tokenizer = AutoTokenizer.from_pretrained(model_name)
        
        print("Loading model...")
        model = AutoModelForCausalLM.from_pretrained(
            model_name,
            torch_dtype=torch.float32, # Use float32 for CPU compatibility
            device_map="cpu", # Force CPU usage
            low_cpu_mem_usage=True, # Optimize memory usage
            trust_remote_code=True
        )
        
        print("Model loaded successfully!")
        
        # Test prompt
        prompt = "<|system|>\nYou are a helpful assistant.\n<|user|>\nwhat is the capital of England?\n<|assistant|>\n"
        
        print(f"Input prompt: {prompt}")
        print("Running inference...")
        
        # Tokenize
        inputs = tokenizer(prompt, return_tensors="pt")
        
        # Generate
        with torch.no_grad():
            outputs = model.generate(
                **inputs,
                max_new_tokens=50,
                temperature=0.7,
                do_sample=True,
                pad_token_id=tokenizer.eos_token_id,
                eos_token_id=tokenizer.eos_token_id
            )
        
        # Decode output
        response = tokenizer.decode(outputs[0], skip_special_tokens=True)
        print("=" * 60)
        print("RESPONSE:")
        print(response)
        print("=" * 60)
        
    except Exception as e:
        print(f"Error: {e}")
        sys.exit(1)
    
    print("Inference completed successfully!")

if __name__ == " __main__":
    main()

```

---

<div class="post-metadata">

**Author:** ![sara](https://avatars.discourse-cdn.com/v4/letter/s/a5b964/32.png) [@sara](https://forums.nanovms.com/u/sara)\
**Post date:** [July 20, 2025, 11:47am UTC](https://forums.nanovms.com/t/run-pytorch-inference-on-nanovm/83/4 "2025-07-20T11:47:48Z")

</div>

This error is reproduced also with simple “hello world” printing script.

---

<div class="post-metadata">

**Author:** ![eyberg](https://avatars.discourse-cdn.com/v4/letter/e/ea666f/32.png) [@eyberg](https://forums.nanovms.com/u/eyberg)\
**Post date:** [July 20, 2025, 2:53pm UTC](https://forums.nanovms.com/t/run-pytorch-inference-on-nanovm/83/5 "2025-07-20T14:53:14Z")

</div>

If you’re reproducing this in just a simple hello world that’d be easiest to diagnose. I don’t see that in what you pasted which means it’s probably in your .venv.

What kind of locale do you have locally and is your host linux/mac/windows?

You can find the locale by running this for example:

```auto
➜ ~ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=

```

If you throw a reproducible hello world into a github repo we could probably find what you are running into. You can also try running something like this [https://stackoverflow.com/questions/21639275/python-syntaxerror-non-ascii-character-xe2-in-file](https://stackoverflow.com/questions/21639275/python-syntaxerror-non-ascii-character-xe2-in-file) to hunt down what file is causing this.

---

<div class="post-metadata">

**Author:** ![sara](https://avatars.discourse-cdn.com/v4/letter/s/a5b964/32.png) [@sara](https://forums.nanovms.com/u/sara)\
**Post date:** [July 21, 2025, 5:49am UTC](https://forums.nanovms.com/t/run-pytorch-inference-on-nanovm/83/6 "2025-07-21T05:49:24Z")

</div>

I get the error only in the nanoVM, when I run the python script on my host, it works well.  
In addition, the file that the error massage mentioned is the python exec file (/venv\_3\_8/bin/python3.8), not the script file. (the error occurs also when I try to run /venv\_3\_8/bin/python3.8 --version - without python script), so it means that the problem is not in the python script.  
I think that the nanoVM does not succeed to parse the python exec bin file correctly.

My host is linux.  
I get the error in both outputs:  
~ locale  
LANG=en\_IL  
LC\_CTYPE=“en\_IL”  
LC\_NUMERIC=“en\_IL”  
LC\_TIME=“en\_IL”  
LC\_COLLATE=“en\_IL”  
LC\_MONETARY=“en\_IL”  
LC\_MESSAGES=“en\_IL”  
LC\_PAPER=“en\_IL”  
LC\_NAME=“en\_IL”  
LC\_ADDRESS=“en\_IL”  
LC\_TELEPHONE=“en\_IL”  
LC\_MEASUREMENT=“en\_IL”  
LC\_IDENTIFICATION=“en\_IL”  
LC\_ALL=

and also when I change it to:  
~ locale  
LANG=en\_US.UTF-8  
LC\_CTYPE=“en\_US.UTF-8”  
LC\_NUMERIC=“en\_US.UTF-8”  
LC\_TIME=“en\_US.UTF-8”  
LC\_COLLATE=“en\_US.UTF-8”  
LC\_MONETARY=“en\_US.UTF-8”  
LC\_MESSAGES=“en\_US.UTF-8”  
LC\_PAPER=“en\_US.UTF-8”  
LC\_NAME=“en\_US.UTF-8”  
LC\_ADDRESS=“en\_US.UTF-8”  
LC\_TELEPHONE=“en\_US.UTF-8”  
LC\_MEASUREMENT=“en\_US.UTF-8”  
LC\_IDENTIFICATION=“en\_US.UTF-8”  
LC\_ALL=en\_US.UTF-8

Simple python script that reproduces the error when I run it on the nanoVM:  
`print("hello world")`

thanks

---

<div class="post-metadata">

**Author:** ![eyberg](https://avatars.discourse-cdn.com/v4/letter/e/ea666f/32.png) [@eyberg](https://forums.nanovms.com/u/eyberg)\
**Post date:** [July 21, 2025, 1:38pm UTC](https://forums.nanovms.com/t/run-pytorch-inference-on-nanovm/83/7 "2025-07-21T13:38:16Z")

</div>

I see - you’re passing the python interpreter as the first argument but the package itself already sets it (if you look in ~/.ops/packages/amd64/eyberg/pytorch\_2.1.1/package.manifest)

```auto
"Args": ["python3"],

```

What you can do here is simply omit the

```auto
    "/venv_3_8/bin/python3.8",

```

in your config. If you need python3.8 you’d need to replace what’s in the package as it’s using python 3.10.

---

<div class="post-metadata">

**Author:** ![sara](https://avatars.discourse-cdn.com/v4/letter/s/a5b964/32.png) [@sara](https://forums.nanovms.com/u/sara)\
**Post date:** [July 22, 2025, 6:53am UTC](https://forums.nanovms.com/t/run-pytorch-inference-on-nanovm/83/8 "2025-07-22T06:53:07Z")

</div>

Also if I run with python3.10 and in my config set:

```auto
  "Args": [
    "python3.10",
    "/llama3_inference.py"
    ]

```

I get the error:  
SyntaxError: Non-UTF-8 code starting with ‘\x80’ in file //python3 on line 2, but no encoding declared; see [PEP 263 – Defining Python Source Code Encodings | peps.python.org](https://python.org/dev/peps/pep-0263/) for details

and if I run without the args setting in the config file, (use the default python interpreter), I get the error:  
Traceback (most recent call last):  
File “//llama3\_inference.py”, line 9, in   
from transformers import AutoTokenizer, AutoModelForCausalLM  
ModuleNotFoundError: No module named ‘transformers’

I need ‘transformers’ module in my script, so that it the reason that I want to run from my venv with the needed modules.

---

<div class="post-metadata">

**Author:** ![francescolavra](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.nanovms.com/francescolavra/32/33_2.png) [@francescolavra](https://forums.nanovms.com/u/francescolavra)\
**Post date:** [July 22, 2025, 7:28am UTC](https://forums.nanovms.com/t/run-pytorch-inference-on-nanovm/83/9 "2025-07-22T07:28:21Z")

</div>

As @eyberg said, you must not insert the name of the Python interpreter in the “Args” array: if you want to run the llama3\_inference.py script, your Args should be:

```auto
  "Args": [
    "/llama3_inference.py"
    ]

```

If you want to run the Python interpreter from your venv, you cannot use the pytorch:2.1.1 package as is, because that includes its own Python interpreter. You should either create your local package which will be a modified version of the the pytorch:2.1.1 package (see [Packages | Ops](https://docs.ops.city/ops/packages#creating-a-custom-or-local-dev-packages)), or do without packages altogether, in which case you would have to re-create the entire environment including your Python interpreter and all needed modules via the “Dirs” and “Files” directives (see [Configuration | Ops](https://docs.ops.city/ops/configuration)).

---

<div class="post-metadata">

**Author:** ![sara](https://avatars.discourse-cdn.com/v4/letter/s/a5b964/32.png) [@sara](https://forums.nanovms.com/u/sara)\
**Post date:** [July 24, 2025, 11:37am UTC](https://forums.nanovms.com/t/run-pytorch-inference-on-nanovm/83/10 "2025-07-24T11:37:24Z")

</div>

At this point, since I’m not converging toward a working solution, I’ve decided to put this task on hold for now.  
If I have any updates or progress on this task in the future, I’ll make sure to share them here.

Thanks for your time and assistance.