Arm64 golang hello-wold fails to run on MacBook Air M1

Hello, I’m trying to build a hello world app in golang following the docs here: MacOS - Ops
I tried with both amd64 and arm64, hope you can give me a hand.

% GOARCH=amd64 GOOS=linux go build .
% ops run --arch=amd64 hello
 100% |████████████████████████████████████████|
running local instance
booting /Users/ernesto/.ops/images/hello ...
Could not allocate dynamic translator buffer
exit status 1

% file hello
hello: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=l7mZm8m5mDX469ZwcApu/WgpDpWI6ig4aqoU29Lme/BlENpzsteh5RDhVcqzih/W0xfzKheXrbDZC6jnDHn, with debug_info, not stripped
% rm hello

% GOARCH=arm64 GOOS=linux go build .
% ops run --arch=arm64 hello
 100% |████████████████████████████████████████|
running local instance
booting /Users/ernesto/.ops/images/hello ...
Could not allocate dynamic translator buffer
exit status 1

% file hello
hello: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=HzCWcvqS6jwMw9MUrc5H/GK5jrmhcgomxNGWUwvKW/gUSyYLwrd42D5gdS1MMV/zVh604LnjO8kqmSG11sz, with debug_info, not stripped

Here is my system info:

% sysctl kern.hv_support
kern.hv_support: 1

% uname -a
Darwin M1.local 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul  5 22:22:52 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T8103 arm64

% go version
go version go1.21.3 darwin/arm64
 % brew list
==> Formulae
aarch64-elf-binutils
aarch64-elf-gcc
aarch64-linux-binutils
abseil
bison
buf
ca-certificates
cairo
capstone
cmake
coreutils
dtc
ec2-ami-tools
ent
fontconfig
freetype
gawk
gettext
giflib
glib
gmp
gnu-sed
gnutls
go
graphite2
grep
harfbuzz
icu4c
jpeg
jpeg-turbo
jsoncpp
libevent
libidn2
libmpc
libnghttp2
libpng
libslirp
libssh
libtasn1
libtiff
libunistring
libusb
libx11
libxau
libxcb
libxdmcp
libxext
libxrender
little-cms2
lz4
lzo
m4
make
mpfr
nasm
ncurses
nettle
opencl-clhpp-headers
opencl-headers
opencl-icd-loader
openjdk
openssl@3
ops
p11-kit
packer
pcre2
pixman
protobuf
protoc-gen-go-grpc
qemu
readline
snappy
socat
texinfo
tree
unbound
vde
wget
x86_64-elf-binutils
x86_64-elf-gcc
xorgproto
xz
zstd

Thanks and regards,
Ernesto Medina.

What version of qemu are you using? Can you run ‘ops profile’?

You might be running into:

% ops profile
Ops version:
Nanos version: 0.0
Qemu version: 5.2.0
OS: darwin
Arch: arm64
Virtualized: false

I installed it with brew:

% brew info ops
==> nanovms/ops/ops: stable 0.1.39, HEAD
OPS - Build and Run Nanos Unikernels
https://nanos.org/
/opt/homebrew/Cellar/ops/0.1.39_1 (3 files, 90.5MB) *
  Built from source on 2023-11-07 at 09:00:08
From: https://github.com/nanovms/homebrew-ops/blob/HEAD/Formula/ops.rb
License: Apache 2.0
==> Dependencies
Build: go ✔
Required: qemu ✔, bufbuild/buf/buf ✔, protoc-gen-go-grpc ✔
==> Options
--HEAD
	Install HEAD version

Ok - so this is definitely an old version of qemu. The latest qemu is 8.1.X .

If using brew you’ll want to update the qemu package - I believe you can do that with:

brew update && brew upgrade

When I installed ops and nanovms I already had the latest qemu version, but got a brew dependency/conflict error, so I unliked the latest one with this command: brew unlink qemu

I’m going to disable the old one and re-enable the latest one.

/opt/homebrew/Cellar/qemu/5.2.0_1/share/qemu
/opt/homebrew/Cellar/qemu/8.1.2/share/qemu

Thanks and regards,
Ernesto Medina.

I re-enabled latest version of qemu as shown here:

%  brew unlink qemu
Unlinking /opt/homebrew/Cellar/qemu/5.2.0_1... 49 symlinks removed.

% brew link qemu
Linking /opt/homebrew/Cellar/qemu/8.1.2... 50 symlinks created.

Then tried again:

% GOARCH=amd64 GOOS=linux go build .
% ops run --arch=amd64 hello
 100% |████████████████████████████████████████|
running local instance
booting /Users/ernesto/.ops/images/hello ...
en1: assigned 10.0.2.15
Hello World!
exit status 1

% rm hello

% GOARCH=arm64 GOOS=linux go build .
% ops run --arch=arm64 hello
 100% |████████████████████████████████████████|
running local instance
booting /Users/ernesto/.ops/images/hello ...
exit status 1

It’s looking better, the amd64 one printed the msg, although in the end it ends with an error.

Yeh, you can ignore that ‘exit status 1’ - there’s no error. I think that was left in there on purpose for a test but it probably shouldn’t be shown to the user regardless as it’s confusing.

2 Likes

Thank you Ian

What about the arm64 one?
I’m targeting an aws t4g (Graviton2 arm64) ec2 instance for this hello world, as of today those are the cheapest ones.

Regards,
Ernesto.

So it looks like you are on a m1 which is an arm machine and you don’t need to pass the arch flag. Can you try just a ‘ops run hello’ for that? If that doesn’t work can you paste ‘ops run hello -v --show-debug’ ?

output:

% ops run hello
 100% |████████████████████████████████████████|
running local instance
booting /Users/ernesto/.ops/images/hello ...
en1: assigned 10.0.2.15

frame 0x00ff800040ee0000 already full

Angel shutdown trap failed; shutting down with PSCI.
QEMU exit code will not reflect program exit code.

then:

% ops run hello -v --show-debug
 100% |████████████████████████████████████████|
Manifest:
	&{root:map[arguments:[hello] children:map[etc:map[passwd:/var/folders/9h/8ygmpcr54sn8b4g9fxvs8zr40000gn/T/hello_temp2646108872/passwd resolv.conf:/var/folders/9h/8ygmpcr54sn8b4g9fxvs8zr40000gn/T/hello_temp2646108872/resolv.conf ssl:map[certs:map[ca-certificates.crt:/Users/ernesto/.ops/common/ca-certificates.crt]]] hello:hello lib:map[x86_64-linux-gnu:map[libnss_dns.so.2:/Users/ernesto/.ops/common/libnss_dns.so.2]] proc:map[sys:map[kernel:map[hostname:/var/folders/9h/8ygmpcr54sn8b4g9fxvs8zr40000gn/T/hello_temp2646108872/hostname]]]] environment:map[IMAGE_NAME:hello NANOS_VERSION:0.0 OPS_VERSION: PWD:/ USER:root] program:/hello] boot:map[children:map[kernel:/Users/ernesto/.ops/0.1.48-arm/kernel.img]] targetRoot: klibHostDir:/Users/ernesto/.ops/0.1.48/klibs}
running local instance
booting /Users/ernesto/.ops/images/hello ...
qemu-system-aarch64 -machine virt -machine gic-version=2 -machine highmem=off -kernel /Users/ernesto/.ops/0.1.48-arm/kernel.img -device virtio-blk-pci,drive=hd0 -semihosting -m 1G -device virtio-rng-pci -accel hvf -cpu host,-rdtscp -no-reboot -cpu host -drive file=/Users/ernesto/.ops/images/hello,format=raw,if=none,id=hd0 -device virtio-net,netdev=n0,mac=f2:a8:7b:8e:d8:98 -netdev user,id=n0 -display none -serial stdio
en1: assigned 10.0.2.15

frame 0x00ff800040ee0000 already full

Angel shutdown trap failed; shutting down with PSCI.
QEMU exit code will not reflect program exit code.

Sometimes gives infinite input so I kill it, and this is the msg:

frame trace:

loaded klibs:
assertion enqueue_n_irqsafe(async_queue_1, &aa, sizeof(aa) / sizeof(u64)) failed at /home/eyberg/go/src/github.com/nanovms/nanos/src/kernel/kernel.h:464 (IP 0xffffffff8004c220)  in async_apply_1(); halt

fra^C trace:

loadedqemu-system-aarch64: terminating on signal 2 from pid 9693 (<unknown process>)
signal: killed

I don’t know but this might be related to your other qemu that’s on the system.

Can you show the output of which qemu-system-aarch64 that is? :

(for instance on this machine)

➜  ~ which qemu-system-aarch64
/opt/homebrew/bin/qemu-system-aarch64

➜  ~ ls -lh /opt/homebrew/bin/qemu-system-aarch64
lrwxr-xr-x@ 1 eyberg  admin    44B Aug 24 21:08 /opt/homebrew/bin/qemu-system-aarch64 -> ../Cellar/qemu/8.1.0/bin/qemu-system-aarch64
% which qemu-system-aarch64
/opt/homebrew/bin/qemu-system-aarch64
ernesto@M1 ~ % ls -lh /opt/homebrew/bin/qemu-system-aarch64
lrwxr-xr-x@ 1 ernesto  admin    44B Nov  7 17:22 /opt/homebrew/bin/qemu-system-aarch64 -> ../Cellar/qemu/8.1.2/bin/qemu-system-aarch64

I removed both qemu versions and reinstalled only the latest one, then tried again to run:

% ops run hello -v --show-debug | head -100
 100% |████████████████████████████████████████|
Manifest:
	&{root:map[arguments:[hello] children:map[etc:map[passwd:/var/folders/9h/8ygmpcr54sn8b4g9fxvs8zr40000gn/T/hello_temp73307946/passwd resolv.conf:/var/folders/9h/8ygmpcr54sn8b4g9fxvs8zr40000gn/T/hello_temp73307946/resolv.conf ssl:map[certs:map[ca-certificates.crt:/Users/ernesto/.ops/common/ca-certificates.crt]]] hello:hello lib:map[x86_64-linux-gnu:map[libnss_dns.so.2:/Users/ernesto/.ops/common/libnss_dns.so.2]] proc:map[sys:map[kernel:map[hostname:/var/folders/9h/8ygmpcr54sn8b4g9fxvs8zr40000gn/T/hello_temp73307946/hostname]]]] environment:map[IMAGE_NAME:hello NANOS_VERSION:0.0 OPS_VERSION: PWD:/ USER:root] program:/hello] boot:map[children:map[kernel:/Users/ernesto/.ops/0.1.48-arm/kernel.img]] targetRoot: klibHostDir:/Users/ernesto/.ops/0.1.48/klibs}
running local instance
booting /Users/ernesto/.ops/images/hello ...
qemu-system-aarch64 -machine virt -machine gic-version=2 -machine highmem=off -kernel /Users/ernesto/.ops/0.1.48-arm/kernel.img -device virtio-blk-pci,drive=hd0 -semihosting -m 1G -device virtio-rng-pci -accel hvf -cpu host,-rdtscp -no-reboot -cpu host -drive file=/Users/ernesto/.ops/images/hello,format=raw,if=none,id=hd0 -device virtio-net,netdev=n0,mac=9e:bb:17:35:8e:7a -netdev user,id=n0 -display none -serial stdio

no fault handler for frame
     frame: 00ff800040e30000
      type: kernel
      spsr: 00000000200003c5
       esr: 0000000096000047 data abort in el1 write
       far: 00ff000100025000
       elr: ffffffff80077d18
       far: 00ff000100025000
active_cpu: 0000000000000000
 stack top: 00ff800040e37ff0

        x0: 00ff000100025000
        x1: 0000000000000000
        x2: 0000000000000210
        x3: 0000000000000000
        x4: 00ff000100025000
        x5: 00ff000100025000
        x6: 0000000000000041
        x7: 00ff000100025210
        x8: 0000000000000042
        x9: 0000000000000042
       x10: 0000000000000000
       x11: 0000000000000026
       x12: 0000000000000001
       x13: 00ff800041003500
       x14: 00ff800041003500
       x15: 0000000000000000
       x16: ffffffff80072f30
       x17: 00ff800040e30100
       x18: 00ff800040e25000
       x19: 00ff000100025000
       x20: 00ff800040ee0000
       x21: 00ff800040e2a000
       x22: ffffffff80141000
       x23: 0000000000000001
       x24: 00ff800041a05000
       x25: 00ff800040e25000
       x26: ffffffff80141000
       x27: 00ff800040e60008
       x28: 00000000000003c0
       x29: 00ff800040e37d60
       x30: ffffffff80028524
        sp: 00ff800040e37d60
      fpcr: 0000000000008000

frame trace:

loaded klibs:

stack trace:


frame 0x00ff800040e30000 already full

frame trace:

loaded klibs:
assertion !frame_is_full(ctx->frame) failed at /home/eyberg/go/src/github.com/nanovms/nanos/src/kernel/mutex.c:109 (IP 0xffffffff80048ff4)  in mutex_lock_internal(); halt

Ok - I guess we’ll need to dive deeper into this issue then. Can you open an issue in GitHub - nanovms/nanos: A kernel designed to run one and only one application in a virtualized environment ?

Thanks,
Ernesto.

I saw Francesco fixed the issue, thank you very much.
So, do you know when a new release including this fix will be available?
I can try to build it from the source if there is no certain date as of now.

Thanks and regards,
Ernesto Medina.

Hi Ernesto,
You don’t have to wait until a new release is available, you can always run the kernel nightly build by adding the -n command line flag to your ops run command. For example: ops run hello -n. This will automatically download the latest nightly build of the kernel (which is updated every day from the master branch of the Nanos repository) and use that in the image being created.
Regards,
Francesco

Thank you Francesco/Ian for enabling me, it worked, and now I can resume my project.

% GOARCH=arm64 GOOS=linux go build .

% ops run hello -n
running local instance
booting /Users/ernesto/.ops/images/hello ...
en1: assigned 10.0.2.15
Hello, World!

% echo $?
0

% file hello
hello: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=ZLZDVoALhRUROaejAMOs/0Cvg6D_tCZwTDgGRtDw3/ruod142Xb-fdQB6Uf6rN/mDr0x-e8W3HpQlGBva1u, with debug_info, not stripped
1 Like

I’ve banned this account as I think it’s a bot.