Troubleshoot dead instance on DO

1

you can discuss non-feature/non-bug stuff on the discussions tab or we do have https://forums.nanovms.com/ as well which I think we’d prefer

as for your particular issue with the ‘9 hours and its dead’ - we’re going to need some more information on what you’re running into; is the instance still up? if so you can drop into the serial console and take a screenshot or copy/paste what you see there and if it’s a kernel issue you’d probably see something; if the instance actually powered off then yes you can download the image and run it locally - if it crashed there might be a crash log inside of it that we can look at too; regardless of whether you have stuff from serial or not we’re going to need something that can reproduce what you’re seeing but first I’d check there

Forward from Github

2

So my application is just a simple rust webserver.

  1. The instance is still in DO, but I think it’s dead, since there is no memory information from klibs digitalocean.
  2. No ssh, how to get into that dead instance? Just in case if it’s alive how can we get into that?
  3. How to download it locally to do the inspection?

Thanks

3

  1. Just to clarify - is the instance ‘on’ or ‘off’? If it’s on can you see if there is any output in the serial console and can screenshot it or cut/paste it?

  2. There is no ssh as there is no support for running more programs than the one that is running currently. So there is no ‘interactivity’ inside the instance.

  3. You can backup the instance via How do I manually back up my Droplet? :: DigitalOcean Documentation but I wouldn’t go through the trouble of doing this until we’ve checked the serial. The problem is more than likely apparent there.

4
  1. It’s off, nothing is shown when I use this command ops instance list -t do -c config.json
    DO
    DO959×1102 37.2 KB
+------+----+--------+---------+-------------+------------+-------+
| NAME | ID | STATUS | CREATED | PRIVATE IPS | PUBLIC IPS | IMAGE |
+------+----+--------+---------+-------------+------------+-------+
5

For your console - I’m looking for something like you might see here:

image
image2442×1096 113 KB

If you click that ‘console’ link it’ll show any stdout/stderr you might have (on DO when the instance is running). I don’t know if you are forwarding any logs to a different logging target but that would work too.

Are you able to replicate this locally? Do you have any code that can replicate what you’re seeing?

1 Like
6

do

When I access the Console the instance is turned off, I turn it on and have this screeenshot.

Are you able to replicate this locally? Do you have any code that can replicate what you’re seeing?

I will try later today.

7

it seems like the server was crashed when there is a flood of requests ( bot ) , however I use the load test locally to test node running inside qemu locally, it works just fine without any problem.

8

Ok sounds great. Just for your reference on some clouds this console will retain logs even when the instance dies/turns off (such as GCP) but I think on Digital Ocean you can only see what is live so if it crashes or whatever you’ll lose the info. So for DO we suggest shipping out logs to some external service. Outside of putting that inside your app you can use the syslog klib to accomplish this: https://docs.ops.city/ops/klibs#syslog .

1 Like