Lots of 'No space left on device' when updating game, unpacking tar files

Anbox Cloud Users
1

Hi, I try to update a game on regular container, but while updating some error occur.

(we need a live update so this is required. This is not a hook.)

When launching a container, it remain ‘prepared’ for long time and fall into the ‘error’ status.

The error log by ‘amc show’ is like below.

Failed creating instance from image: Unpack failed: Failed to run: tar --restrict --force-local -C /var/snap/lxd/common/lxd/storage-pools/ams0/images/595ad93200a4b2047917018ff8fc72deee11e
bb6427c5ac8acd27ff9b224c857 --numeric-owner --xattrs-include=* -xf -: exit status 2

and trailing lots of ‘No space left on device’ error. Below is some of those error.

....
tar: rootfs/etc/ssl/certs/Atos_TrustedRoot_2011.pem: Cannot create symlink to ‘/usr/share/ca-certificates/mozilla/Atos_TrustedRoot_2011.crt’: No space left on device

I am sure that the disk size is enough for that game and no special setting for this.

resources:
  cpus: 2
  memory: 4.00GB
  disk-size: 32.00GB
  gpu-slots: 1

I use anbox cloud version 1.21.1, and also test more recent or old version(1.20.2) but still same error happen.

So could you help me find the reason and solve this problem?

(I suspect that as there are lots of tar files for updating game, so maybe there are chances that some kind of inode for lxd just run out.)

(+ I also find out this happens when unpacking only one file :frowning: )

2

Hey @ruin09 ,

Can you provide us with the output of sudo anbox-cloud-appliance.buginfo (assuming you’re running the Anbox Cloud Appliance)? Alternatively the output of the following commands:

$ sudo zpool status
$ sudo zfs list
$ sudo df -h
$ lxc storage ls
$ amc node show lxd0

It looks like you’re running out of storage space somewhere. The output should help us to figure what it is.

Thanks!

1 Like
3

I am using cloud (with juju), not appliance.

and below is the command result that you provide.

ocs@ip-10-246-254-37:~$ sudo zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
ams0  1.46T  3.18G  1.46T        -         -     0%     0%  1.00x    ONLINE  -
 
 
ocs@ip-10-246-254-37:~$ zfs list
NAME                                                                                USED  AVAIL     REFER  MOUNTPOINT
ams0                                                                               3.18G  1.41T       24K  legacy
ams0/buckets                                                                         24K  1.41T       24K  legacy
ams0/containers                                                                     860K  1.41T       24K  legacy
ams0/containers/ams-co322ng46sd9chuo54g0                                            836K  1.41T     1.84G  -
ams0/custom                                                                          24K  1.41T       24K  legacy
ams0/deleted                                                                        144K  1.41T       24K  legacy
ams0/deleted/buckets                                                                 24K  1.41T       24K  legacy
ams0/deleted/containers                                                              24K  1.41T       24K  legacy
ams0/deleted/custom                                                                  24K  1.41T       24K  legacy
ams0/deleted/images                                                                  24K  1.41T       24K  legacy
ams0/deleted/virtual-machines                                                        24K  1.41T       24K  legacy
ams0/images                                                                        3.17G  1.41T       24K  legacy
ams0/images/07be09b4cd5a9aca561850d44e839f4aec42d28bf8f8a44efb9d3c9e4cc22e48_ext4  1.34G  1.41T     1.34G  -
ams0/images/d4f3564e4bb152159d2180dd7824f20930f4190c1629f65e7e63849318f1cca4_ext4  1.84G  1.41T     1.84G  -
ams0/virtual-machines                                                                24K  1.41T       24K  legacy
 
 
ocs@ip-10-246-254-37:~$ df -h
Filesystem       Size  Used Avail Use% Mounted on
/dev/root        243G   54G  189G  23% /
tmpfs             63G     0   63G   0% /dev/shm
tmpfs             26G  2.2M   26G   1% /run
tmpfs            5.0M     0  5.0M   0% /run/lock
efivarfs         512K  4.9K  508K   1% /sys/firmware/efi/efivars
/dev/nvme0n1p15   98M  6.3M   92M   7% /boot/efi
tmpfs            1.0M     0  1.0M   0% /var/snap/lxd/common/ns
tmpfs             13G  4.0K   13G   1% /run/user/1001
 
ocs@ip-10-246-254-37:~$ lxc storage ls
Error: The LXD daemon doesn't appear to be started (socket path: /var/snap/lxd/common/lxd/unix.socket)
 
 
root@anbox-cloud-ams:/home/ocs# lxc storage list
If this is your first time running LXD on this machine, you should also run: lxd init
To start your first container, try: lxc launch ubuntu:22.04
Or for a virtual machine: lxc launch ubuntu:22.04 --vm
 
+------+--------+--------+-------------+---------+-------+
| NAME | DRIVER | SOURCE | DESCRIPTION | USED BY | STATE |
+------+--------+--------+-------------+---------+-------+
 
ocs@anbox-cloud-ams:~$ amc node show lxd0
name: lxd0
managed: true
status: online
unschedulable: false
architecture: aarch64
disk:
  pool: ams0
  size: 1TB
network:
  address: 10.246.254.37
  bridge-mtu: 9001
config:
  public-address: 10.246.254.37
  cpus: 64
  cpu-allocation-rate: 4
  memory: 128GB
  memory-allocation-rate: 2
  gpu-slots: 64
  gpu-encoder-slots: 64
  tags: []
gpus:
  0:
    pci-address: "0002:01:00.0"
    render-name: renderD128
    slots: 32
    encoder-slots: 32
    numa-node: 0
  1:
    pci-address: "0002:03:00.0"
    render-name: renderD129
    slots: 32
    encoder-slots: 32
    numa-node: 0
vpus: {}

Do you think I should change some options with LXD ?

Thanks a lot for your support.

4

Thanks for the support. I solve this problem by setting volume.size to 32GB in config.

I find this in lxd community.

Refer to https://documentation.ubuntu.com/lxd/en/stable-5.0/howto/storage_volumes/#howto-storage-volumes