Hello selfhosted community, something weird just happened to my setup while running a routine update.

I’m running docker containers on a couple Debian LXCs through Proxmox, and a regular apt-get upgrade just wiped all my configurations. Somehow it seems to have gutted my databases and deleted the compose.yml files without a trace remaining. Thankfully all my data seems to be intact as far as I can tell.

Did I royally mess something up in all of my configurations or in doing the update? This has never happened to me before. Thankfully I have a backup for the configs that’s about 6 days old, but it’s still extremely annoying. Any hints? Thanks

  • Player2@lemm.eeOPEnglish
    9·
    9 months ago

    Update in case anyone is interested: I figured out what caused the problem. When I mounted the new drive I used to store my configurations onto Proxmox, I completely forgot to make the relevant /etc/fstab entry. The drive mounted successfully so I didn’t realize at the time that I had forgotten to do that step. The update I ran from apt-get included a kernel update, so I restarted the machine to complete it. Since I hadn’t modified fstab, my new drive was not mounted when the system started up again. Even though the drive wasn’t mounted correctly, I still somehow had access to some incomplete version of the files in its directory (no idea how that works). So no fault of Docker, LXCs or Proxmox, purely PEBKAC.

    Despite getting the files back I will still work towards a more resilient system and more regular backups.

  • mikyopii@programming.devEnglish
    8·
    9 months ago

    I’m not sure. All I know is that Proxmox doesn’t recommend using Docker in LXCs. It seems Proxmox has much more control over the internals of the container and can break stuff.

    https://pve.proxmox.com/wiki/Linux_Container

    If you want to run application containers, for example, Docker images, it is recommended that you run them inside a Proxmox QEMU VM. This will give you all the advantages of application containerization, while also providing the benefits that VMs offer, such as strong isolation from the host and the ability to live-migrate, which otherwise isn’t possible with containers.

    • Player2@lemm.eeOPEnglish
      53·
      9 months ago

      It was just a matter of time then until something broke? Guess I’ll need to do some more research on how to best manage the services I want to run. Good thing I didn’t come into this hobby hoping to reduce the amount of headaches

      • towerful@programming.devEnglish
        2·
        9 months ago

        I run a debian VM for hosting docker things.
        Its like 500mb extra disk space, and a small ram/cpu hit. But its been solid for me

  • Crogdor@lemmy.worldEnglish
    6·
    9 months ago

    You didn’t happen to change an unprivileged container to privileged, or vice versa, after creating it, right? Doing so can break filesystem permissions, which could have resulted in something like this.

    • Player2@lemm.eeOPEnglish
      2·
      9 months ago

      Interesting, I think I did actually do that. I guess my best bet for now is to just nuke the LXCs then and move everything to a new VM. Thanks for the advice!

  • MangoPenguin@lemmy.blahaj.zoneEnglish
    5·
    9 months ago

    That’s pretty weird, where were the files stored that went missing? Just in your home folder?

    Definitely recommend setting up regular daily backups, you can use the built in proxmox backup system, or run whatever tool you want inside each VM/CT.

    • Player2@lemm.eeOPEnglish
      1·
      9 months ago

      The files were on a mountpoint, completely separate drive. This has also been the case for all the previous times I ran an update, though I did recently move these files to a different drive mounted the same way. I got some sort of permission wrong maybe?

      Will definitely set up a better backup system as soon as I can

      • MangoPenguin@lemmy.blahaj.zoneEnglish
        1·
        9 months ago

        Very odd, I think something else is going on not related to LXC/Proxmox. I’ve got LXC containers with external mounts and internal storage and haven’t seen any data loss over the years they’ve been running.