Debian 10 (Buster), can't mount loop devices

When using Debian 10 as the host to build a CustomPiOS, pulling the latest stable release of Raspbian (Buster) the build script fails when it gets to mounting the loop devices. After the script fails, the following loop devices exist:

loop0 loop1 loop2 loop3 loop4 loop5 loop6 loop7 loop-control

Update:

Edited /etc/modules to add the line loop max_loop=128 and then reran my build script. It still fails creating loop devices but the output of /sbin/losetup -a is :

/dev/loop0: []: (/home/waldo/CommunityOS/src/workspace/2019-07-10-raspbian-buster-lite.img), offset 276824064

Are you building on vanilla Debian 10 or Rasbian?
I am not sure what how to reproduce this other than using vagrant.

Building directly on vanilla Debian 10 in an Oracle VM on Windows 10.

Even after pulling only the devel branch (using -b devel) with the git clone command and rerunning my script it still fails when trying to mount the loop devices. I am including the build.log so you can see what is happening.
build.log

I believe I may have solved this.

What didn't work was to create an entry in /etc/module for loop.
What worked was to create a configuration file for modprobe, /etc/modprobe.conf, then add the entry options loop max_loop=128. Save the file, then when modprobe is executed with loop the script is able to create more loop devices.

I tried this out in two distinct virtual machines (remember I am running VirtualBox on a Windows 10 host). The first was an upgraded VM from Debian 9 (stretch) to Debian 10 (buster). That one was hard, because it worked fine in Debian 9 but failed after the upgrade. The other VM was a fresh install of Debian 10, which also failed to create loop devices.

I have not tested whether or not the image actually runs on an rPi, but building on Debian does complete without errors. I need to purchase another rPi, since I am now using the three I own in other projects and I can't sacrifice my current OctoPrint setup for testing.

I am closing this issue.

This is worth documenting in the wiki

First, I spoke too soon. only the upgraded VM, which worked fine using Debian 9 (stretch) works with the above file added after upgrading to Debian 10 (buster). I still can't get a new VM of Debian 10 (buster) to build. It still fails at trying to mount loop devices.

I found other inconsistencies between building in buster vs stretch and may open other issues, unless I can find work-arounds.

So the behavior is odd, Debian is very temperamental with regard to loop devices. If I come from a clean boot, I can build an image. If I try to build it again in the same session it fails saying that loop is unavailable. At the very least maybe the cleanup or error routine should check for loop devices and remove them explicitly.

I am reopening since this really isn't solved.

you can also try using rhe vagrant build method

…

On Aug 5, 2019 03:09, "WheresWaldo" ***@***.***> wrote: So the behavior is odd — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#55?email_source=notifications&email_token=AACPQJWLFJVWSOTIW3KD53TQC5VUTA5CNFSM4IIGVI32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QMPUA#issuecomment-518047696>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACPQJWQUYU6ZKXNIWQV6NDQC5VUTANCNFSM4IIGVI3Q> .

I have never used that method, so I will need to study it to see how to set it up.

Just updating I a getting this two, both on my arm64 raspbian installation and also in ZynthaianOS project.
Its random - as in sometimes it happens, sometimes it doesn't. On my install it seems always to work after it fails. Investigating but would always love help with this, since its not 100% reproducible, I have to wait and try catch it.

I haven't built another OS in a while, I settled on Stretch for my PiOS since I could never get Buster to work. And on another note, the other guy who was working on a rewrite of the RoboOS user interface stopped working on it. My project is dead now, for all intents and purposes, since I am not a programmer, just capable of assembling tools is all.

Sorry to hear that :(

Will stress that for me the issue is random.
OctoPi nightly fail rate is about 15% from this issue over the past month. So its annoying but should not stop you from building altogether.

Looks like there is an open issue on Docker
moby/moby#27886

Cool, I guess in the future you would also use https://github.com/guysoft/UbuntuDockerPi , which ships with docker + docker-compose :)

If you could point out what was different from your last setup it might help others in this issue.

Could it be that the Pi simply isn't fast enough to trigger the issue? From my notes it seems that the errors point to a timing issue with the loop device:

losetup: /dev/loop14: detach failed: No such device or address
mount: /distro/workspace/mount/: failed to setup loop device for /distro/workspace/2020-02-13-raspbian-buster-lite.img.
losetup: 2020-02-13-raspbian-buster-lite.img: failed to set up loop device: Resource temporarily unavailable

1 and 2 are the most common versions of the problem. 3 is quite rare but more telling of the actual problem IMO.

The problem seems more likely to happen on small projects (like a bare distro out of src/make_custom_pi_os -g).

@Fulg

@Fulg quite possible. I was considering to build a distro, that builds other distros. It could ship with Jenkins and have CI/CD nightly builds, you just give it a github repo and it builds it.
Then we could test timing scenario like that with similar hardware and software. Never had time for that though.

@guysoft

@guysoft I understand that you much prefer using docker images to build the images, but I see the wiki mentions Raspbian as a host system to build with out-of-the-box and I remember making it work on the previous buster image some time ago! 😄

I tried again straight from the vanilla (latest to date) raspios-lite image on a RPi 4 but failed :(

sudo apt install -y git p7zip-full python3 lsof
git clone https://github.com/guysoft/CustomPiOS.git
CustomPiOS/src/make_custom_pi_os -g mytest
sudo modprobe loop
mytest/src/build_dist

Finishes with the infamous

losetup -f --show -o 272629760 2020-02-13-raspbian-buster-lite.img
losetup: 2020-02-13-raspbian-buster-lite.img: failed to set up loop device: Permission denied

I have also tried build-dist with the raspios image with the same result

Issue #46 details the same problem on the Ubuntu host because of overused loop devices, but losetup returns nothing in my case.

build.log

Note: I am building it straight on the SD card

@amrsoll

@amrsoll This is not related to this issue, you are not running with root permissions and you are getting a permission error.
Run sudo mytest/src/build_dist
The reason I am moving mostly to support docker is scenario where I have no idea what is going in the host system, it lets you keep track of the environment.
Build on Raspbian/Debian/RaspberryPi OS should work.

I build in DigitalOcean using a:

doctl compute droplet create rmspi-build --size 1gb --image debian-10-x64 --region nyc1

Experienced this on my first attempt, re-try would succeed, 3rd would fail, 4th succeeded. Log is gone now but I can spin this up and give it a try if you need it.

In dmesg I saw:

loop_set_status: loop0 () has still dirty pages (nrpages=30)

This is my build script I hacked together.

sudo modprobe loop

apt update
apt install p7zip-full qemu-system-arm qemu-user-static backblaze-b2 git curl

cd /usr/src/
git clone https://github.com/toddejohnson/rmspi.git RMSPi
git clone https://github.com/guysoft/CustomPiOS.git CustomPiOS
echo "/usr/src/CustomPiOS/src" > /usr/src/RMSPi/src/custompios_path
CURRENT_RASPBIAN=$(curl https://downloads.raspberrypi.org/raspbian_lite/images/ | grep raspbian | tail -n 1 | awk -F "href=\"" '{print $2}' | awk -F "/" '{print $1}')
CURRENT_RASPBIAN_FILE=$(curl http://downloads.raspberrypi.org/raspbian_lite/images/${CURRENT_RASPBIAN}/ | grep .zip | head -n 1 | awk -F "href=\"" '{print $2}' | awk -F "\">" '{print $1}')
curl -L -o "/usr/src/RMSPi/src/image/${CURRENT_RASPBIAN_FILE}" https://downloads.raspberrypi.org/raspbian_lite/images/${CURRENT_RASPBIAN}/${CURRENT_RASPBIAN_FILE}

cd /usr/src/RMSPi/src

./build_dist

cd /usr/src/RMSPi/src/workspace
CURRENT_IMG_FILE=$(basename $CURRENT_RASPBIAN_FILE .zip)
OUT_IMG_FILE=$(basename $CURRENT_RASPBIAN_FILE .zip| cut -d'-' -f -3)-rmspi-$(basename $CURRENT_RASPBIAN_FILE .zip| cut -d'-' -f 5-)-0-4
mv ${CURRENT_IMG_FILE}.img ${OUT_IMG_FILE}.img

7za a ${OUT_IMG_FILE}.zip ${OUT_IMG_FILE}.img

I used to use and tried to start with ubuntu-18-04-x64 although I had similar errors, gave up and decided Debian would be a better fit for this.

[Docker] mount /dev in container

Hey,
So possible workaround.

One that didn't work:
I tested running ls from the host and container (wrote a flask server to run ls and then pulled that from the container, no luck)

One I am going to test now:
This commit mmmspatz/pi-gen@0c84b0d based on this: moby/moby#27886 (comment) , suggests that adding to the docker compose
/dev:/dev as a volume fixes this. Will update.

Hi, lacking some context here but here's what I learned:

docker run --privileged gives the container access to everything in the host's /dev/, but new device nodes created in the host after the container is launched don't automatically show up in the container. So when you run losetup -f in the container, you'll see a new /dev/loop[n] appear in the host but not in the container, then losetup will fail. Mounting the host's /dev in the container fixes this and solves the problem.

A thought I have is: maybe you could create your loopback devices in the host before launching the container, then make use of them after launch? idk.

@mmmspatz

@mmmspatz that's what should happen, but there is a bug that it doesn't always update correctly in the container.
With your commit I suspect that performing a bind mount to /dev might get it to update correctly. Possibly because a bind mount to a folder updates differently.

Not sure if this is helpful, but over on the My Naturewatch Camera repo we use CustomPiOS to build the camera image with GitHub Actions. I used to get this error sometimes when building images on a Pi 3B, but we haven't seen it happen in a while.

@mikevanis

@mikevanis Indeed the github actions builds I have here don't get that error neither. I don't know why.

Also Naturewatch is cool! I recently released and also going to blog post about the rpi-imager fork I wrote that can accept community-maintained images: https://github.com/guysoft/pi-imager

If you want to PR a .json file to this repo, would love to have that added to the list:
https://github.com/guysoft/pi-imager-web/
I was considering it to message you about that. But if you are already here thought I might as well suggest.

Just updating that this is still an issue.
The workaround that I use ATM is to docker stop/docker start the container.

I have a similar issue. I tried to build a custom image with docker. Everytime i get this error:

mount: /distro/src/workspace/mount/boot/firmware: mount point does not exist. dmesg(1) may have more information after failed mount system call.

build.log

Any other workarounds?

@joungle

@joungle your error is a different one. You are building for an Armbian x84 images and it does not have a boot/firmware folder since PCs use a BIOS to discover hardware and not a device tree that is stored in boot/firmware like embedded devices.
You can try and workaround it by setting in config:

export BASE_ROOT_PARTITION=1

Also do share if you get FullPageOS working on x86, there are people asking for it and its GPL-3.0 license:
guysoft/FullPageOS#436

Navigation Menu

Repository navigation

Debian 10 (Buster), can't mount loop devices #55

5 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Debian 10 (Buster), can't mount loop devices #55

Description

Activity

WheresWaldo commented on Jul 31, 2019

guysoft commented on Jul 31, 2019

WheresWaldo commented on Jul 31, 2019

WheresWaldo commented on Aug 4, 2019

guysoft commented on Aug 4, 2019

WheresWaldo commented on Aug 4, 2019

WheresWaldo commented on Aug 5, 2019

guysoft commented on Aug 5, 2019

WheresWaldo commented on Aug 5, 2019

guysoft commented on Dec 12, 2019

WheresWaldo commented on Dec 12, 2019

guysoft commented on Dec 15, 2019

guysoft commented on Dec 29, 2019

5 remaining items

guysoft commented on May 4, 2020

Fulg commented on May 4, 2020

guysoft commented on May 5, 2020

amrsoll commented on Jun 18, 2020

guysoft commented on Jun 21, 2020

toddejohnson commented on Aug 14, 2020

guysoft commented on Mar 15, 2022

mmmspatz commented on Mar 15, 2022

guysoft commented on Mar 15, 2022

mikevanis commented on Mar 16, 2022

guysoft commented on Mar 17, 2022

guysoft commented on Nov 29, 2023

joungle commented on Jan 20, 2025

guysoft commented on Jan 21, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions