Fan turning off and tinypilot "/opt/tinypilot/venv/bin/python: no such file or directory" error
- @david
On one of our new tinypilots we've had a few weird issues that may be unrelated, after setting it up and having it work for a few weeks, we stopped being able to connect to the tinypilot and received the attached errors.
We attempted re-imaging the SD card a few times and on the 3rd try it appears to no longer be giving us this error, but I did notice that the fan on this particular unit is behaving differently, it's louder and sometimes stops altogether, which I have not noticed on any of the other units we have.
Please let me know if I can provide any other information.
David @david2024-05-14 11:58:37.731ZHi @asala, I'm sorry you're running into these issues with your TinyPilot.
the fan on this particular unit is behaving differently, it's louder and sometimes stops altogether, which I have not noticed on any of the other units we have.
Newer TinyPilot Voyager 2a devices have basic fan control based on the device's CPU temperature. If the CPU is cool enough, the fan turns off. If the CPU temperature increases, the fan will turn on again. Since this device is likely newer than your other units, this fan control update probably explains this change in behavior.
If the fan is considerably louder than your other devices (and has noises that aren't just airflow), there could be an issue with the fan itself. I've reached out to our customer support team to help with this.
Thanks for sharing those screenshots of the missing file issue.
It looks like your microSD card may have experienced filesystem corruption. Typically, filesystem corruption presents itself with missing files/directories. So your screenshots showing the missing python file suggests filesystem corruption may have been the cause of that particular issue.
The solution (as you've found) is to re-flash the microSD card. Now that you're back up and running, I'd recommend enabling TinyPilot's read-only filesystem to help increase the lifetime of your microSD card and to reduce the risk of filesystem corruption recurring.
If you don't already, I'd also recommend shutting down your device via the web interface instead of pulling the power cable. On TinyPilot's web interface, navigate to System > Power and then click "Shut Down".
I hope that helps! Please let me know if you have any questions.
- AAaron @asala
Hi David,
Thanks for the response, so far after re-flashing the SD card it seems like it is working properly again. Good to know about the revision to the Tinypilot, the fan doesn't sound like grinding or anything so I'm sure that's all that it is.
In the meantime however, we had another one of our tinypilot voyager 2a devices experience the same issue. This one however plugging the MicroSD card into a computer results in an instant blue screen. We've tried both the SD card reader built directly into the laptop as well as a USB-MicroSD card adapter and got the same behavior both times.
I'm beginning to get a little concerned that two different TinyPilots have both experienced the same issue with MicroSD card corruption, we use these devices out at customer sites and they are not always as well equipped to handle assisting us with running through a reset procedure. Is this something that we should be expecting to happen fairly frequently? We have ~5 of these devices in production currently but would most likely begin expanding this to 40+ units, but we can't be relying on these devices if every month we have a few that need to be reset, and subsequently re-setup.
David @david2024-05-31 14:24:29.866Zthe fan doesn't sound like grinding or anything
Thanks for the update, @asala! This definitely sounds like it's just the updated fan control.
plugging the MicroSD card into a computer results in an instant blue screen.
To clarify, when you insert the microSD card into your computer to perform a factory reset, your computer crashes with a blue screen?
I haven't heard of that happening before. It sounds like it could be an issue with how the computer is reading the SD card. Does this blue screen happen only with this particular microSD card?
Is this something that we should be expecting to happen fairly frequently?
I understand the apprehension if you've experienced this twice.
The risk of filesystem corruption increases the more data your device writes to its microSD card. You may also encounter this issue more frequently if your TinyPilot is permanently powered on, or if your TinyPilot experiences unexpected power loss while running.
The underlying issue is that microSD cards are a wearable storage medium, meaning that their reliability degrades over time. Having said that, only a small percentage of TinyPilot users experience filesystem corruption.
To improve reliability, we focus on minimizing these failures by reducing the amount of disk writes TinyPilot's software makes. Our 2.6.0 release highlights one of the significant changes we made to our update process to help with this. Additionally, we continuously look for improvements to prevent unnecessary writes to the disk.
There are multiple additional things you can do to help prevent this kind of filesystem corruption from occurring.
My first suggestion is to enable TinyPilot's read-only filesystem as soon as the system is set up and configured the way you want it. The read-only filesystem will prevent your device from writing to the microSD card, instead using its RAM to write data while running. This reduces the number of disk writes, prolonging the life of the microSD card and reducing the risk of filesystem corruption.
Sudden power loss (either through a power cut or removing the device's power cable when running) can cause these filesystem corruption issues. You can safely shut down your TinyPilot device by going to System > Power > Shutdown in the web interface. Once the device shuts down, you can safely remove the power cable.
Depending on your reliability requirements, you may want to consider booting your TinyPilot device from USB, using a USB drive or even an external drive (e.g., an SSD).
Making a clone of a (fully configured and set up) microSD card might also be useful. A clone would allow you to quickly flash a different microSD card with a working and configured system if anything happens to go wrong. For your particular use case of sending a device to a customer site, you could send a duplicate microSD with the TinyPilot as a redundancy, just in case.
I hope that helps! Please let me know if you have any questions.
- AAaron @asala
I'll definitely give the read only file system a go. Would it be possible to backup the microSD card to a USB stick plugged into the Tinypilot? Effectively having a single hot swap replacement in the case of corruption of the SD card?
- David @david2024-06-04 12:33:54.786Z
Would it be possible to backup the microSD card to a USB stick plugged into the Tinypilot?
I think it's possible to clone from a live microSD card on-device to a USB drive, however, I don't think it's a great option. There are a few caveats with cloning a live disk, like file changes, or issues if your target disk is slightly smaller in capacity than the source disk (two disks with a 32GB capacity can vary is actual capacity).
I just tested cloning from a live microSD card to a USB drive on my own device, but it wasn't super successful and it was quite slow.
The simplest option to clone/backup the microSD card is with Balena Etcher's "Clone" feature (or a similar tool). You'd insert both the microSD card and USB stick into a PC and clone the SD card directly to the USB drive.
Is there a specific reason you'd like to clone on-device?
- AAaron @asala
Thanks for the response, the reason I was hoping it would be possible to clone on the device itself is ease of use for our customers. Since we send these directly onsite to our customers for setup, in order for us to take a clone of the SD card after it was setup we'd have to have them use and SD card reader and plug everything into a computer rather than just asking them to plug a USB stick into the tinypilot itself.
- David @david2024-06-11 11:40:25.032Z
Thanks for the explanation!
Can I take a little time to do some more research and testing here? I'll try and find a consistent method of cloning a microSD card on a live device to a USB drive.
I'll provide an update later this week (Friday at the latest).
Please let me know if you have any questions in the meantime.
- In reply toasala⬆:David @david2024-06-14 12:37:34.199Z
Thanks for your patience, @asala!
I've tested this a little more, and I've found a way to clone the microSD card on a live device. However, there are a couple of caveats.
First, you need to make sure your TinyPilot has its read-only filesystem enabled.
Next, you need to make sure the device you're cloning your microSD to (either a USB drive or a microSD card in a USB adapter) has a larger capacity than the microSD card your TinyPilot is running on.
With the drive inserted into a USB port, SSH into your TinyPilot and run the following command:
lsblklsblkwill show information about the connected drive devices:NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 1 29.7G 0 disk └─sda1 8:1 1 29.7G 0 part mmcblk0 179:0 0 29.7G 0 disk ├─mmcblk0p1 179:1 0 256M 0 part /boot └─mmcblk0p2 179:2 0 29.5G 0 part /In this instance,
/dev/sdais the drive I've connected to my USB port (It may be different for you) andmmcblk0is the microSD card currently running.Now, to clone the drive, run the following command:
# Change the value of DRIVE to your drive's path. DRIVE="/dev/sda" sudo dd if=/dev/mmcblk0 of="${DRIVE}" bs=4M status=progressIt may take a while to clone the drive, but the
ddcommand shows progress.Once the command finishes and you've cloned your drive, you can boot from the new cloned drive. If it's a USB drive, you should remove the microSD card from TinyPilot's microSD slot first. I'll also note that you should clone the drive while the system is healthy and working.
I hope that helps! Please let me know if you have any questions.