10 sec+ Latency w/ Full Raspbian OS
I installed Raspbian OS, used USB HDMI capture device and was experiencing over 10 seconds of latency on the video feed. I reached out to you via e-mail and you suggested maybe trying the CSI bridge. Ordered and installed today.
I was thinking that we were having an issue with the capture device but after installing the CSI Bridge we are still experiencing about 10 seconds of delay. I'm wondering if the pi is overheating?
Also, as I was reading through your instructions I noticed that you recommended running Raspbian Lite OS. Is this mandatory for functionality or have you tested it on "heavier" OS's?
I would like to be able to utilize the functionality of the GUI while also being able to use TinyPilot and I am wondering if this has been the cause of the latency the entire time? What do you think?
- 7 replies
- Michael Lynch @michael2021-02-11 19:23:24.468Z
Thanks for reporting this! Sorry about the latency you're hitting.
OS as the cause
A 10 second delay would be a 50x increase over the normal latency, so it's unlikely just a difference in OS.
Raspberry Pi OS Lite is the only officially supported OS, but it should work fine with the full version. You can run the
topcommand to check whether your CPU is at 100% or if you're out of memory.
Throttling as the cause
Throttling is an interesting idea. It shouldn't be happening unless your device is in an unusual environment or you're otherwise running heavy loads.
Can you try these commands?
sudo vcgencmd measure_temp sudo vcgencmd get_throttled
If your Pi has throttled the CPU since boot, you'll see a non-zero code for
get_throttledthat corresponds to these values.
Network link as the cause
My guess is that the bottleneck is the network link. How are you connecting to the TinyPilot? Is it on the same local LAN or is the connection going over the Internet?
The video stream is in MJPEG, which is very bandwidth-hungry. I'm planning to migrate away from MJPEG to a more efficient encoding in the next few months, but there's a workaround in the meantime.
You can reduce the network load by dialing down the streaming settings. To do this, SSH into the TinyPilot and run:
sudo nano /lib/systemd/system/ustreamer.service
ExecStartsection with this:
ExecStart=/opt/ustreamer/ustreamer \ --host 127.0.0.1 \ --port 8001 \ --encoder omx \ --format uyvy \ --workers 3 \ --drop-same-frames 30 \ --persistent \ --dv-timings \ --quality 50 \ --desired-fps 10 \ --resolution 640x480
Save the changes and then run this command to restart the stream with the new settings:
sudo systemctl daemon-reload && sudo service ustreamer restart
Thanks Mike! This mostly fixed the issue. I am down to about a half of a second latency now which is a considerable improvement. I am also suspecting it may be a power management issue as well. I switched out the power supply and that improved it to about 2 seconds of lag. Changing the settings you mentioned above got me to where I am now.
I know the pi needs 5V and 3a to run but with the extra peripherals (e.g. LTE HAT, CSI Bridge, USB Hub and HDMI splitter) I am suspecting I need more juice to run everything. Although the only 2 things I had connected when the issue first presented itself were the CSI bridge and the LTE HAT.
Temp was 43.8
Throttled came back as 0x5500
Switched power supply and throttled now says 0x0
- Michael Lynch @michael2021-02-11 22:48:45.001Z
It might be a weak power supply rather than the Pi drawing more than 3 A.
Before choosing the power supply that I include in official kits, I asked my electrical engineering consultants to test several options, and some of them failed to deliver their advertised 5 V / 3 Amps. The one that performed best in their tests was the FONKEN 18W charger.
- DMatthew Scott @dolosinc2021-02-12 02:08:38.208Z
Tested new power supply and it is putting out correct voltage and amps consistently.
However, the problem is BACK!
The interesting part is that if I refresh the webpage it instantly displays real time output from the host computer. This is about 7 seconds faster than not refreshing the page.
Is there something getting hung up in the code that I am missing here? I'd be willing to do a shared terminal session with you if you want to take a peek. I could give you access to my VPN as well if you want to see the issue first hand? I am completely at a loss here...
- DMatthew Scott @dolosinc2021-02-12 05:13:41.004Z
I think I may have narrowed it down to an issue with the pi overheating itself. When the pi gets to about 45 degrees it starts slowing down significantly. Strangely though the command you gave me to check for throttling consistently returns 0x0.
I am installing a much better heat sync and fan tomorrow to address the heating issue. I have been able to keep it at 39 degrees with my current setup but when we were testing it was out of the case and therefore heat became a serious issue. Without fan we were running at right around 56.
If the issue persists further I will leave an update here. Still willing to let you take a look if your interested.
- Michael Lynch @michael2021-02-12 22:37:55.147Z
I think the heat is a red herring. 45 C isn't very hot for a CPU. I don't think the Pi typically begins throttling until the 65-70 C range.
What is the connection between the browser and the TinyPilot? You mention VPN, so I assume it's over the Internet. Does it go through any web proxies? MJPEG is an uncommon protocol, so web proxies often handle it poorly. Some browsers also get slower if you leave the page open for long periods because they leak memory when trying to render MJPEG.
Interestingly enough we have literally identical latency over our VPN using LTE HAT as we do when connected through ethernet directly to our LAN. I honestly didn't expect much of a difference but because at least some latency in the video is always present it is indistinguishable to the naked eye.
No proxies either.
I'll troubleshoot with the browser idea a bit. I am notorious for having 20 tabs open at any given time, sometimes for weeks.. Haha