TinyPilot has been perfect for months, suddenly USB input lag. Reboot doesn't help :(
Hey guys, our TP has been perfect for months. Suddenly, I've got about a 1 second delay between each key (!) arriving at the host. I've rebooted through the UI and CLI, no change.
I grabbed a shareable log, https://logs.tinypilotkvm.com/DdW0xGP3 if that helps. I've had this problem happen before, and resolved it by some combination of power cycle or unplug-replug sequence, unfortunately this unit is locked away in a datacenter that I was hoping not to visit until next month.
Anything we can try to reset this? Thanks!
- K@khmann
Uh, nevermind I guess? Suddenly it's back to normal and I haven't rebooted. I'm a UNIX networking guy by trade, so I did check all the bandwidths and latencies, etc., so I guess it fixed itself?
I went ahead and got another log https://logs.tinypilotkvm.com/E7BPFyDW incase you're interested.EDIT: NOPE! No sooner did I close this tab then it came back - https://logs.tinypilotkvm.com/LoRt91Gs
The issue seems unaffected by browser restarts (Safari, Firefox). I've been pinging it, and latency rarely exceeds 50ms.I've gotten my work done for now, and the key input latency seems to be intermittantly better, but if you've got any ideas for the future I'll take 'em
- CIn reply tokhmann⬆:Charles Hague @cghague2022-11-08 13:33:55.777Z
Hi @khmann - thanks for reaching out to us with your query about lag.
There can be a number of different causes for latency and lag. Can you please let me know how you're connecting to your TinyPilot device (e.g. local network, private VPN, cloud VPN provider etc)? Are any other elements affected by the same lag (e.g. mouse movement or the rest of the TinyPilot web interface)?
I am connected to the site via private L2TP VPN from my (Monterey OS) Intel iMac. Today I'm using WiFi, and it's not the best, but I have experienced this same issue over Ethernet, and most of the time this issue is not present over WiFi. The 2.4.1 TinyPilot is set to 80% quality, 30 fps, and has a 200Mbps symmetrical Internet connection. The web interface responds very well, and when I ssh into the TinyPilot the key response is excellent. I do not currently have any mouse-enabled systems to test with.
It's strange, because as I mentioned it's been consistently perfect since September, without any reboot or maintenance to either TinyPilot or the connected host. I have experienced this problem before, directly connected via ethernet to the same switch as the TinyPilot, and "fixed" it by unplugging and replugging power and usb cables, but I don't have that option right now, so I would love any ideas ie: "echo 1 > /sys/usb/something_or_other/reset_power" or "modprobe -r usb_doohickey", or anything else.
I've linked a video where you can see how fast I can type in an ssh session to TinyPilot, and how slow the host connected to TinyPilot (in it's slow mood) is receiving keystrokes. Normally when TinyPilot is working correctly, I can't type faster than it responds. I may get ahead of a screen update for an instant, but when the display catches up all my keystrokes are there. In the video you can (barely) see the up-to-1 second interval between keystrokes received by the host. This is not how it normally behaves...
Anyway, I appreciate any insight. Thanks!
- CIn reply tokhmann⬆:Charles Hague @cghague2022-11-09 23:32:03.291Z
Thank you for the detailed response!
I can see from the
pingresults in the clip that there is some minimal packet loss and a bit of jitter. That would be unlikely to affect keyboard input but it is possible it could affect the MJPEG video stream. I rewatched the clip with that in mind and I noticed at various points the cursor appears to freeze which would support the theory that the video is lagging rather than the input.Could you please try to verify this by running
watch -d -n 1 dateon the target machine and observing the video stream? The command will simply display the time every second until cancelled (using Ctrl+C) and should make it easy to spot if the video freezes, slows down or speeds up excessively.If the issue does appear to be the video stream rather than the input then it could be worth switching to the H264 format as it may handle the jitter and packet loss better.
Man... I knew you were going to call me out on that lost packet, but it's an intermittent problem and it wasn't convenient for me to switch to Ethernet that day.
The point of the ssh to the tinyPilot was to show there was very little latency in normal TCP packets. That doesn't change, even while operating the host.
I don't really think codec latency is my issue, as if the display was delayed all the keystrokes would just show up in a batch when the screen refreshed, instead of trickling in every 200ms or more; Also, in the video, the host cursor keeps flashing between received characters!
I'll compose another host video, via ethernet, with output such as you've requested. Thanks!