No internet connection
  1. Home
  2. Technical Support

502 Bad Gateway after reboot

By @dbak
    2022-10-17 16:43:16.585Z

    Issued a reboot to the system and after reboot, trying to connect to the web service results in a 502 bad gateway.

    =====================================================

    Checking logs, see the following being repeated over and over.

    Oct 17 08:31:23 0200707e systemd[1]: tinypilot.service: Main process exited, code=exited, status=1/FAILURE
    Oct 17 08:31:23 0200707e systemd[1]: tinypilot.service: Failed with result 'exit-code'.
    Oct 17 08:31:23 0200707e systemd[1]: tinypilot.service: Service RestartSec=100ms expired, scheduling restart.
    Oct 17 08:31:23 0200707e systemd[1]: tinypilot.service: Scheduled restart job, restart counter is at 5.
    Oct 17 08:31:23 0200707e systemd[1]: Stopped TinyPilot - RPi-based virtual KVM.
    Oct 17 08:31:23 0200707e systemd[1]: Started TinyPilot - RPi-based virtual KVM.
    Oct 17 08:31:23 0200707e python[1627]: Traceback (most recent call last):
    Oct 17 08:31:23 0200707e python[1627]: File "app/main.py", line 13, in
    Oct 17 08:31:23 0200707e python[1627]: import api
    Oct 17 08:31:23 0200707e python[1627]: File "/opt/tinypilot/app/api.py", line 8, in
    Oct 17 08:31:23 0200707e python[1627]: import auth
    Oct 17 08:31:23 0200707e python[1627]: File "/opt/tinypilot/app/auth.py", line 19, in
    Oct 17 08:31:23 0200707e python[1627]: import video_service
    Oct 17 08:31:23 0200707e python[1627]: ModuleNotFoundError: No module named 'video_service'
    Oct 17 08:31:23 0200707e systemd[1]: tinypilot.service: Main process exited, code=exited, status=1/FAILURE

    =====================================================

    Any idea how I can get this working again?

    Reply
    • 10 replies
    1. C
      Charles Hague @cghague2022-10-17 19:15:24.870Z

      Hi @dbak - thanks for reaching out, I'm sorry to hear your TinyPilot isn't restarting correctly.

      This seems to be a new issue that we've not seen before and so we'll need to dig in a bit further. Would you be able to share the full log with us and, if possible, advise if you made any changes (such as installing updates) before rebooting?

      Reply
      1. @dbak
          2022-10-17 19:20:09.434Z

          I'll need to check that I have the OK to share the entire log.

          No recent changes were made. From the web interface, ran the check for update and it failed. Checked with our network team and there were no blocks or requests of any kind from the device. As the first troubleshooting test, I issued a reboot command from SSH session and the result was the 502 after the reboot. I issued another after that, same result.

          Reply
          1. In reply tocghague⬆:
            @dbak
              2022-10-17 19:31:53.943Z

              Let me know which log file you need.

              Reply
            • C
              In reply todbak⬆:
              Charles Hague @cghague2022-10-18 18:25:35.459Z

              Thanks for the update @dbak.

              TinyPilot can automatically gather the relevant logs and generate a link that you can share with us. This is usually done via the web interface but as that isn't possible you can instead trigger the same process over SSH by running sudo /opt/tinypilot-privileged/collect-debug-logs. You'll be prompted before anything is shared.

              It sounds increasingly like this could be a case of filesystem corruption. As the system is still bootable you might be able to fix the issue by forcing a reinstallation of TinyPilot as follows:

              if [[ -f /opt/tinypilot-updater/install ]]; then
  (sudo apt remove -y tinypilot || true) && \
    cd /opt/tinypilot-updater && \
    sudo ./install
else
  /opt/tinypilot/quick-install
fi

              It's unusual for a corrupted filesystem to remain bootable and other areas of the filesystem may still be affected even if the above does get TinyPilot up and running. I appreciate it can be a hassle but if possible I'd recommend reflashing the microSD card for a clean start.

              I hope this is helpful. Please let me know if you have any questions!

              Reply
              1. @dbak
                  2022-10-18 18:39:56.039Z

                  Debug logs have been uploaded:

                  http://logs.tinypilotkvm.com/dHhvwVaw

                  Reply
                  1. @dbak
                      2022-10-18 18:43:03.412Z

                      As I am not local to the device, re-flashing is not a desirable option. The whole point of buying your product was to allow portable kvm's to issue to geographically remote locations. This is the first device we have been testing and since we had no issues prior, we have 4 more on the way. If we have to re-flash constantly that will be an issue.

                      Reply
                  2. C
                    In reply todbak⬆:
                    Charles Hague @cghague2022-10-18 20:48:39.133Z

                    Thanks for getting back to me so quickly - I'm sorry to hear that the reinstall attempt failed.

                    The logs seem to suggest that several files may be missing. Depending on the cause you might be able to remedy this by reinstalling from the web. To do so please visit the download page, enter your details and then follow the instructions on how to "Install from command-line".

                    I completely understand your concerns about having to reflash the devices frequently. Once you restore your device to a working state, you can drastically reduce the risk of filesystem corruption on the TinyPilot by enabling the read-only filesystem.

                    Reply
                    1. @dbak
                        2022-10-18 21:59:43.185Z

                        So the cybersecurity team is blocking that website saying it is malicious. I'll provide you with more details when I have them. Do you have a more reputable site that we can download your factory image from?

                        Reply
                        1. Michael Lynch @michael2022-10-18 22:08:50.226Z

                          That's strange. tinypilotkvm.com is our official webpage, and we've never had issues with firewalls blocking it.

                          I've just sent you an email with the installation command and a link to our microSD image not on the tinypilotkvm.com domain.

                          Reply
                      • Progress
                        with handling this problem
                      • @dbak
                          2022-10-18 19:02:47.242Z

                          so the script for reinstall fails. not seeing any install directory under tinypilot-updater, nor does quick-install exist under /opt/tinypilot.

                          pilot@0200707E:~ $ cd /opt
                          pilot@0200707E:/opt $ ls
                          tinypilot tinypilot-privileged tinypilot-updater ustreamer vc
                          pilot@0200707E:/opt $ cd tinypilot-updater/
                          pilot@0200707E:/opt/tinypilot-updater $ ls -la
                          total 12
                          drwxr-xr-x 3 root root 4096 Oct 17 07:51 .
                          drwxr-xr-x 7 root root 4096 Dec 1 2021 ..
                          drwxr-xr-x 3 root root 4096 Oct 17 07:51 venv
                          pilot@0200707E:/opt/tinypilot-updater $ cd ..
                          pilot@0200707E:/opt $ cd tinypilot
                          pilot@0200707E:/opt/tinypilot $ ls -la
                          total 156
                          drwxr-xr-x 13 tinypilot tinypilot 4096 Sep 29 12:49 .
                          drwxr-xr-x 7 root root 4096 Dec 1 2021 ..
                          drwxr-xr-x 11 tinypilot tinypilot 4096 Sep 29 12:49 app
                          -rw-r--r-- 1 tinypilot tinypilot 213 Mar 24 2022 app_settings.cfg
                          -rw-r--r-- 1 tinypilot tinypilot 4548 Sep 29 12:34 ARCHITECTURE.md
                          drwxr-xr-x 2 tinypilot tinypilot 4096 Sep 29 12:34 .circleci
                          -rw-r--r-- 1 tinypilot tinypilot 13864 Sep 29 12:34 CONTRIBUTING.md
                          -rw-r--r-- 1 tinypilot tinypilot 56 Sep 29 12:34 COPYRIGHT
                          drwxr-xr-x 2 tinypilot tinypilot 4096 Sep 29 12:34 .dependabot
                          -rw-r--r-- 1 tinypilot tinypilot 85 Sep 29 12:34 dev_requirements.txt
                          drwxr-xr-x 4 tinypilot tinypilot 4096 Sep 29 12:49 dev-scripts
                          drwxr-xr-x 2 tinypilot tinypilot 4096 Sep 29 12:34 e2e
                          drwxr-xr-x 8 tinypilot tinypilot 4096 Sep 29 12:49 .git
                          -rw-r--r-- 1 tinypilot tinypilot 838 Sep 29 12:34 .gitignore
                          drwxr-xr-x 2 tinypilot tinypilot 4096 Sep 29 12:34 hooks
                          -rw-r--r-- 1 tinypilot tinypilot 0 Sep 29 12:34 init.py
                          -rw-r--r-- 1 tinypilot tinypilot 9243 Sep 29 12:34 LICENSE
                          -rw-r--r-- 1 tinypilot tinypilot 65 Sep 29 12:34 MANIFEST.in
                          -rw-r--r-- 1 tinypilot tinypilot 1140 Sep 29 12:34 NOTICE
                          -rw-r--r-- 1 tinypilot tinypilot 97 Sep 29 12:34 package.json
                          -rw-r--r-- 1 tinypilot tinypilot 23 Sep 29 12:34 .prettierignore
                          -rw-r--r-- 1 tinypilot tinypilot 9940 Sep 29 12:34 .pylintrc
                          drwxr-xr-x 2 tinypilot tinypilot 4096 Sep 29 12:34 readme-assets
                          -rw-r--r-- 1 tinypilot tinypilot 3225 Sep 29 12:34 README.md
                          -rw-r--r-- 1 tinypilot tinypilot 360 Sep 29 12:34 requirements.txt
                          drwxr-xr-x 2 tinypilot tinypilot 4096 Sep 29 12:34 scripts
                          -rw-r--r-- 1 tinypilot tinypilot 493 Sep 29 12:34 setup.py
                          -rw-r--r-- 1 tinypilot tinypilot 165 Sep 29 12:34 .shellcheckrc
                          -rw-r--r-- 1 tinypilot tinypilot 32 Sep 29 12:34 .style.yapf
                          drwxr-xr-x 6 tinypilot tinypilot 4096 Mar 24 2022 venv
                          drwxr-xr-x 2 tinypilot tinypilot 4096 Sep 29 12:34 .vscode
                          -rw-r--r-- 1 tinypilot tinypilot 21 Sep 29 12:34 .yapfignore
                          pilot@0200707E:/opt/tinypilot $

                          Reply
                          Reply (discussion)Add progress note