No internet connection
  1. Home
  2. Technical Support

502 Bad Gateway nginx/1.14.2

By Jeff Crews @jeffcrews
    2022-02-23 20:27:43.798Z

    Just got a new Voyager 2 new out of the box.
    When attempting to browse I get:
    502 Bad Gateway
    nginx/1.14.2

    What do I do here?

    Solved in post #11, click to view
    • 24 replies

    There are 24 replies. Estimated reading time: 23 minutes

    1. J
      Jeff Crews @jeffcrews
        2022-02-23 20:29:37.930Z

        SSH to the IP address I get connection refused from an IP on the same subnet.
        Is this system required to have Internet access?
        I powered it up a couple of times without network.
        Is this unit supposed to have a graceful shutdown procedure when I unplugged it?

        1. It should boot up without Internet, but it's possible that there's a bug in that process that's more likely to occur when there's no internet connection.

          SSH is disabled by default, but you can enable it by mounting the microSD on another computer and placing an empty file called ssh on the boot partition. Normally you'd be able to enable/disable SSH from the web interface, but obviously that's not helpful when you can't access the web interface.

          If you're able to add the file and SSH in, can you share the log output from this command?

          sudo journalctl -u cycle-tls-keys
          

          On first boot, TinyPilot generates TLS keys that are unique to your device and then enables the TinyPilot web server. It looks like something in that process failed or is consistently failing, so the logs will help us understand what's happening.

          The other option is to re-flash your microSD. If you don't have a microSD reader, we can ship you a replacement microSD.

          1. JJeff Crews @jeffcrews
              2022-02-23 21:42:35.410Z

              login as: pilot
              pilot@192.168.0.103's password:
              Linux tinypilot 5.10.63-v7l+ #1496 SMP Wed Dec 1 15:58:56 GMT 2021 armv7l

              The programs included with the Debian GNU/Linux system are free software;
              the exact distribution terms for each program are described in the
              individual files in /usr/share/doc/*/copyright.

              Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
              permitted by applicable law.

              Wi-Fi is currently blocked by rfkill.
              Use raspi-config to set the country before use.

              pilot@tinypilot:~ $ sudo journalctl -u cycle-tls-keys
              -- Logs begin at Thu 2021-12-02 03:17:02 GMT, end at Thu 2021-12-02 03:19:20 GMT
              -- No entries --
              pilot@tinypilot:~ $

          2. J
            In reply tojeffcrews:
            Jeff Crews @jeffcrews
              2022-02-23 21:42:55.701Z

              I have not ran raspi-config

              1. J
                In reply tojeffcrews:
                Jeff Crews @jeffcrews
                  2022-02-23 21:45:12.445Z

                  maybe this helps some in looking /var/log/nginx

                  
                  ::::::::::::::
                  error.log
                  ::::::::::::::
                  2021/12/02 02:42:38 [error] 1030#1030: *5 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.0.38, server: tinypilot, request: "GET / HTTP/2.0", u
                  pstream: "http://127.0.0.1:8000/", host: "tinypilot"
                  2021/12/02 02:42:43 [error] 1030#1030: *5 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.0.38, server: tinypilot, request: "GET / HTTP/2.0", u
                  pstream: "http://127.0.0.1:8000/", host: "tinypilot"
                  2021/12/02 02:45:21 [error] 1030#1030: *5 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.0.38, server: tinypilot, request: "GET / HTTP/2.0", u
                  pstream: "http://127.0.0.1:8000/", host: "tinypilot"
                  2021/12/02 02:46:25 [error] 1030#1030: *13 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.0.38, server: tinypilot, request: "GET / HTTP/2.0",
                  upstream: "http://127.0.0.1:8000/", host: "tinypilot.local"
                  2021/12/02 02:50:25 [error] 1031#1031: *16 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.0.38, server: tinypilot, request: "GET / HTTP/1.1",
                  upstream: "http://127.0.0.1:8000/", host: "192.168.0.103"
                  2021/12/02 03:18:49 [error] 664#664: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.0.38, server: tinypilot, request: "GET / HTTP/1.1", ups
                  tream: "http://127.0.0.1:8000/", host: "192.168.0.103"
                  
                  1. JJeff Crews @jeffcrews
                      2022-02-23 21:48:41.737Z

                      from /var/log/syslog

                      Dec  2 03:25:21 tinypilot systemd[1]: tinypilot.service: Main process exited, code=exited, status=1/FAILURE
                      Dec  2 03:25:21 tinypilot systemd[1]: tinypilot.service: Failed with result 'exit-code'.
                      Dec  2 03:25:21 tinypilot systemd[1]: tinypilot.service: Service RestartSec=100ms expired, scheduling restart.
                      Dec  2 03:25:21 tinypilot systemd[1]: tinypilot.service: Scheduled restart job, restart counter is at 307.
                      Dec  2 03:25:21 tinypilot systemd[1]: Stopped TinyPilot - RPi-based virtual KVM.
                      Dec  2 03:25:21 tinypilot systemd[1]: Started TinyPilot - RPi-based virtual KVM.
                      Dec  2 03:25:22 tinypilot python[1155]: 2021-12-02 03:25:22.770 __main__        INFO Starting app
                      Dec  2 03:25:22 tinypilot python[1155]: Traceback (most recent call last):
                      Dec  2 03:25:22 tinypilot python[1155]:   File "app/main.py", line 44, in <module>
                      Dec  2 03:25:22 tinypilot python[1155]:     SECRET_KEY=secret_key.get_or_create(),
                      Dec  2 03:25:22 tinypilot python[1155]:   File "/opt/tinypilot/app/secret_key.py", line 101, in get_or_create
                      Dec  2 03:25:22 tinypilot python[1155]:     return _get()
                      Dec  2 03:25:22 tinypilot python[1155]:   File "/opt/tinypilot/app/secret_key.py", line 58, in _get
                      Dec  2 03:25:22 tinypilot python[1155]:     'The secret key value must be a string of 32 bytes.')
                      Dec  2 03:25:22 tinypilot python[1155]: secret_key.InvalidSecretKeyError: The secret key value must be a string of 32 bytes.
                      Dec  2 03:25:22 tinypilot systemd[1]: tinypilot.service: Main process exited, code=exited, status=1/FAILURE
                      Dec  2 03:25:22 tinypilot systemd[1]: tinypilot.service: Failed with result 'exit-code'.
                      Dec  2 03:25:23 tinypilot systemd[1]: tinypilot.service: Service RestartSec=100ms expired, scheduling restart.
                      Dec  2 03:25:23 tinypilot systemd[1]: tinypilot.service: Scheduled restart job, restart counter is at 308.
                      Dec  2 03:25:23 tinypilot systemd[1]: Stopped TinyPilot - RPi-based virtual KVM.
                      Dec  2 03:25:23 tinypilot systemd[1]: Started TinyPilot - RPi-based virtual KVM.
                      Dec  2 03:25:24 tinypilot python[1157]: 2021-12-02 03:25:24.267 __main__        INFO Starting app
                      Dec  2 03:25:24 tinypilot python[1157]: Traceback (most recent call last):
                      Dec  2 03:25:24 tinypilot python[1157]:   File "app/main.py", line 44, in <module>
                      Dec  2 03:25:24 tinypilot python[1157]:     SECRET_KEY=secret_key.get_or_create(),
                      Dec  2 03:25:24 tinypilot python[1157]:   File "/opt/tinypilot/app/secret_key.py", line 101, in get_or_create
                      Dec  2 03:25:24 tinypilot python[1157]:     return _get()
                      Dec  2 03:25:24 tinypilot python[1157]:   File "/opt/tinypilot/app/secret_key.py", line 58, in _get
                      Dec  2 03:25:24 tinypilot python[1157]:     'The secret key value must be a string of 32 bytes.')
                      Dec  2 03:25:24 tinypilot python[1157]: secret_key.InvalidSecretKeyError: The secret key value must be a string of 32 bytes.
                      Dec  2 03:25:24 tinypilot systemd[1]: tinypilot.service: Main process exited, code=exited, status=1/FAILURE
                      Dec  2 03:25:24 tinypilot systemd[1]: tinypilot.service: Failed with result 'exit-code'.
                      Dec  2 03:25:24 tinypilot systemd[1]: tinypilot.service: Service RestartSec=100ms expired, scheduling restart.
                      Dec  2 03:25:24 tinypilot systemd[1]: tinypilot.service: Scheduled restart job, restart counter is at 309.
                      Dec  2 03:25:24 tinypilot systemd[1]: Stopped TinyPilot - RPi-based virtual KVM.
                      Dec  2 03:25:24 tinypilot systemd[1]: Started TinyPilot - RPi-based virtual KVM.
                      
                      
                      
                      
                      
                      1. Oh! That's strange. Somehow your flask key got corrupted. Can you share the output of these commands?

                        sudo ls -l /home/tinypilot/.flask-secret-key && sudo xxd /home/tinypilot/.flask-secret-key
                        

                        (normally you wouldn't publish the secret key contents, but in this case, the file is corrupt, so it doesn't matter)

                        1. JJeff Crews @jeffcrews
                            2022-02-23 21:52:56.902Z

                            Funny, I was just hanging around in that folder:

                            
                            root@tinypilot:/home/tinypilot# ls -al
                            total 12
                            drwxr-xr-x 2 tinypilot tinypilot 4096 Dec  2 02:33 .
                            drwxr-xr-x 5 root      root      4096 Jan 10  2022 ..
                            -rw------- 1 tinypilot tinypilot    0 Dec  2 02:33 .flask-secret-key
                            -rw-r--r-- 1 tinypilot tinypilot  206 Jan 10  2022 settings.yml
                            root@tinypilot:/home/tinypilot# sudo ls -l /home/tinypilot/.flask-secret-key && sudo xxd /home/tinypilot/.flask-secret-key
                            -rw------- 1 tinypilot tinypilot 0 Dec  2 02:33 /home/tinypilot/.flask-secret-key
                            root@tinypilot:/home/tinypilot#
                            
                            
                            1. JJeff Crews @jeffcrews
                                2022-02-23 21:55:10.224Z

                                I guess I do not need sudo when I already went root so I could have more ability to inspect the files!

                                1. In reply tojeffcrews:
                                  Michael Lynch @michael2022-02-23 21:55:10.420Z2022-03-24 19:33:55.991Z

                                  Weird! Your flask secret key file is empty. It's possible that you powered it off at exactly the wrong time so that TinyPilot had created the key but hadn't populated it. I'll file a bug for that, as it's dumb behavior for TinyPilot to keep failing there.

                                  You should be able to fix it with this command

                                  sudo mv /home/tinypilot/.flask-secret-key /home/tinypilot/old-flask-secret-key && \
                                    sudo service tinypilot restart
                                  

                                  Update: This is now fixed in TinyPilot Pro 2.4.0 and TinyPilot Community 1.7.0.

                                  ReplySolution
                          • D
                            In reply tojeffcrews:
                            @Diederichs
                              2022-02-23 23:07:38.939Z

                              FYI: Just received my first Voyager 2 today. Same thing happened. The initial description is accurate to my observations. I will let you know if a solution works for me as well.

                              1. D@Diederichs
                                  2022-02-24 00:24:31.398Z

                                  The device wasn't showing up on my DHCP Server list (didn't know IP)... and I couldn't ssh to pilot@tinypilot. Shrug.
                                  I ended up downloading the image and re-writng the microSD. I followed the instructions on https://tinypilotkvm.com/faq/factory-reset. They are well written.

                                  Rebooted and the device worked.

                                • J
                                  In reply tojeffcrews:
                                  Jeff Crews @jeffcrews
                                    2022-02-24 00:29:48.647Z

                                    moving my comment from here to the end

                                    1. J
                                      In reply tojeffcrews:
                                      Jeff Crews @jeffcrews
                                        2022-02-24 00:31:23.728Z

                                        moving this comment to the end

                                        1. S
                                          In reply tojeffcrews:
                                          @seguse
                                            2022-10-23 23:17:54.210Z2022-10-24 14:13:54.961Z

                                            Same issue happened to me on two units. I didn't know about the "shutdown gracefully" until after reading this post. I, however, did have SSH access to both of my units and SSH was working. I was getting the same output about my flask secret key being empty. Since I had SSH access, I found this page to install from command line: [private URL removed]

                                            After running that command after being SSH'ed into my units, they came back up with no issues. I did order a SD card reader and some extra SD cards just in case I needed to do the re-flash. I also followed this guide here to enable the read-only filesystem: https://tinypilotkvm.com/faq/read-only-filesystem

                                            I've had issues where the TinyPilot would just lock up and I couldn't gracefully shutdown so I had to flip the power switch.

                                            1. David @david2022-10-24 11:23:26.396Z

                                              Hi @seguse, I'm sorry that you ran into this issue. And thanks for sharing the info about your experience with it!

                                              I've had issues where the TinyPilot would just lock up and I couldn't gracefully shutdown so I had to flip the power switch.

                                              If you have enabled the read-only filesystem, you should be better protected from corrupting the filesystem if you can't gracefully shut-down (as per one of Michael's comments in this thread). But it sounds like you are setup to re-flash the SD cards if needed in the future, so it seems like you're prepared just in case.

                                              Hope that helps! Please let me know if you have any questions or issues in the future.

                                            2. P
                                              In reply tojeffcrews:
                                              Peter Quigley @pquigley
                                                2023-04-05 13:09:34.523Z

                                                Hi @michael. I have a similar problem as Jeff did here. I updated it from a very old previous version (v0.0.1, actually) to current and then this error occurred. When I try to start the service, I get the errors about the flask secret key, but it seems like it's struggling to find the directory, even though it exists and permissions are correct. I am still able to ssh into it, but will take it in from our remote site today to flash the microSD. Would you mind sending an image to me to reflash it?

                                                Log: https://logs.tinypilotkvm.com/SFlLasQR

                                                1. Sure, I've just sent you a download link via email.

                                                2. Progress
                                                  with handling this problem
                                                3. J
                                                  Jeff Crews @jeffcrews
                                                    2022-02-23 22:16:15.734Z

                                                    I am working now! This is pretty nice.
                                                    Do you recommend we always shutdown gracefully using the GUI? Or can we just unplug it?
                                                    Does this device need Internet?
                                                    How is this able to be accessed via a hostname like https://tinypilot without an entry in my recursive DNS servers?

                                                    1. Michael Lynch @michael2022-02-23 22:42:47.562Zreplies tojeffcrews:

                                                      Do you recommend we always shutdown gracefully using the GUI? Or can we just unplug it?

                                                      Yeah, microSD is unfortunately not very robust against power cuts, so it's best to shut down gracefully through the GUI.

                                                      If you want to be able to cut power without a graceful shutdown, I recommend enabling the read-only filesystem overlay. That makes it so that disk writes stay in memory instead of getting flushed to disk, so it's much harder to corrupt the microSD. If you ever do corrupt the microSD, you can reflash it.

                                                      Does this device need Internet?

                                                      It only needs Internet for system updates, but otherwise it works fine without Internet.

                                                      How is this able to be accessed via a hostname like https://tinypilot without an entry in my recursive DNS servers?

                                                      Sorcery!

                                                      No, just kidding. You're likely using a DHCP server, and so your system is querying the DHCP server to find out which host is using the hostname tinypilot, so that's how it's resolving based on hostname. It should work with any host on your local network.

                                                      1. J
                                                        Jeff Crews @jeffcrews
                                                          2022-02-23 23:18:42.425Z

                                                          My workstation uses Google Public DNS using a static IP. So it does not consult the DHCP server.
                                                          The TinyPilot is on DHCP at the moment.
                                                          I still do not understand how my workstation can resolve this hostname.
                                                          I would feel better knowing I should be able to reach this machine if I gave it a static IP address.
                                                          Do you have instructions for turning off the DHCP client and using a static IP?
                                                          I still do not understand the sorcery enough to trust this in an urgent situation.
                                                          I figure I could do this manually by editing config files but would prefer as close to a supported solution as possible.

                                                          1. J
                                                            Jeff Crews @jeffcrews
                                                              2022-02-23 23:21:49.892Z

                                                              I think with a static IP or a solid understanding of the hostname method I would set this up to be read only using your instructions so TinyPilot does not need a graceful shutdown.
                                                              Is there a release notes page I can subscribe to for email updates when there are new releases?
                                                              Is there a way to see what version of the TinyPilot software I have installed?

                                                              1. Michael Lynch @michael2022-02-23 23:37:27.179Zreplies tojeffcrews:

                                                                Do you have instructions for turning off the DHCP client and using a static IP?

                                                                Yes, this post explains how to set a static IP.

                                                                My workstation uses Google Public DNS using a static IP. So it does not consult the DHCP server.

                                                                Your system could still consult the DHCP server even if it uses a static IP and uses an external DNS server.

                                                                You can check conclusively how your system is getting the IP if you capture packets with Wireshark and try different hostnames in the browser (like tinypilot1) and see what queries go out. There are a few different network protocols your system could be using to resolve the hostname.

                                                                Is there a release notes page I can subscribe to for email updates when there are new releases?

                                                                Yes, you can find the release notes here and subscribe to the mailing list to find out when new releases come out.

                                                                Is there a way to see what version of the TinyPilot software I have installed?

                                                                The normal way is to go to System > Update, which will show you your version and the latest available version. But if you're running without Internet connectivity, then it will fail.

                                                                The other way is to go to System > Logs. You'll see the installed version (currently 2.3.2) next to "TinyPilot version" at the top of the log.

                                                                1. J
                                                                  Jeff Crews @jeffcrews
                                                                    2022-02-24 00:32:20.263Zreplies tomichael:

                                                                    testing this out for real work and found I somehow got where the keyboard no longer worked. I was getting each letter I typed coming up with a ^ in front of it.
                                                                    so I rebooted the TinyPilot using the GUI and confirmed it rebooted via SSH to show uptime and the keyboard still does not work. now I just get nothing I type being echoed back to me
                                                                    tinypilot shows keystrokes at bottom of the screen but my linux server console shows no typing...I am going to try unplugging and replugging the USB from my server I am working on to see if this shakes the keyboard back into service.
                                                                    that did not work to help the keyboard still getting mismatched keystrokes to the server I am trying to manage,...not sure if something is wonky in my browser...trying an incognito Chrome tab to access TinyPilot to see if it helps...and...that fixed it...keyboard is fine now.