No internet connection
  1. Home
  2. Technical Support

Reverse Proxy Settings?

By @jarrah
    2021-11-07 21:13:18.347Z

    Hi Michael,

    I'm in awe of what you've built here. Reading your development journey and then building products and a company around what is an amazing solution is really inspiring! I hope it all goes well for you!

    Also thank you for keeping a free version going for those of us who don't need a commercial solution. The installation process was a technical marvel - I'm going to dive into your Ansible playbook and learn from it! I recently created a playbook to automate a process at work that interacts with domain controllers and storage arrays making my life much easier as a result, so I'm hoping to learn more from what you created. :)

    Anyway, enough rambling and onto my question!

    I have a reverse proxy set up that uses Traefik and Letsencrypt, so that I can externally access my internal applications via https. (pretty much based on this article: https://www.smarthomebeginner.com/traefik-2-docker-tutorial/

    When I access Tinypilot via my external URL (https://tinyp.mydomain.com) it opens the web interface but I'm unable to control the mouse or keyboard (it says Disconnected, bottom left). Local connections work fine.

    Do I need to configure any particular settings within Tinypilot to make it work? I saw here (Tiny Pilot Pro Http Server, Caddy Http Reverse Proxy, and Logging into TinyPilot within an iFrame) someone suggested changing the Nginx conf file, but that didn't work for me unfortunately.

    In the file though I see that "location /stream" is trying to redirect to "proxy_pass http://ustreamer;" which I guess won't work through my proxy. Do you know if it's possible to work around this by any chance please?

    The logs give the following errors:

    2021/11/07 20:28:00 [error] 590#590: *719 upstream prematurely closed connection while reading response header from upstream, client: <Internal Traefik Reverse Proxy IP>, server: tinypilot, request: "GET /socket.io/?EIO=4&transport=polling&t=Npy5V7E&sid=AioAqKYehRXeOj7FAAEc HTTP/1.1", upstream: "http://127.0.0.1:8000/socket.io/?EIO=4&transport=polling&t=Npy5V7E&sid=AioAqKYehRXeOj7FAAEc", host: "tinyp.mydomain.com", referrer: "https://tinyp.mydomain.com/"
    
    

    Thanks.

    • 7 replies
    1. Thanks for the kind words!

      It looks like Traefik is causing the websockets handshake to fail on the /socket.io endpoint. I'm not familiar with the Traefik proxy, so I unfortunately can't say what the specific issue is.

      In the file though I see that "location /stream" is trying to redirect to "proxy_pass http://ustreamer;" which I guess won't work through my proxy. Do you know if it's possible to work around this by any chance please?

      In theory, this should work fine. Chaining proxies together in series works as long as the configuration is correct. What should happen is:

      [browser]--->[traefik proxy]--->[TinyPilot nginx]--->[TinyPilot uStreamer]
      

      So Traefik doens't need direct access to your uStreamer endpoint as long as it can forward traffic to your device's nginx interface.

      This video by DBTech might be helpful, though he's using Nginx Proxy Manager instead of Traefik:

      There are also a few other methods for cloud access on our FAQ.

      Sidenote: do you have protections in place to authenticate you when you visit tinyp.yourdomain.com? I recommend putting protections in place to ensure that someone can't access your web interface just by scanning your IP or guessing a simple URL.

      1. J@jarrah
          2021-11-08 21:35:39.350Z2021-11-08 22:08:39.612Z

          Could it be that I need a route to port 8000/8001 for uStreamer?

          $ sudo netstat -plnt
          Active Internet connections (only servers)
          Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
          tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      608/sshd
          tcp        0      0 127.0.0.1:8000          0.0.0.0:*               LISTEN      598/python
          tcp        0      0 127.0.0.1:8001          0.0.0.0:*               LISTEN      596/ustreamer
          tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      603/nginx: master p
          tcp6       0      0 :::22                   :::*                    LISTEN      608/sshd
          tcp6       0      0 :::80                   :::*                    LISTEN      603/nginx: master p
          

          At the moment my reverse proxy works by forwarding the external https 443 connection to the internal port 80.

          Sidenote: do you have protections in place to authenticate you when you visit tinyp.yourdomain.com?

          Yes luckily I do, I use OAuth as a front-end login which is backed up by two-factor auth. Thanks for checking though!

          EDIT: I'm making progress but I've run out of time today. What I've realised is that I need to make the stream URL work via reverse proxy because the following works fine internally: http://my.int.ip.address/stream.

          As it happens I've just managed to make the stream work by using this direct URL https://tinyp.mydomain.com/stream and using the rule: "Host(tinyp.mydomain.com) && PathPrefix(/stream) Traefik rule, so I just need to work out how to combine the two... That's tomorrows job. :)

          (this issue gave me the idea to try the PathPrefix option: https://github.com/tiny-pilot/tinypilot/issues/403)

          1. Glad to see the progress!

            Could it be that I need a route to port 8000/8001 for uStreamer?

            No, the traefik proxy only needs to route to the tinypilot's port 80.

            8000 and 8001 are the ports that TinyPilot listens on internally. nginx listens for external connections on port 80 and routes them internally to the right backends (on localhost:8000 and localhost:8001). The traefik proxy should be routing all traffic to the TinyPilot's port 80.

            1. J@jarrah
                2021-11-09 16:18:27.697Z

                Thank you for helping clarify my understanding on how TinyPilot uses the ports, it was very useful.

                So, knowing this I've identified that the problem appears to be with controlling the mouse and keyboard via the web interface when connecting via the proxy. I've confirmed that uStreamer works fine on the default URL (i.e. not using /stream) by playing a video and seeing that it transmits the stream without issues when accessing via the proxy.

                I wonder then what could be blocking the interactive element of the interface? I guess it has something to do with the "Disconnected" status because the two coincide.

                Just in case anything stands out for you, here are the Middlewares I use that define what headers the proxy allows. I must admit I don't understand most of what it's doing, I just copied the suggestions from the https://www.smarthomebeginner.com/traefik-2-docker-tutorial/ guide. :)

                    middlewares-secure-headers:
                      headers:
                        accessControlAllowMethods:
                          - GET
                          - OPTIONS
                          - PUT
                        accessControlMaxAge: 100
                        hostsProxyHeaders:
                          - "X-Forwarded-Host"
                        sslRedirect: true
                        stsSeconds: 63072000
                        stsIncludeSubdomains: true
                        stsPreload: true
                        forceSTSHeader: true
                        # # frameDeny: true #overwritten by customFrameOptionsValue
                        customFrameOptionsValue: "allow-from https:mydomain.com" #CSP takes care of this but may be needed for organizr.
                        contentTypeNosniff: true
                        browserXssFilter: true
                        # # sslForceHost: true # add sslHost to all of the services
                        # # sslHost: "example.com"
                        referrerPolicy: "same-origin"
                        # # Setting contentSecurityPolicy is more secure but it can break things. Proper auth will reduce the risk.
                        # # the below line also breaks some apps due to 'none' - sonarr, radarr, etc.
                        # # contentSecurityPolicy: "frame-ancestors '*.example.com:*';object-src 'none';script-src 'none';"
                        # # featurePolicy: "camera 'none'; geolocation 'none'; microphone 'none'; payment 'none'; usb 'none'; vr 'none';"
                        customResponseHeaders:
                          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex,"
                          server: ""
                
                1. It sounds like the websockets connection is failing.

                  The "disconnected" status in the corner is tracking the status of the websockets connection between the browser and the TinyPilot device. TinyPilot receives mouse events and keystrokes through the websockets connection, so that's why those are failing.

                  Nothing jumps out to me from your configuration, but you may need to add something to allow websockets to work through the Traefik proxy. The websockets connection goes through the /socket.io route. In nginx, we have to add special settings for that route to make the websockets connection work.

                  1. J@jarrah
                      2021-11-09 21:27:24.656Z2021-11-10 20:33:29.185Z

                      I'm going to have to park this one for now because almost 4 hours of Googling and testing various combinations of Traefik, websockets, socket.io, etc with various suggestions has left me stumped.

                      I'll post my Traefik setup in case someone in the future is able to figure it out. For now I'll just RDP to an internal machine via Guacamole and connect to TinyPilot via its web browser.

                      I'll keep having a Google every so often because I won't be able to drop this until it's figured out. :)

                      docker-compose.yml

                      version: "3.8"
                      services:
                        traefik:
                          container_name: traefik
                          image: traefik
                          restart: unless-stopped
                          command: # CLI arguments
                            - --global.checkNewVersion=true
                            - --global.sendAnonymousUsage=true
                            - --entryPoints.http.address=:80
                            - --entryPoints.https.address=:443
                              # Allow these IPs to set the X-Forwarded-* headers - Cloudflare IPs: https://www.cloudflare.com/ips/
                            - --entrypoints.https.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22
                            - --entryPoints.traefik.address=:8080
                            - --api=true
                            - --api.dashboard=true
                            - --log=true
                            - --log.level=INFO # (Default: error) DEBUG, INFO, WARN, ERROR, FATAL, PANIC
                            - --accessLog=true
                            - --accessLog.filePath=/traefik.log
                            - --accessLog.bufferingSize=100 # Configuring a buffer of 100 lines
                            - --accessLog.filters.statusCodes=400-499
                            - --providers.docker=true
                            - --providers.docker.endpoint=tcp://socket-proxy:2375
                            - --providers.docker.exposedByDefault=false
                            - --entrypoints.https.http.tls.options=tls-opts@file
                            - --entrypoints.https.http.tls.certresolver=dns-cloudflare
                            - --entrypoints.https.http.tls.domains[0].main=$DOMAINNAME # Pulls main cert for second domain
                            - --entrypoints.https.http.tls.domains[0].sans=*.$DOMAINNAME # Pulls wildcard cert for second domain
                            - --providers.docker.network=t2_proxy
                            - --providers.docker.swarmMode=false
                            - --providers.file.directory=/rules # Load dynamic configuration from one or more .toml or .yml files in a directory
                            - --providers.file.watch=true # Only works on top level files in the rules folder
                            - --certificatesResolvers.dns-cloudflare.acme.email=$CLOUDFLARE_EMAIL
                            - --certificatesResolvers.dns-cloudflare.acme.storage=/acme.json
                            - --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.provider=cloudflare
                            - --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.resolvers=1.1.1.1:53,1.0.0.1:53
                            - --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.delayBeforeCheck=90 # To delay DNS check and reduce LE hitrate
                          networks:
                            t2_proxy:
                              ipv4_address: 192.168.90.254 # You can specify a static IP
                            socket_proxy:
                          depends_on:
                            - socket-proxy
                            - oauth
                          security_opt:
                            - no-new-privileges:true
                          ports:
                            - target: 80
                              published: 80
                              protocol: tcp
                              mode: host
                            - target: 443
                              published: 443
                              protocol: tcp
                              mode: host
                          volumes:
                            - $DOCKERDIR/traefik2/rules:/rules # file provider directory
                            # - /var/run/docker.sock:/var/run/docker.sock:ro # Use Docker Socket Proxy instead for improved security
                            - $DOCKERDIR/traefik2/acme/acme.json:/acme.json # cert location - you must touch this file and change permissions to 600
                            - $DOCKERDIR/traefik2/traefik.log:/traefik.log # for fail2ban - make sure to touch file before starting container
                            - $DOCKERDIR/shared:/shared
                          environment:
                            - CF_API_EMAIL_FILE=/run/secrets/cloudflare_email
                            - CF_API_KEY_FILE=/run/secrets/cloudflare_api_key
                          secrets:
                            - cloudflare_email
                            - cloudflare_api_key
                          labels:
                            - "com.centurylinklabs.watchtower.enable=true"
                            - "traefik.enable=true"
                            # HTTP-to-HTTPS Redirect
                            - "traefik.http.routers.http-catchall.entrypoints=http"
                            - "traefik.http.routers.http-catchall.rule=HostRegexp(`{host:.+}`)"
                            - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
                            - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
                            # - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=http"
                            # HTTP Routers
                            - "traefik.http.routers.traefik-rtr.entrypoints=https"
                            - "traefik.http.routers.traefik-rtr.rule=Host(`traefik.$DOMAINNAME`)"
                            - "traefik.http.routers.traefik-rtr.tls=true"
                            ## Services - API
                            - "traefik.http.routers.traefik-rtr.service=api@internal"
                            ## Middlewares
                            - "traefik.http.routers.traefik-rtr.middlewares=chain-oauth@file"
                      
                        # Including a container to show how the labels are used
                        # Dozzle - Real-time Docker Log Viewer
                        dozzle:
                          image: amir20/dozzle:latest
                          container_name: dozzle
                          restart: unless-stopped
                          networks:
                            - t2_proxy
                            - socket_proxy
                          depends_on:
                            - socket-proxy
                          security_opt:
                            - no-new-privileges:true
                          ports:
                            - "$DOZZLE_PORT:8080"
                          environment:
                            DOZZLE_LEVEL: info
                            DOZZLE_TAILSIZE: 300
                            DOZZLE_FILTER: "status=running"
                            # DOZZLE_FILTER: "label=log_me" # limits logs displayed to containers with this label
                            DOCKER_HOST: tcp://socket-proxy:2375
                          # volumes:
                          #  - /var/run/docker.sock:/var/run/docker.sock # Use Docker Socket Proxy instead for improved security
                          labels:
                            - "com.centurylinklabs.watchtower.enable=true"
                            - "traefik.enable=true"
                            ## HTTP Routers
                            - "traefik.http.routers.dozzle-rtr.tls=true"
                            - "traefik.http.routers.dozzle-rtr.entrypoints=https"
                            - "traefik.http.routers.dozzle-rtr.rule=Host(`logs.$DOMAINNAME`)"
                            ## Middlewares
                            - "traefik.http.routers.dozzle-rtr.middlewares=chain-oauth@file"
                            ## HTTP Services
                            - "traefik.http.routers.dozzle-rtr.service=dozzle-svc"
                            - "traefik.http.services.dozzle-svc.loadbalancer.server.port=8080"
                      

                      traefik/rules folder
                      tls-opts.yml

                      tls:
                        options:
                          tls-opts:
                            minVersion: VersionTLS12
                            cipherSuites:
                              - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
                              - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
                              - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
                              - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
                              - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
                              - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
                              - TLS_AES_128_GCM_SHA256
                              - TLS_AES_256_GCM_SHA384
                              - TLS_CHACHA20_POLY1305_SHA256
                              - TLS_FALLBACK_SCSV # Client is doing version fallback. See RFC 7507
                            curvePreferences:
                              - CurveP521
                              - CurveP384
                            sniStrict: true
                      

                      middleware-chains.yml

                      http:
                        middlewares:
                          chain-oauth:
                            chain:
                              middlewares:
                                - middlewares-rate-limit
                                - middlewares-https-redirect
                                - middlewares-secure-headers
                                - middlewares-oauth
                      

                      middlewares.yml

                      http:
                        middlewares:
                      
                          middlewares-rate-limit:
                            rateLimit:
                              average: 100
                              burst: 50
                      
                          middlewares-https-redirect:
                            redirectScheme:
                              scheme: https
                      
                          middlewares-secure-headers:
                            headers:
                              accessControlAllowMethods:
                                - GET
                                - OPTIONS
                                - PUT
                              accessControlMaxAge: 100
                              hostsProxyHeaders:
                                - "X-Forwarded-Host"
                              # sslRedirect: true
                              stsSeconds: 63072000
                              stsIncludeSubdomains: true
                              stsPreload: true
                              forceSTSHeader: true
                              customFrameOptionsValue: "allow-from https:mydomain.com" #CSP takes care of this but may be needed for organizr.
                              contentTypeNosniff: true
                              browserXssFilter: true
                              referrerPolicy: "same-origin"
                              # added customRequestHeaders for TinyPilot as per some web suggestions, but either I have this in the wrong place or it doesn't work
                              customRequestHeaders:
                                X-Forwarded-Proto: "http". # tried https here too with no luck
                              customResponseHeaders:
                                X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex,"
                                server: ""
                      
                          middlewares-oauth:
                            forwardAuth:
                              address: "http://oauth:4181" # Make sure you have the OAuth service in docker-compose.yml
                              trustForwardHeader: true
                              authResponseHeaders:
                                - "X-Forwarded-User"
                      

                      app-tinyp.yml

                      http:
                        routers:
                          tinyp-rtr:
                            rule: "Host(`tinyp.mydomain.com`)" 
                            entryPoints:
                              - https
                            middlewares:
                              - chain-oauth
                            service: tinyp-svc
                            tls: {}
                      
                        services:
                          tinyp-svc:
                            loadBalancer:
                              servers:
                                - url: "http://192.168.0.50"
                      
            2. Progress
            3. J
              @jarrah
                2021-11-10 20:33:12.961Z

                Update notes - not expecting a response, just sharing what I've found so far.
                Similar issue described here - https://github.com/traefik/traefik/issues/5533
                Suggestion is to use the following in Traefik which I have included in middlewares.yml, but not sure if I have put it in the right place.
                "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"

                customRequestHeaders:
                  X-Forwarded-Proto: https
                

                A comment "There was a commit to the EngineIO repo recently that added a check against HTTP_X_FORWARDED_PROTO", with a commit ref further down, but that was for EngineIO v3.11.0 and it looks like TinyPilot uses v4.0.1