Reverse Proxy Settings?
Hi Michael,
I'm in awe of what you've built here. Reading your development journey and then building products and a company around what is an amazing solution is really inspiring! I hope it all goes well for you!
Also thank you for keeping a free version going for those of us who don't need a commercial solution. The installation process was a technical marvel - I'm going to dive into your Ansible playbook and learn from it! I recently created a playbook to automate a process at work that interacts with domain controllers and storage arrays making my life much easier as a result, so I'm hoping to learn more from what you created. :)
Anyway, enough rambling and onto my question!
I have a reverse proxy set up that uses Traefik and Letsencrypt, so that I can externally access my internal applications via https. (pretty much based on this article: https://www.smarthomebeginner.com/traefik-2-docker-tutorial/
When I access Tinypilot via my external URL (https://tinyp.mydomain.com) it opens the web interface but I'm unable to control the mouse or keyboard (it says Disconnected, bottom left). Local connections work fine.
Do I need to configure any particular settings within Tinypilot to make it work? I saw here (Tiny Pilot Pro Http Server, Caddy Http Reverse Proxy, and Logging into TinyPilot within an iFrame) someone suggested changing the Nginx conf file, but that didn't work for me unfortunately.
In the file though I see that "location /stream" is trying to redirect to "proxy_pass http://ustreamer;" which I guess won't work through my proxy. Do you know if it's possible to work around this by any chance please?
The logs give the following errors:
2021/11/07 20:28:00 [error] 590#590: *719 upstream prematurely closed connection while reading response header from upstream, client: <Internal Traefik Reverse Proxy IP>, server: tinypilot, request: "GET /socket.io/?EIO=4&transport=polling&t=Npy5V7E&sid=AioAqKYehRXeOj7FAAEc HTTP/1.1", upstream: "http://127.0.0.1:8000/socket.io/?EIO=4&transport=polling&t=Npy5V7E&sid=AioAqKYehRXeOj7FAAEc", host: "tinyp.mydomain.com", referrer: "https://tinyp.mydomain.com/"
Thanks.
- Michael Lynch @michael2021-11-08 18:09:32.052Z
Thanks for the kind words!
It looks like Traefik is causing the websockets handshake to fail on the
/socket.io
endpoint. I'm not familiar with the Traefik proxy, so I unfortunately can't say what the specific issue is.In the file though I see that "location /stream" is trying to redirect to "proxy_pass http://ustreamer;" which I guess won't work through my proxy. Do you know if it's possible to work around this by any chance please?
In theory, this should work fine. Chaining proxies together in series works as long as the configuration is correct. What should happen is:
[browser]--->[traefik proxy]--->[TinyPilot nginx]--->[TinyPilot uStreamer]
So Traefik doens't need direct access to your uStreamer endpoint as long as it can forward traffic to your device's nginx interface.
This video by DBTech might be helpful, though he's using Nginx Proxy Manager instead of Traefik:
There are also a few other methods for cloud access on our FAQ.
Sidenote: do you have protections in place to authenticate you when you visit tinyp.yourdomain.com? I recommend putting protections in place to ensure that someone can't access your web interface just by scanning your IP or guessing a simple URL.
Could it be that I need a route to port 8000/8001 for uStreamer?
$ sudo netstat -plnt Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 608/sshd tcp 0 0 127.0.0.1:8000 0.0.0.0:* LISTEN 598/python tcp 0 0 127.0.0.1:8001 0.0.0.0:* LISTEN 596/ustreamer tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 603/nginx: master p tcp6 0 0 :::22 :::* LISTEN 608/sshd tcp6 0 0 :::80 :::* LISTEN 603/nginx: master p
At the moment my reverse proxy works by forwarding the external https 443 connection to the internal port 80.
Sidenote: do you have protections in place to authenticate you when you visit tinyp.yourdomain.com?
Yes luckily I do, I use OAuth as a front-end login which is backed up by two-factor auth. Thanks for checking though!
EDIT: I'm making progress but I've run out of time today. What I've realised is that I need to make the stream URL work via reverse proxy because the following works fine internally: http://my.int.ip.address/stream.
As it happens I've just managed to make the stream work by using this direct URL https://tinyp.mydomain.com/stream and using the
rule: "Host(
tinyp.mydomain.com) && PathPrefix(
/stream)
Traefik rule, so I just need to work out how to combine the two... That's tomorrows job. :)(this issue gave me the idea to try the PathPrefix option: https://github.com/tiny-pilot/tinypilot/issues/403)
- Michael Lynch @michael2021-11-08 22:24:20.624Z
Glad to see the progress!
Could it be that I need a route to port 8000/8001 for uStreamer?
No, the traefik proxy only needs to route to the tinypilot's port 80.
8000 and 8001 are the ports that TinyPilot listens on internally. nginx listens for external connections on port 80 and routes them internally to the right backends (on localhost:8000 and localhost:8001). The traefik proxy should be routing all traffic to the TinyPilot's port 80.
Thank you for helping clarify my understanding on how TinyPilot uses the ports, it was very useful.
So, knowing this I've identified that the problem appears to be with controlling the mouse and keyboard via the web interface when connecting via the proxy. I've confirmed that uStreamer works fine on the default URL (i.e. not using /stream) by playing a video and seeing that it transmits the stream without issues when accessing via the proxy.
I wonder then what could be blocking the interactive element of the interface? I guess it has something to do with the "Disconnected" status because the two coincide.
Just in case anything stands out for you, here are the Middlewares I use that define what headers the proxy allows. I must admit I don't understand most of what it's doing, I just copied the suggestions from the https://www.smarthomebeginner.com/traefik-2-docker-tutorial/ guide. :)
middlewares-secure-headers: headers: accessControlAllowMethods: - GET - OPTIONS - PUT accessControlMaxAge: 100 hostsProxyHeaders: - "X-Forwarded-Host" sslRedirect: true stsSeconds: 63072000 stsIncludeSubdomains: true stsPreload: true forceSTSHeader: true # # frameDeny: true #overwritten by customFrameOptionsValue customFrameOptionsValue: "allow-from https:mydomain.com" #CSP takes care of this but may be needed for organizr. contentTypeNosniff: true browserXssFilter: true # # sslForceHost: true # add sslHost to all of the services # # sslHost: "example.com" referrerPolicy: "same-origin" # # Setting contentSecurityPolicy is more secure but it can break things. Proper auth will reduce the risk. # # the below line also breaks some apps due to 'none' - sonarr, radarr, etc. # # contentSecurityPolicy: "frame-ancestors '*.example.com:*';object-src 'none';script-src 'none';" # # featurePolicy: "camera 'none'; geolocation 'none'; microphone 'none'; payment 'none'; usb 'none'; vr 'none';" customResponseHeaders: X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex," server: ""
- Michael Lynch @michael2021-11-09 17:27:31.468Z
It sounds like the websockets connection is failing.
The "disconnected" status in the corner is tracking the status of the websockets connection between the browser and the TinyPilot device. TinyPilot receives mouse events and keystrokes through the websockets connection, so that's why those are failing.
Nothing jumps out to me from your configuration, but you may need to add something to allow websockets to work through the Traefik proxy. The websockets connection goes through the
/socket.io
route. In nginx, we have to add special settings for that route to make the websockets connection work.I'm going to have to park this one for now because almost 4 hours of Googling and testing various combinations of Traefik, websockets, socket.io, etc with various suggestions has left me stumped.
I'll post my Traefik setup in case someone in the future is able to figure it out. For now I'll just RDP to an internal machine via Guacamole and connect to TinyPilot via its web browser.
I'll keep having a Google every so often because I won't be able to drop this until it's figured out. :)
docker-compose.yml
version: "3.8" services: traefik: container_name: traefik image: traefik restart: unless-stopped command: # CLI arguments - --global.checkNewVersion=true - --global.sendAnonymousUsage=true - --entryPoints.http.address=:80 - --entryPoints.https.address=:443 # Allow these IPs to set the X-Forwarded-* headers - Cloudflare IPs: https://www.cloudflare.com/ips/ - --entrypoints.https.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22 - --entryPoints.traefik.address=:8080 - --api=true - --api.dashboard=true - --log=true - --log.level=INFO # (Default: error) DEBUG, INFO, WARN, ERROR, FATAL, PANIC - --accessLog=true - --accessLog.filePath=/traefik.log - --accessLog.bufferingSize=100 # Configuring a buffer of 100 lines - --accessLog.filters.statusCodes=400-499 - --providers.docker=true - --providers.docker.endpoint=tcp://socket-proxy:2375 - --providers.docker.exposedByDefault=false - --entrypoints.https.http.tls.options=tls-opts@file - --entrypoints.https.http.tls.certresolver=dns-cloudflare - --entrypoints.https.http.tls.domains[0].main=$DOMAINNAME # Pulls main cert for second domain - --entrypoints.https.http.tls.domains[0].sans=*.$DOMAINNAME # Pulls wildcard cert for second domain - --providers.docker.network=t2_proxy - --providers.docker.swarmMode=false - --providers.file.directory=/rules # Load dynamic configuration from one or more .toml or .yml files in a directory - --providers.file.watch=true # Only works on top level files in the rules folder - --certificatesResolvers.dns-cloudflare.acme.email=$CLOUDFLARE_EMAIL - --certificatesResolvers.dns-cloudflare.acme.storage=/acme.json - --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.provider=cloudflare - --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.resolvers=1.1.1.1:53,1.0.0.1:53 - --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.delayBeforeCheck=90 # To delay DNS check and reduce LE hitrate networks: t2_proxy: ipv4_address: 192.168.90.254 # You can specify a static IP socket_proxy: depends_on: - socket-proxy - oauth security_opt: - no-new-privileges:true ports: - target: 80 published: 80 protocol: tcp mode: host - target: 443 published: 443 protocol: tcp mode: host volumes: - $DOCKERDIR/traefik2/rules:/rules # file provider directory # - /var/run/docker.sock:/var/run/docker.sock:ro # Use Docker Socket Proxy instead for improved security - $DOCKERDIR/traefik2/acme/acme.json:/acme.json # cert location - you must touch this file and change permissions to 600 - $DOCKERDIR/traefik2/traefik.log:/traefik.log # for fail2ban - make sure to touch file before starting container - $DOCKERDIR/shared:/shared environment: - CF_API_EMAIL_FILE=/run/secrets/cloudflare_email - CF_API_KEY_FILE=/run/secrets/cloudflare_api_key secrets: - cloudflare_email - cloudflare_api_key labels: - "com.centurylinklabs.watchtower.enable=true" - "traefik.enable=true" # HTTP-to-HTTPS Redirect - "traefik.http.routers.http-catchall.entrypoints=http" - "traefik.http.routers.http-catchall.rule=HostRegexp(`{host:.+}`)" - "traefik.http.routers.http-catchall.middlewares=redirect-to-https" - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https" # - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=http" # HTTP Routers - "traefik.http.routers.traefik-rtr.entrypoints=https" - "traefik.http.routers.traefik-rtr.rule=Host(`traefik.$DOMAINNAME`)" - "traefik.http.routers.traefik-rtr.tls=true" ## Services - API - "traefik.http.routers.traefik-rtr.service=api@internal" ## Middlewares - "traefik.http.routers.traefik-rtr.middlewares=chain-oauth@file" # Including a container to show how the labels are used # Dozzle - Real-time Docker Log Viewer dozzle: image: amir20/dozzle:latest container_name: dozzle restart: unless-stopped networks: - t2_proxy - socket_proxy depends_on: - socket-proxy security_opt: - no-new-privileges:true ports: - "$DOZZLE_PORT:8080" environment: DOZZLE_LEVEL: info DOZZLE_TAILSIZE: 300 DOZZLE_FILTER: "status=running" # DOZZLE_FILTER: "label=log_me" # limits logs displayed to containers with this label DOCKER_HOST: tcp://socket-proxy:2375 # volumes: # - /var/run/docker.sock:/var/run/docker.sock # Use Docker Socket Proxy instead for improved security labels: - "com.centurylinklabs.watchtower.enable=true" - "traefik.enable=true" ## HTTP Routers - "traefik.http.routers.dozzle-rtr.tls=true" - "traefik.http.routers.dozzle-rtr.entrypoints=https" - "traefik.http.routers.dozzle-rtr.rule=Host(`logs.$DOMAINNAME`)" ## Middlewares - "traefik.http.routers.dozzle-rtr.middlewares=chain-oauth@file" ## HTTP Services - "traefik.http.routers.dozzle-rtr.service=dozzle-svc" - "traefik.http.services.dozzle-svc.loadbalancer.server.port=8080"
traefik/rules folder
tls-opts.ymltls: options: tls-opts: minVersion: VersionTLS12 cipherSuites: - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 - TLS_AES_128_GCM_SHA256 - TLS_AES_256_GCM_SHA384 - TLS_CHACHA20_POLY1305_SHA256 - TLS_FALLBACK_SCSV # Client is doing version fallback. See RFC 7507 curvePreferences: - CurveP521 - CurveP384 sniStrict: true
middleware-chains.yml
http: middlewares: chain-oauth: chain: middlewares: - middlewares-rate-limit - middlewares-https-redirect - middlewares-secure-headers - middlewares-oauth
middlewares.yml
http: middlewares: middlewares-rate-limit: rateLimit: average: 100 burst: 50 middlewares-https-redirect: redirectScheme: scheme: https middlewares-secure-headers: headers: accessControlAllowMethods: - GET - OPTIONS - PUT accessControlMaxAge: 100 hostsProxyHeaders: - "X-Forwarded-Host" # sslRedirect: true stsSeconds: 63072000 stsIncludeSubdomains: true stsPreload: true forceSTSHeader: true customFrameOptionsValue: "allow-from https:mydomain.com" #CSP takes care of this but may be needed for organizr. contentTypeNosniff: true browserXssFilter: true referrerPolicy: "same-origin" # added customRequestHeaders for TinyPilot as per some web suggestions, but either I have this in the wrong place or it doesn't work customRequestHeaders: X-Forwarded-Proto: "http". # tried https here too with no luck customResponseHeaders: X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex," server: "" middlewares-oauth: forwardAuth: address: "http://oauth:4181" # Make sure you have the OAuth service in docker-compose.yml trustForwardHeader: true authResponseHeaders: - "X-Forwarded-User"
app-tinyp.yml
http: routers: tinyp-rtr: rule: "Host(`tinyp.mydomain.com`)" entryPoints: - https middlewares: - chain-oauth service: tinyp-svc tls: {} services: tinyp-svc: loadBalancer: servers: - url: "http://192.168.0.50"
- Progress
- J@jarrah
Update notes - not expecting a response, just sharing what I've found so far.
Similar issue described here - https://github.com/traefik/traefik/issues/5533
Suggestion is to use the following in Traefik which I have included in middlewares.yml, but not sure if I have put it in the right place.
"traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
customRequestHeaders: X-Forwarded-Proto: https
A comment "There was a commit to the EngineIO repo recently that added a check against HTTP_X_FORWARDED_PROTO", with a commit ref further down, but that was for EngineIO v3.11.0 and it looks like TinyPilot uses v4.0.1