Falls off network ("Destination host unreachable")

2022-07-11 12:44:50.493Z

Hi,

my tinypilot (connected via WIFI, not ethernet) become unreachable via SSH every 72hrs and i found out that it is actually quite a common issue for rasp pi
https://forums.raspberrypi.com/viewtopic.php?t=64620

https://raspberrypi.stackexchange.com/questions/5098/falls-off-network-destination-host-unreachable

I have run the command below:
sudo journalctl --unit=dhcpcd | curl -F '_=<-' https://logs.tinypilotkvm.com

it returned
Jul 05 11:54:15 tinypilot systemd[1]: Starting dhcpcd on all interfaces...
Jul 05 11:54:15 tinypilot dhcpcd[458]: dev: loaded udev
Jul 05 11:54:16 tinypilot dhcpcd[458]: wlan0: starting wpa_supplicant
Jul 05 11:54:16 tinypilot dhcpcd-run-hooks[573]: wlan0: starting wpa_supplicant
Jul 05 11:54:16 tinypilot dhcpcd[458]: wlan0: connected to Access Point ' Jul 05 11:54:16 tinypilot dhcpcd[458]: eth0: waiting for carrier Jul 05 11:54:16 tinypilot dhcpcd[458]: wlan0: waiting for carrier Jul 05 11:54:22 tinypilot dhcpcd[458]: wlan0: carrier acquired Jul 05 11:54:22 tinypilot dhcpcd[458]: wlan0: connected to Access Point SINGTEL-E204_2.4G'
Jul 05 11:54:22 tinypilot dhcpcd[458]: DUID 00:01:00:01:29:dd:b5:d4:e4:5f:01:b0:b8:2d
Jul 05 11:54:22 tinypilot dhcpcd[458]: wlan0: IAID 01:b0:b8:2e
Jul 05 11:54:22 tinypilot dhcpcd[458]: wlan0: adding address fe80::fc43:ed36:5706:4b6
Jul 05 11:54:22 tinypilot dhcpcd[458]: wlan0: soliciting an IPv6 router
Jul 05 11:54:23 tinypilot dhcpcd[458]: wlan0: soliciting a DHCP lease
Jul 05 11:54:25 tinypilot dhcpcd[458]: wlan0: offered 192.168.1.74 from 192.168.1.254
Jul 05 11:54:25 tinypilot dhcpcd[458]: wlan0: probing address 192.168.1.74/24
Jul 05 11:54:30 tinypilot dhcpcd[458]: wlan0: leased 192.168.1.74 for 14400 seconds
Jul 05 11:54:30 tinypilot dhcpcd[458]: wlan0: adding route to 192.168.1.0/24
Jul 05 11:54:30 tinypilot dhcpcd[458]: wlan0: adding default route via 192.168.1.254
Jul 05 11:54:30 tinypilot dhcpcd[458]: forked to background, child pid 700
Jul 05 11:54:30 tinypilot systemd[1]: Started dhcpcd on all interfaces.
Jul 05 11:54:36 tinypilot dhcpcd[700]: wlan0: no IPv6 Routers available

can advise which config file is the correct file to amend? as seems like interfaces is deprecated

Please note that this file is written to be used with dhcpcd

For static IP, consult /etc/dhcpcd.conf and 'man dhcpcd.conf'

so my question now is how to resolve this issue as sudo reboot is not working and the only way to resolve this for now is to restart it (pull out the power). it really drastically impacted my overseas work

Reply

20 replies

There are 20 replies. Estimated reading time: 24 minutes

Diego @diego
2022-07-11 22:07:41.691Z
Hello @ord00223 - Thank you for posting your question here in the Forum.

I'm sorry you are having this issue. It sounds like a DHCP lease problem if it happens every 72 hours.

Have you tried to log into your router? There may be settings there you could adjust, like for example DHCP lease expiry time, or you could instruct your router to always give the same IP address to your TinyPilot.

If you want to setup a static IP address in your TinyPilot, we have a procedure for it. You can see it here. If you want to do it by hand, the file to edit is /etc/dhcpcd.conf but you need to reboot after editing the file. Additionally, I do not recommend you do this, as you may be locked out of your device and require a factory reset to fix it.

What error are you getting when you try to run sudo reboot - that's odd, something is broken there if you are unable to issue this command.
Reply
1. O @ord00223
  2022-07-12 00:31:36.651Z
  Hi Diego,
  
  Have you tried to log into your router? There may be settings there you could adjust, like for example DHCP lease expiry time, or you could instruct your router to always give the same IP address to your TinyPilot.
  -unfortunately i do not have the credentials to log in to the router
  
  If you want to setup a static IP address in your TinyPilot, we have a procedure for it. You can see it here. If you want to do it by hand, the file to edit is /etc/dhcpcd.conf but you need to reboot after editing the file. Additionally, I do not recommend you do this, as you may be locked out of your device and require a factory reset to fix it.
  
  i have tried follow the procedure to setup the static IP before but it didnt work. any cmd i can run to check if i have successfully setup the static IP?
  
  please show me the manual way to edit in dhcpcd.conf as well. thanks
  
  What error are you getting when you try to run sudo reboot - that's odd, something is broken there if you are unable to issue this command.
  
  no error. each time i run sudo reboot it goes back to the issue - falls off network destination host unreachable. the only way for me to reboot is to physically turn off and on the power
  Reply
  Diego @diego
  2022-07-13 01:10:22.248Z
  Hi @ord00223,
  
  It would be very helpful if you could get access to the login credentials of your router. Is there any way you can get this?
  
  For the static IP address procedure, please remember that if you have enabled the read only filesystem, then any changes will be lost upon reboot. You should disable read only filesystem, apply changes, then you can enable read only filesystem again. Also, the procedure in our website to setup static IP address is geared towards the ethernet port (eth0), and not the WIFI interface (wlan0). That may be another reason why it didn't work for you when you tried. Please find the manual procedure to set up a static IP address below (be aware that if this is missconfigured, you may lose complete access to your TinyPilot, and you will need to perform a factory reset to fix it):
  
  # Edit the file by hand, use nano or vi, whatever you're most comfortable with sudo nano /etc/dhcpcd.conf # Add the following lines at the end of the file (edit the IP addresses to your needs) interface wlan0 static ip_address=192.168.1.20 static routers=192.168.1.1 static domain_name_servers=192.168.1.1 8.8.8.8 1.1.1.1 # Reboot to apply your changes sudo reboot
  
  I'm not sure I fully understand what is going on with sudo reboot. I believe you are able to run the command, but then afterwards you are unable to connect to the TinyPilot and you therefore need to perform a power cycle in order to fix it. Would that describe the situation accurately? Have you tried connecting via straight IP address instead of via https://tinypilot?
  
  Reply
  O @ord00223
  2022-07-13 02:30:42.241Z
  Hi Diego,
  
  It would be very helpful if you could get access to the login credentials of your router. Is there any way you can get this?
  nope unfortunately that router was purchased by my landlord and she has no idea about the credentials as well.
  
  For the static IP address procedure, please remember that if you have enabled the read only filesystem, then any changes will be lost upon reboot. You should disable read only filesystem, apply changes, then you can enable read only filesystem again. Also, the procedure in our website to setup static IP address is geared towards the ethernet port (eth0), and not the WIFI interface (wlan0). That may be another reason why it didn't work for you when you tried. Please find the manual procedure to set up a static IP address below (be aware that if this is missconfigured, you may lose complete access to your TinyPilot, and you will need to perform a factory reset to fix it):
  I have disabled the read-only setting long time ago.
  the procedure only for eth0? that explains why it didnt work as I'm using wlan0, not eth0. i will try configure it to wlan
  any line to edit under fallback to static IP if DHCP fails part?
  
  I'm not sure I fully understand what is going on with sudo reboot. I believe you are able to run the command, but then afterwards you are unable to connect to the TinyPilot and you therefore need to perform a power cycle in order to fix it. Would that describe the situation accurately? Have you tried connecting via straight IP address instead of via https://tinypilot?
  yes that explained accurately. cut off power is the only way to resolve it. Both IP address and URL not working after the reboot cmd. it shows Reply from : Destination Host Unreachable
  
  Reply
  O @ord00223
  2022-07-13 06:18:46.254Z
  Hi Diego,
  
  the Destination host unreachable issue happened again, today :(
  
  i have set the static IP in dhcpcd.conf and reboot
  if i run cmd hostname -I, it shows the static IP now
  
  is it possible to set fallback profile for wlan0 too? i have googled search alot but all are set for eth0..
  
  Reply
  O @ord00223
  2022-07-13 06:27:02.638Z
  Hi Diego,
  
  sorry just an addon.
  since the error in log is ' wlan0: no IPv6 Routers available'
  do i need to set for static_ip6_address as well?
  
  Reply
  Diego @diego
  2022-07-13 23:57:34.475Z
  
  any line to edit under fallback to static IP if DHCP fails part?
  is it possible to set fallback profile for wlan0 too? i have googled search alot but all are set for eth0..
  
  Yes, I believe you can set fallback profile for wlan0, it is just another network interface after all. You can follow the example in /etc/dhcpcd.conf and adjust the interface name and IP addresses. See below the example found in /etc/dhcpcd.conf. However, I don't think this will have any effect if you are already configuring the interface to have a static IP address. I'm sorry to insist, but please be careful when playing with /etc/dhcpcd.conf; remember you might get locked out if this file is missconfigured and the only way to fix it will be to perform a factory reset.
  
  # It is possible to fall back to a static IP if DHCP fails: # define static profile #profile static_eth0 #static ip_address=192.168.1.23/24 #static routers=192.168.1.1 #static domain_name_servers=192.168.1.1 # fallback to static profile on eth0 #interface eth0 #fallback static_eth0
  
  since the error in log is ' wlan0: no IPv6 Routers available'
  do i need to set for static_ip6_address as well?
  
  No, I don't think you need to set that. The error in the log is informational. You can set it if you feel that will help, but I believe it is not necessary.
  
  yes that explained accurately. cut off power is the only way to resolve it. Both IP address and URL not working after the reboot cmd. it shows Reply from : Destination Host Unreachable
  
  This is strange. It would be great if you could inspect sudo journalctl and look at one of the times the issue happens, to see what's going on. It looks like wlan0 is not reconnecting properly after a reboot, but it connects OK after a power cycle. There shouldn't be any differences in both methods, as far as I can tell.
  
  Reply
  O @ord00223
  2022-07-14 10:48:58.446Z
  Hi DIego,
  
  thanks for the reply. I have set the static IP address via dhcpcd.conf.
  
  since you said that fallback ip and ipv6 is not mandatory then i will just leave it as it is.
  
  lets just observe for 3 more days. I really hope that this issue can be resolved. :(
  
  Reply
  Diego @diego
  2022-07-14 23:35:06.585Z
  Hi @ord00223 - Thanks for the update. Yeah, I also hope this issue to be resolved quickly for you. I understand your frustration so far.
  
  If this issue happens again, I would definately suggest (again) to look out for the router's login credentials. Also, trying out ethernet connection instead of WIFI might lead to a resolution.
  
  Reply
  O @ord00223
  2022-07-19 05:51:03.314Z
  Hi Diego,
  
  it was working for 5days without any issue since the day I set the static IP. which is good news. But Not until when I run sudo reboot command today. It lost connection again, same error Reply from IP address : Destination Host Unreachable.
  Again, only power cycle resolved the issue.
  
  I really think something is wrong with the reboot config....
  
  Reply
  In reply toord00223⬆:
  Diego @diego
  2022-07-20 20:24:58.283Z
  Hi @ord00223 - It looks like the static IP address configuration has solved your issue, at least partially (5 days working instead of 3 and only because the system was rebooted).
  
  About the reboot not working as expected... Could you inspect sudo journalctl and look at one of the times the issue happens (or look at the time you last rebooted), to see what's going on. It looks to me like wlan0 is not reconnecting properly after a reboot, but it connects OK after a power cycle. A close inspection to the mentioned logs might give us some clue.
  
  Reply
  O @ord00223
  2022-07-30 01:33:28.746Z
  Hi Diego,
  
  sorry for the late reply.
  the read-only setting was enabled so I don't think there is any useful message to be provided.
  
  the device is up for 10 days without any issue now.
  so I guess Static IP is the solution (without rebooting the device via command, of course).
  I am currently overseas so I guess I will try not to reboot the device or running any command for now.
  
  Thanks again for your all along support.
  
  Reply
  Diego @diego
  2022-08-02 20:24:34.315Z
  @ord00223 - Thank you for the update. I thought you said you had disabled the read only filesystem. My bad if I didn't understand that correctly. Let us know if you have further questions and/or comments, I'll be glad to answer them! Have a great day!
  
  Reply
  O @ord00223
  2022-08-03 00:38:24.108Z
  Hi Diego,
  
  yes it was disabled before, but then I enabled it back after my last reboot
  
  btw I saw a post from another user Roberto:
  I'm also running into stuck problems rebooting the tinypilot, which is a huge issue since I'm using it remotely, so I can't physically turn it off/on again. I guess those rebooting getting stuck is related to the "broken" filesystem.
  
  guess I am not the only user having issue with rebooting the device. Possible mine also having the corrupted filesystem? but anyway my overseas trip has been extended. hopefully the device can survive another 30days
  
  thanks again
  
  Reply
  O @ord00223
  2022-08-03 10:33:42.577Z2022-08-03 10:41:03.019Z
  Hi Diego,
  
  LOL the device got disconnected again now. right after 10hrs I posted the latest update lol. what an unlucky day
  
  but well at least it worked for 13days lol
  
  Reply
  Diego @diego
  2022-08-04 00:38:57.686Z
  @ord00223 - Yes it may be possible that the reason reboot fails is because of a corrupted root file system. But then, the system boots up fine after a power cycle. There is no way to know what's happening without a close inspection to the system logs. If you have further issues, and if you agree, please disable the read only file system, so that we can get access to the logs.
  
  Reply
  O @ord00223
  2022-08-08 01:01:54.607Z
  Hi Diego,
  
  not sure if this logs help.
  
  https://logs.tinypilotkvm.com/SQZRI0no
  it was disconnected again today.
  
  Reply
  Diego @diego
  2022-08-08 23:38:40.778Z
  Hi @ord00223 - Thank you for uploading those logs.
  
  So for what I can see, it looks like the TinyPilot is getting disconnected from the router and/or signal is lost.
  
  Aug 8 01:28:07 tinypilot dhcpcd[644]: wlan0: carrier lost Aug 8 01:28:07 tinypilot dhcpcd[644]: wlan0: deleting address fe80::fc43:ed36:5706:4b6 Aug 8 01:28:07 tinypilot dhcpcd[644]: wlan0: deleting route to 192.168.1.0/24 Aug 8 01:28:07 tinypilot dhcpcd[644]: wlan0: deleting default route via 192.168.1.254 Aug 8 01:44:13 tinypilot dhcpcd[644]: wlan0: carrier acquired Aug 8 01:44:13 tinypilot dhcpcd[644]: wlan0: IAID 01:b0:b8:2e Aug 8 01:44:13 tinypilot dhcpcd[644]: wlan0: adding address fe80::fc43:ed36:5706:4b6 Aug 8 01:44:13 tinypilot dhcpcd[644]: wlan0: probing address 192.168.1.74/24 Aug 8 01:44:13 tinypilot dhcpcd[644]: wlan0: soliciting an IPv6 router Aug 8 01:44:18 tinypilot dhcpcd[644]: wlan0: using static address 192.168.1.74/24 Aug 8 01:44:18 tinypilot dhcpcd[644]: wlan0: adding route to 192.168.1.0/24 Aug 8 01:44:18 tinypilot dhcpcd[644]: wlan0: adding default route via 192.168.1.254 Aug 8 01:44:27 tinypilot dhcpcd[644]: wlan0: no IPv6 Routers available
  
  Please note how at 01:28:07 carrier is lost, and then it is acquired again at 01:44:13.
  
  Have you noticed any pattern in the times the TinyPilot gets disconnected? This time it happened between 1 and 2 am. Does it always happen around this time (during the night)?
  
  Is the WIFI signal strength good at all times? I believe the message is pretty descriptive: "carrier lost", it means no WIFI signal. Therefore dhcpd proceeds to delete routes for the interface affected (wlan0). Then after "carrier acquired" again, it configures the routes again. It is odd to see that it takes 26 minutes in between "carrier lost" and "carrier acquired". Is the router being rebooted during these minutes?
  
  Do you recall at what time you powercycled the TinyPilot to recover from this "link down/up" situation? I am not able to see that in the logs. I'm still baffled you have to powercycle the unit after this, the logs show dhcpd correctly configuring routes for the interface after "carrier acquired".
  
  I'm thinking perhaps this otherwise may be caused by your router disconnecting the TinyPilot after a certain amount without any traffic. If you want to try, this page contains a script to be added to cron to ping the router's IP address every couple of minutes, to keep traffic alive. I'm not sure this will solve your problem, but it is worth to try, in my humble opinion.
  
  Reply
O
In reply toord00223⬆:
@ord00223
2022-08-09 04:03:16.592Z
Hi Diego,

here's the latest log for today. it got disconnected again. I guess static IP is not the solution
https://logs.tinypilotkvm.com/BcWX205J

Have you noticed any pattern in the times the TinyPilot gets disconnected? This time it happened between 1 and 2 am. Does it always happen around this time (during the night)?

I am in +8 timezone country. so it should be around 8 - 9am here. I not sure what time the device got disconnected as I just leave it idle after my work is done. normally is after 6 - 7pm. Then i just remote in my laptop again around 8 -9 am the next day. this is the time I realized it was disconnected

Is the WIFI signal strength good at all times? I believe the message is pretty descriptive: "carrier lost", it means no WIFI signal. Therefore dhcpd proceeds to delete routes for the interface affected (wlan0). Then after "carrier acquired" again, it configures the routes again. It is odd to see that it takes 26 minutes in between "carrier lost" and "carrier acquired". Is the router being rebooted during these minutes?
-the router is just outside my room so the signal is excellent. no one is at my home so impossible the router was being rebooted

Do you recall at what time you powercycled the TinyPilot to recover from this "link down/up" situation? I am not able to see that in the logs. I'm still baffled you have to powercycle the unit after this, the logs show dhcpd correctly configuring routes for the interface after "carrier acquired".

cant recall what time but I did ask my neighbor to powercycle the device yesterday after found it was disconnected. maybe around 8 - 9am also?

I'm thinking perhaps this otherwise may be caused by your router disconnecting the TinyPilot after a certain amount without any traffic. If you want to try, this page contains a script to be added to cron to ping the router's IP address every couple of minutes, to keep traffic alive. I'm not sure this will solve your problem, but it is worth to try, in my humble opinion.

I realized last time I setup a zoom call for 13days (20hrs per day) due to my project deployment, maybe thats the reason why the device was up for 13days without any issue as there is traffic (?).

sure I have added the script to the cronjob. lets hope it keeps the traffic alive. thanks for the page
Reply
1. Diego @diego
  2022-08-10 23:07:52.777Z
  @ord00223:
  
  sure I have added the script to the cronjob. lets hope it keeps the traffic alive. thanks for the page
  
  You are welcome! Hopefully that script helps!
  Reply

ReplyAdd progress note