Troubleshooting
Known problems
Default G/W for Igor
Igor can't find it's local network:
hxd@morgana ~/net> ssh -J igor.lan 192.168.50.159
channel 0: open failed: connect failed: open failed
stdio forwarding failed
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535Use web UI to access console and exec as root:
ip route add default via 192.168.100.1Troubleshooting
Default G/W for Igor
Igor can't find it's local network:
hxd@morgana ~/net> ssh -J igor.lan 192.168.50.159
channel 0: open failed: connect failed: open failed
stdio forwarding failed
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535Use web UI to access console and exec as root:
ip route add default via 192.168.100.1TCP dump
Installing on OpenWRT:
opkg update
opkg install tcpdumpDHCP
tcpdump -vvv -i any udp port 67 and port 68Network Issues
Wi-Fi slow
Slow access to HOME from laptop on MGMT wi-fi.
UPDATE: 2023-10-28
Looks like laptop gets very low Rx rate (throughput from haru to Morgana) of ~17Mbit or even 6Mbit:
iperf3 -c 192.168.100.20 -p 2345 -R -t 9999
Connecting to host 192.168.100.20, port 2345
Reverse mode, remote host 192.168.100.20 is sending
[ 5] local 192.168.100.161 port 42966 connected to 192.168.100.20 port 2345
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.75 MBytes 14.7 Mbits/sec
[ 5] 1.00-2.00 sec 1.64 MBytes 13.8 Mbits/sec
[ 5] 2.00-3.00 sec 1.64 MBytes 13.8 Mbits/sec
[ 5] 3.00-4.00 sec 1.55 MBytes 13.0 Mbits/sec
[ 5] 4.00-5.00 sec 1.62 MBytes 13.6 Mbits/sec
[ 5] 5.00-6.00 sec 1.63 MBytes 13.6 Mbits/sec
^C[ 5] 6.00-6.14 sec 252 KBytes 14.4 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-6.14 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-6.14 sec 10.1 MBytes 13.8 Mbits/sec receiver
At the same time I get like 60Mbit sending data out:
iperf3 -c 192.168.100.20 -p 2345 -t 9999
Connecting to host 192.168.100.20, port 2345
[ 5] local 192.168.100.161 port 45200 connected to 192.168.100.20 port 2345
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 10.1 MBytes 85.0 Mbits/sec 0 469 KBytes
[ 5] 1.00-2.00 sec 6.71 MBytes 56.3 Mbits/sec 0 469 KBytes
[ 5] 2.00-3.00 sec 7.08 MBytes 59.4 Mbits/sec 0 502 KBytes
[ 5] 3.00-4.00 sec 6.40 MBytes 53.7 Mbits/sec 0 529 KBytes
[ 5] 4.00-5.00 sec 7.77 MBytes 65.2 Mbits/sec 0 529 KBytes
[ 5] 5.00-6.00 sec 7.77 MBytes 65.2 Mbits/sec 0 529 KBytes
[ 5] 6.00-7.00 sec 7.01 MBytes 58.8 Mbits/sec 0 587 KBytes
[ 5] 7.00-8.00 sec 8.47 MBytes 71.0 Mbits/sec 0 621 KBytes
^C[ 5] 8.00-8.40 sec 2.41 MBytes 50.4 Mbits/sec 0 621 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-8.40 sec 63.8 MBytes 63.7 Mbits/sec 0 sender
[ 5] 0.00-8.40 sec 0.00 Bytes 0.00 bits/sec receiverHaru reports low MCS values of 10 or lower.
| Server |
SSID |
Channel |
Client |
Location |
Client -> Server Mbit |
Server <- Client Mbit |
|
| Justine | Haru Legacy |
6 |
L (android) |
living room | 25 |
25 |
|
| Justine | Haru Legacy | 6 |
futaba |
living room | 21 |
9 |
|
| Justine | Haru Legacy | 6 |
futaba |
kitchen |
25 |
10 |
|
| Justine | Haru Legacy | 6 |
futaba |
by Haru |
26 |
11 |
|
| Justine | Haru Legacy | 11 |
futaba | living room | 26 |
11 |
|
| Justine | Haru MGMT | 11 |
morgana | living room | 53 |
23 |
|
| Justine | Haru Legacy | 6 |
morgana |
living room | 50 |
11 |
|
| Justine | Haru |
? |
morgana | living room | 89 |
72 | |
| Justine | Haru (IE) |
52 | morgana | living room | 95 |
95 |
|
| Justine | Haru MGMT (IE) |
11 | morgana | living room | 92 |
92 |
|
| Justine | Haru MGMT (IE) |
11 | morgana | kitchen | 95 |
94 |
|
| Justine |
Haru (IE/1G) |
11 |
morgana | kitchen | 53 |
321 |
*1 |
| Justine |
Haru MTMT (IE/1G) |
52 |
morgana |
kitchen |
44 |
92 |
*1 |
*1 - both washing and drying going on
Config changes:
- Set max power to 30dBm
- channel switched to 5 (was 6): 15-25 on morgana and 13 on futaba
- channel 11: 10 to 25 on moragna, 16 on futaba
- channel switched to 5 (was 6): 15-25 on morgana and 13 on futaba
- Setting region to IE(!):
- 95/95 morgana -> justine over Haru channel 52
- BINGO!: 92/92 on MGMT channel 11
- This get me to 100Mbit; but I get much higher rates reported for wifi like 866/780 Mbit on Haru
- eth0 on Speed: 100Mb/s!
- Looks like faulty cable but I also rebooted Haru so could be that as well, although on secondary port I was getting 1Gbit with laptop before restart.
- goro reports only lan8 (haru) at 100Mbit/s but others at 1Gbit
Mon Oct 30 19:30:27 2023 kern.info kernel: [8584550.111467] RTL8380 Link change: status: 1, ports 8000 Mon Oct 30 19:30:28 2023 kern.info kernel: [8584550.992936] rtl83xx-switch switch@1b000000 lan8: Link is Up - 100Mbps/Full - flow control rx/tx Mon Oct 30 19:30:28 2023 kern.info kernel: [8584551.003123] switch: port 8(lan8) entered blocking state Mon Oct 30 19:30:28 2023 kern.info kernel: [8584551.009306] switch: port 8(lan8) entered forwarding state Mon Oct 30 19:30:28 2023 daemon.notice netifd: Network device 'lan8' link is up - I have replaced the cable and now have 1Gbps
Mon Oct 30 20:32:29 2023 kern.info kernel: [8588272.090512] rtl83xx-switch switch@1b000000 lan8: Link is Up - 1Gbps/Full - flow control rx/tx
- Looks like faulty cable but I also rebooted Haru so could be that as well, although on secondary port I was getting 1Gbit with laptop before restart.
UPDATE: 2023-11-05
There was a packet drop between Haru and Goro. I have replaced the cable that goes from Goro to power supply (for Haru).
After replacing the cable link dropped to 100Mbps, reconnecting it got me 1Gbit.
I have also switched channels for 5GHz radio to use:
- channel: 104 (5520MHz)
- width: 160MHz
Now everything is very fast. In kitchen I got 356/407 Mbps using Haru!
UPDATE: 2023-11-11
After power issue the network went up in bad state.
I was getting 300Mbit one way and only 46Mbit and high packet loss the other way.
I decided to route another cable to Haru.
Looks like Primary interface is the only one that can take power so I left it connected to the power supply but disconnected it from the swich.
New, longer cable now connects switch to Haru Secondary port which is part of a bridge setup with Primary port so no configuration changes was needed.
Now I am getting up from Morgana in kitchen 657Mbit and down 334Mbit from Justine.
DHCP no responses, no IP assigned
Looks like Graphene OS uses random MAC for every connection attempt to Haru:
13:34:45.185659 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from b2:7f:5d:03:bf:61, length 288
13:34:46.144502 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from b2:7f:5d:03:bf:61, length 288
13:34:48.135488 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from b2:7f:5d:03:bf:61, length 288
13:34:52.237864 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from b2:7f:5d:03:bf:61, length 288
13:35:00.303513 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from b2:7f:5d:03:bf:61, length 288
13:35:10.015298 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 8a:ba:ab:74:33:23, length 288
13:35:11.105965 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 8a:ba:ab:74:33:23, length 288
13:35:13.292155 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 8a:ba:ab:74:33:23, length 288
13:35:17.375826 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 8a:ba:ab:74:33:23, length 288
13:35:24.972812 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 8a:ba:ab:74:33:23, length 288
13:43:05.585122 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 5a:88:7a:60:55:fd, length 288
13:43:06.607297 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 5a:88:7a:60:55:fd, length 288
13:43:08.473318 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 5a:88:7a:60:55:fd, length 288
13:43:12.547523 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 5a:88:7a:60:55:fd, length 288
13:43:21.310413 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 5a:88:7a:60:55:fd, length 288
13:43:27.942393 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from ca:9a:d3:13:a3:1c, length 288
13:43:28.860544 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from ca:9a:d3:13:a3:1c, length 288
13:43:30.774370 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from ca:9a:d3:13:a3:1c, length 288
13:43:34.697002 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from ca:9a:d3:13:a3:1c, length 288
13:43:43.292732 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from ca:9a:d3:13:a3:1c, length 288This may lead to depletion of IP addresses (pool has up to 150 to allocate).
To clean up the pool SSH to Caroline:
service dnsmasq stop
mv /tmp/dhcp.leases /tmp/dhcp.leases.bac
service dnsmasq startTo mitigate the issue I have reduced leas time from 30 days to 24 hours.
VPN clients coming from outside are NAT'ed
They will looks like justine, not their actual VPN IP, since devices can use caroline as their default G/W.
No access to SERVER VLAN from HOME with justine G/W
VPN clients can access justine MGMT interface IP
11:06:10.811023 vpn In IP 172.17.1.10 > 192.168.100.2: ICMP echo request, id 44951, seq 814, length 64
11:06:10.811041 vpn Out IP 192.168.100.2 > 172.17.1.10: ICMP echo reply, id 44951, seq 814, length 64I have set up more strict forwarding rules:
Chain FORWARD (policy DROP 52 packets, 4368 bytes)
num pkts bytes target prot opt in out source destination
9 35 5441 ACCEPT all -- enp1s0 vpn 192.168.0.0/24 0.0.0.0/0
10 32 5244 ACCEPT all -- vpn enp1s0 0.0.0.0/0 192.168.0.0/24but this does not help.
This is because it is not going through FORWARD but through INPUT:
iptables -A INPUT -i vpn -d 192.168.100.0/24 -j LOG[604557.640819] IN=vpn OUT= MAC= SRC=172.17.1.10 DST=192.168.100.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=56812 DF PROTO=ICMP TYPE=8 CODE=0 ID=44951 SEQ=1064Adding the above rule will block it. But is this normal that any IP assigned to any interface can be used when routing to a G/W?
FIXED: I have added the following rule:
iptables -A INPUT ! -i mgmt -d 192.168.100.0/24 -j DROPFOLLOWUP: Is this the same for other devices? Try static route to them and access MGMT IP.
ip route add 192.168.100.50 via 192.168.0.50 dev wlan0And it will ping on 192.168.100.50. So services bound only to 192.168.100.50 will be exposed to HOME VLAN devices.
CONCLUSION: Binding to selected IP address does not protect service from being accessed from another network interface without extra firewall rule to prevent this!
What makes localhost (lo) interface special - services bound to it are not accessible from other interfaces? It has local routing tables set up by default.
Same problem for caroline
Same with caroline:
ip route add 192.168.100.1 via 192.168.0.1 dev wlan0I can now access https://192.168.100.1/cgi-bin/luci/ while on HOME VLAN, probably will work form SERVER and any other as well!
FIXED: DROP all INPUT on all interfaces apart from MGMT. Added ACCEPT rules for LAN, GUEST and SERVER VLANs for DHCP (UDP 67), DNS (TCP/UDP: 53) and ICMP.
I could not use the negative interface setup as in case of justine. Probably should use default INPUT DROP on justine as well and only allow mgmt interface traffic as well.
Would this be same for goro and haru?
They don't have IP on HOME network but they have interface. Injecting packet for MGMT IP to their HOME interface may be (using MAC/ARP) possible but they would not respond since they have no routing to HOME network?
Access to web services from internal network
Need to manually add static route for devices using default DHCP G/W (justine) when going to local server services like video.hexadust.net.
ip route add 46.7.126.16 dev eth0 via 192.168.0.1
ping video.hexadust.netThings to try:
- Try to push static routes from DHCP - this did not work for some reason
- Set up static route on justine
- Set up SNAT on justnie
- Use bridge layer DNAT: https://ebtables.netfilter.org/br_fw_ia/br_fw_ia.html
- Set up split DNS
video.hexadust.net 192.168.50.159- This would require justine to forward to SERVER VLAN since ann uses 0.2 as default G/W
- Set up separate DNS for internal access
No more IPs on DHCP
Looks like Horizon box eats up leases: https://www.boards.ie/discussion/2057720465/my-new-virgin-media-stb-issued-two-lan-i-ps
I have removed leases from /var/lib/dhcp.lease file on caroline.
Justine access is laggy; DNS is slow
Happens after OpenWRT updates.
Try disconnecting and connecting network cables for Haru and uplink on goro.
UPDATE: 2023-11-05: I have replaced both cables that connect Haru and Goro. The link tends to drop to 100Mbit if Goro end is disconnected, reconnecting fixes it.
Igor does not set it's default route after boot
While it is configured in web UI it is not taking effect on boot and results in it unable to find local network. VMs stuff will work OK though.
TODO
YunoHost access to internet via default IP instead of VPN
Not good for IRC etc.
Clone the VM and put copy in HOME VLAN, remove public web stuff from it, remove web clients from SERVER VLAN one.