ip rule cannot match based on the source IPv4 address
Hi guys, I'm doing a "split routing table" on my router, and I'm importing routes into specific kernel routing tables via bird. So I'm doing some strange routing by using ip rule. [root@archlinux ~]# ip -4 rule 0: from all lookup local 32751: from 134.195.121.118 lookup 101 32752: from all iif Tanuki_IX lookup 147 32753: from all iif ens19 lookup 147 32754: from all iif cu_gre6 lookup 247 32755: from all iif cu lookup 147 32756: from all iif openvpn_stuix.5 lookup 247 32757: from all iif openvpn_stuix lookup 147 32758: from all iif openvpn_c1v lookup 147 32759: from all iif openvpn_c1v.5 lookup 247 32760: from all iif openvpn_Eric.5 lookup 247 32761: from all iif openvpn_Eric lookup 147 32762: from all iif ll-ix lookup 147 32763: from all iif ll-ix_gre6 lookup 247 32764: from all iif KSKB lookup 147 32765: from all iif KSKB_gre6 lookup 247 32766: from all lookup main 32767: from all lookup default As you could see, IPv4 from 134.195.121.118/32 will using table 101 which only contain [root@archlinux ~]# ip -4 route show table 101 default via 33.0.0.6 dev openvpn_Eric [root@archlinux ~]# When I am mtr to 8.8.8.8, it still uses the main route table NOT table 101 which does not follow the ip rule. [root@archlinux ~]# mtr -a 134.195.121.118 8.8.8.8 [root@archlinux ~]# tcpdump -i any host 134.195.121.118 tcpdump: data link type LINUX_SLL2 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes 13:30:11.991460 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33008, length 44 13:30:11.998947 openvpn_Eric Out IP archlinux.palace-2 > 172.17.0.1.palace-2: UDP, length 28 13:30:11.999059 openvpn_Eric Out IP archlinux.33915 > 172.17.0.1.palace-2: UDP, length 28 13:30:11.999384 openvpn_Eric Out IP archlinux.27004 > 172.17.0.1.palace-2: UDP, length 28 13:30:12.006020 openvpn_Eric Out IP archlinux.palace-2 > 59.200.0.1.palace-2: UDP, length 28 13:30:12.006096 openvpn_Eric Out IP archlinux.33915 > 59.200.0.1.palace-2: UDP, length 28 13:30:12.006136 openvpn_Eric Out IP archlinux.27004 > 59.200.0.1.palace-2: UDP, length 28 13:30:12.017489 openvpn_Eric Out IP archlinux.palace-2 > 208.99.49.1.palace-2: UDP, length 28 13:30:12.017628 openvpn_Eric Out IP archlinux.33915 > 208.99.49.1.palace-2: UDP, length 28 13:30:12.017650 openvpn_Eric Out IP archlinux.27004 > 208.99.49.1.palace-2: UDP, length 28 13:30:12.023034 openvpn_Eric Out IP archlinux.palace-2 > 172.23.89.1.palace-2: UDP, length 28 13:30:12.023147 openvpn_Eric Out IP archlinux.33915 > 172.23.89.1.palace-2: UDP, length 28 13:30:12.023202 openvpn_Eric Out IP archlinux.27004 > 172.23.89.1.palace-2: UDP, length 28 13:30:12.027182 openvpn_Eric Out IP archlinux.palace-2 > 172.17.0.1.24452: UDP, length 28 13:30:12.027203 openvpn_Eric Out IP archlinux.33915 > 172.17.0.1.24452: UDP, length 28 13:30:12.027212 openvpn_Eric Out IP archlinux.27004 > 172.17.0.1.24452: UDP, length 28 13:30:12.031218 openvpn_Eric Out IP archlinux.palace-2 > 59.200.0.1.24452: UDP, length 28 13:30:12.031235 openvpn_Eric Out IP archlinux.33915 > 59.200.0.1.24452: UDP, length 28 13:30:12.031243 openvpn_Eric Out IP archlinux.27004 > 59.200.0.1.24452: UDP, length 28 13:30:12.036170 openvpn_Eric Out IP archlinux.palace-2 > 208.99.49.1.24452: UDP, length 28 13:30:12.036192 openvpn_Eric Out IP archlinux.33915 > 208.99.49.1.24452: UDP, length 28 13:30:12.036200 openvpn_Eric Out IP archlinux.27004 > 208.99.49.1.24452: UDP, length 28 13:30:12.040852 openvpn_Eric Out IP archlinux.palace-2 > 172.23.89.1.24452: UDP, length 28 13:30:12.040873 openvpn_Eric Out IP archlinux.33915 > 172.23.89.1.24452: UDP, length 28 13:30:12.040881 openvpn_Eric Out IP archlinux.27004 > 172.23.89.1.24452: UDP, length 28 13:30:12.046089 openvpn_Eric Out IP archlinux.palace-2 > 172.17.0.1.24453: UDP, length 28 13:30:12.046119 openvpn_Eric Out IP archlinux.33915 > 172.17.0.1.24453: UDP, length 28 13:30:12.046138 openvpn_Eric Out IP archlinux.27004 > 172.17.0.1.24453: UDP, length 28 13:30:12.051427 openvpn_Eric Out IP archlinux.palace-2 > 59.200.0.1.24453: UDP, length 28 13:30:12.051453 openvpn_Eric Out IP archlinux.33915 > 59.200.0.1.24453: UDP, length 28 13:30:12.051467 openvpn_Eric Out IP archlinux.27004 > 59.200.0.1.24453: UDP, length 28 13:30:12.055046 openvpn_Eric Out IP archlinux.palace-2 > 208.99.49.1.24453: UDP, length 28 13:30:12.055075 openvpn_Eric Out IP archlinux.33915 > 208.99.49.1.24453: UDP, length 28 13:30:12.055085 openvpn_Eric Out IP archlinux.27004 > 208.99.49.1.24453: UDP, length 28 13:30:12.058847 openvpn_Eric Out IP archlinux.palace-2 > 172.23.89.1.24453: UDP, length 28 13:30:12.058874 openvpn_Eric Out IP archlinux.33915 > 172.23.89.1.24453: UDP, length 28 13:30:12.058886 openvpn_Eric Out IP archlinux.27004 > 172.23.89.1.24453: UDP, length 28 13:30:12.091913 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33009, length 44 13:30:12.096047 openvpn_Eric Out IP archlinux.palace-2 > 7.254.0.1.palace-2: UDP, length 28 13:30:12.096081 openvpn_Eric Out IP archlinux.33915 > 7.254.0.1.palace-2: UDP, length 28 13:30:12.096092 openvpn_Eric Out IP archlinux.27004 > 7.254.0.1.palace-2: UDP, length 28 13:30:12.101389 openvpn_Eric Out IP archlinux.palace-2 > 7.254.0.1.24452: UDP, length 28 13:30:12.101409 openvpn_Eric Out IP archlinux.33915 > 7.254.0.1.24452: UDP, length 28 13:30:12.101422 openvpn_Eric Out IP archlinux.27004 > 7.254.0.1.24452: UDP, length 28 13:30:12.192666 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33010, length 44 13:30:12.293006 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33011, length 44 13:30:12.393416 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33012, length 44 13:30:12.494029 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33013, length 44 13:30:12.607660 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33014, length 44 13:30:12.679414 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33015, length 44 13:30:12.966665 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33019, length 44 13:30:13.039053 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33020, length 44 13:30:14.045371 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33034, length 44 13:30:14.116878 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33035, length 44 13:30:14.188702 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33036, length 44 13:30:14.260455 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33037, length 44 13:30:14.332277 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33038, length 44 13:30:14.404019 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33039, length 44 13:30:14.475861 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33040, length 44 13:30:14.547705 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33041, length 44 13:30:14.619529 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33042, length 44 13:30:14.691361 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33043, length 44 13:30:14.763157 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33044, length 44 13:30:14.834959 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33045, length 44 13:30:14.906978 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33046, length 44 13:30:14.978577 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33047, length 44 13:30:15.050371 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33048, length 44 13:30:15.122227 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33049, length 44 13:30:15.193915 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33050, length 44 13:30:15.265629 ens18 Out IP archlinux > dns.google: ICMP echo request, id 65299, seq 33051, length 44 13:30:15.325557 openvpn_Eric Out IP archlinux.palace-2 > root-mia-01.zerotier.com.palace-2: UDP, length 137 13:30:15.325578 openvpn_Eric Out IP archlinux.33915 > root-mia-01.zerotier.com.palace-2: UDP, length 137 13:30:15.325586 openvpn_Eric Out IP archlinux.27004 > root-mia-01.zerotier.com.palace-2: UDP, length 137 13:30:15.328638 openvpn_Eric Out IP archlinux.palace-2 > root-zrh-01.zerotier.com.palace-2: UDP, length 137 13:30:15.328657 openvpn_Eric Out IP archlinux.33915 > root-zrh-01.zerotier.com.palace-2: UDP, length 137 13:30:15.328667 openvpn_Eric Out IP archlinux.27004 > root-zrh-01.zerotier.com.palace-2: UDP, length 137 13:30:15.329981 openvpn_Eric Out IP archlinux.palace-2 > root-sgp-01.zerotier.com.palace-2: UDP, length 137 13:30:15.329998 openvpn_Eric Out IP archlinux.33915 > root-sgp-01.zerotier.com.palace-2: UDP, length 137 13:30:15.330007 openvpn_Eric Out IP archlinux.27004 > root-sgp-01.zerotier.com.palace-2: UDP, length 137 13:30:15.332916 openvpn_Eric Out IP archlinux.palace-2 > 104.194.8.134.palace-2: UDP, length 137 13:30:15.332934 openvpn_Eric Out IP archlinux.33915 > 104.194.8.134.palace-2: UDP, length 137 13:30:15.332941 openvpn_Eric Out IP archlinux.27004 > 104.194.8.134.palace-2: UDP, length 137 As you could see, the package still using the default route table not table 101. This happens on Debian at the same time, and I think it's a problem about Linux, not just ArchLinux. I also tested passing ip -6 rule add from some ipv6 address table 101. It can work normally, that is to say, there is a matching problem with ipv4 in ip rule, but this phenomenon does not exist in ipv6. Best, *Brandon Zhi* HUIZE LTD www.huize.asia <https://huize.asia/>| www.ixp.su | Twitter This e-mail and any attachments or any reproduction of this e-mail in whatever manner are confidential and for the use of the addressee(s) only. HUIZE LTD can’t take any liability and guarantee of the text of the email message and virus.
On 3/19/23 4:06 AM, Brandon Zhi wrote:
Hi guys,
Hi,
I'm doing a "split routing table" on my router, and I'm importing routes into specific kernel routing tables via bird. So I'm doing some strange routing by using ip rule.
I've got systems doing similar.
[root@archlinux ~]# ip -4 rule 0: from all lookup local 32751: from 134.195.121.118 lookup 101 ... 32766: from all lookup main 32767: from all lookup default
As you could see, IPv4 from 134.195.121.118/32 will using table 101
ACK
When I am mtr to 8.8.8.8, it still uses the main route table NOT table 101 which does not follow the ip rule.
Have you tried using something other than `mtr`? What happens if you simply try to use `ping` or `telnet` or `netcat`?
[root@archlinux ~]# mtr -a 134.195.121.118 8.8.8.8
[root@archlinux ~]# tcpdump -i any host 134.195.121.118 tcpdump: data link type LINUX_SLL2 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
Please tweak your tcpdump command line and re-run your test. Please: - match the source (134.195.121.118) and destination (8.8.8.8) - disable name resolution (-n)
As you could see, the package still using the default route table not table 101.
Unfortunately I'm not far enough through my cup of coffee to use the information at hand to deduce anything. What happens when you run the following command: ip route get from 134.195.121.118 8.8.8.8
This happens on Debian at the same time, and I think it's a problem about Linux, not just ArchLinux. I also tested passing ip -6 rule add from some ipv6 address table 101. It can work normally, that is to say, there is a matching problem with ipv4 in ip rule, but this phenomenon does not exist in ipv6.
I feel like this isn't a BIRD specific issue. I've had problems in the past with ip rules relying on the source address. I think /some/, but not all, of it has to do with when and / or how the program in question picks the source address. Though I do see that you specified the source (BIND) address for `mtr` via it's `-a` option. I've also seen some systems that PBR using source address doesn't work reliably and I never took the time to figure out why. I think I'd re-send a copy of the message that I'm replying to off to the Linux Advanced Routing & Traffic Control (LARTC) mailing list if I were you as I think that's probably a better place than here. But I wouldn't hold my breath as that list has been very hit or miss for a while now. -- Grant. . . . unix || die
Hi Grant, Thanks for the reply. I have recently tested ip rule on my company's server. Our network structure is like this. GRE Server(IN) >>>>> Server(SG)>>> upstream. The ip rule on Server(IN) 0: from all lookup local 32760: from all iif gre_sg lookup 147 32762: from 156.236.18.0/24 lookup 147 32763: from 185.81.217.0/24 lookup 147 32764: from 46.23.100.0/22 lookup 147 32765: from 46.23.110.0/24 lookup 147 32766: from all lookup main 32767: from all lookup default 20230227:~# ip route get 8.8.8.8 8.8.8.8 via 139.84.140.1 dev enp1s0f0 src 139.84.140.60 uid 0 cache root@20230227:~# ip route show table 147 | grep 8.8.8.0/24 8.8.8.0/24 via 10.0.5.1 dev gre_sg proto bird metric 32 When I did ping -B 46.23.100.0 8.8.8.8 on the server (IN), I'm not sure how to bind the source address by using ping. 20230227:~# ping -B 46.23.100.0 8.8.8.8 PING 8.8.8.8 (8.8.8.8) from 46.23.100.0 : 56(124) bytes of data. 20230227:~# tcpdump -i any host 46.23.100.0 -n tcpdump: data link type LINUX_SLL2 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes 10:57:09.328396 lo In IP 46.23.100.0 > 46.23.100.0: ICMP echo request, id 47075, seq 1, length 64 10:57:10.360390 lo In IP 46.23.100.0 > 46.23.100.0: ICMP echo request, id 47075, seq 2, length 64 10:57:11.384382 lo In IP 46.23.100.0 > 46.23.100.0: ICMP echo request, id 47075, seq 3, length 64 10:57:12.408391 lo In IP 46.23.100.0 > 46.23.100.0: ICMP echo request, id 47075, seq 4, length 64 10:57:13.432389 lo In IP 46.23.100.0 > 46.23.100.0: ICMP echo request, id 47075, seq 5, length 64 10:57:14.456384 lo In IP 46.23.100.0 > 46.23.100.0: ICMP echo request, id 47075, seq 6, length 64 10:57:15.480390 lo In IP 46.23.100.0 > 46.23.100.0: ICMP echo request, id 47075, seq 7, length 64 10:57:16.504398 lo In IP 46.23.100.0 > 46.23.100.0: ICMP echo request, id 47075, seq 8, length 64 Then I tried MTR 20230227:~# mtr -a 46.23.100.0 8.8.8.8 The output from tcpdump shows that it's still using the default table, not table 147. ip007-20230227:~# tcpdump -i any host 46.23.100.0 -n tcpdump: data link type LINUX_SLL2 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes 10:59:28.525837 enp1s0f0 Out IP 46.23.100.0 > 8.8.8.8: ICMP echo request, id 50435, seq 33038, length 44 10:59:28.597606 enp1s0f0 Out IP 46.23.100.0 > 8.8.8.8: ICMP echo request, id 50435, seq 33039, length 44 10:59:28.669374 enp1s0f0 Out IP 46.23.100.0 > 8.8.8.8: ICMP echo request, id 50435, seq 33040, length 44 10:59:28.741155 enp1s0f0 Out IP 46.23.100.0 > 8.8.8.8: ICMP echo request, id 50435, seq 33041, length 44 10:59:28.812956 enp1s0f0 Out IP 46.23.100.0 > 8.8.8.8: ICMP echo request, id 50435, seq 33042, length 44 10:59:28.884728 enp1s0f0 Out IP 46.23.100.0 > 8.8.8.8: ICMP echo request, id 50435, seq 33043, length 44 10:59:28.956542 enp1s0f0 Out IP 46.23.100.0 > 8.8.8.8: ICMP echo request, id 50435, seq 33044, length 44 10:59:29.028369 enp1s0f0 Out IP 46.23.100.0 > 8.8.8.8: ICMP echo request, id 50435, seq 33045, length 44 10:59:29.100148 enp1s0f0 Out IP 46.23.100.0 > 8.8.8.8: ICMP echo request, id 50435, seq 33046, length 44 However, when I use netcat. It follows the ip rule. -20230227:~# nc -s 46.23.100.0 -u 8.8.8.8 53 hello tcpdump output on Server(IN) 20230227:~# tcpdump -i any host 46.23.100.0 -n tcpdump: data link type LINUX_SLL2 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes 10:59:28.525837 enp1s0f0 Out IP 46.23.100.0 > 8.8.8.8: ICMP echo request, id 50435, seq 33038, length 44 10:59:28.597606 enp1s0f0 Out IP 46.23.100.0 > 8.8.8.8: ICMP echo request, id 50435, seq 33039, length 44 10:59:28.669374 enp1s0f0 Out IP 46.23.100.0 > 8.8.8.8: ICMP echo request, id 50435, seq 33040, length 44 10:59:28.741155 enp1s0f0 Out IP 46.23.100.0 > 8.8.8.8: ICMP echo request, id 50435, seq 33041, length 44 10:59:28.812956 enp1s0f0 Out IP 46.23.100.0 > 8.8.8.8: ICMP echo request, id 50435, seq 33042, length 44 10:59:28.884728 enp1s0f0 Out IP 46.23.100.0 > 8.8.8.8: ICMP echo request, id 50435, seq 33043, length 44 10:59:28.956542 enp1s0f0 Out IP 46.23.100.0 > 8.8.8.8: ICMP echo request, id 50435, seq 33044, length 44 10:59:29.028369 enp1s0f0 Out IP 46.23.100.0 > 8.8.8.8: ICMP echo request, id 50435, seq 33045, length 44 10:59:29.100148 enp1s0f0 Out IP 46.23.100.0 > 8.8.8.8: ICMP echo request, id 50435, seq 33046, length 44 tcpdump output on Server(SG) -2252644:~# tcpdump -i any host 46.23.100.0 -n tcpdump: data link type LINUX_SLL2 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes 11:01:25.167122 vultr_IN In IP 46.23.100.0.14417 > 8.8.8.8.53: domain [length 6 < 12] (invalid) 11:01:25.167142 eth0 Out IP 46.23.100.0.14417 > 8.8.8.8.53: domain [length 6 < 12] (invalid) The income packages disappear, I don't think it's related to this question. Perhaps DNS should not reply to my "hello" package or I misconfigure the BGP session, which causes the income packages to go to another server. So maybe some software like mtr does not follow the IP rule? Even though the source address is selected. Linux Advanced Routing & Traffic Control (LARTC) mailing list I have subscribed to the mailing list on here, could you CC brandon@huize.asia for the thread? List: lartc; ( subscribe / unsubscribe )
Info:
This is the mailing list for Linux Advanced Routing and Traffic Control discussion.
Archives:
Best, *Brandon Zhi* HUIZE LTD www.huize.asia <https://huize.asia/>| www.ixp.su | Twitter This e-mail and any attachments or any reproduction of this e-mail in whatever manner are confidential and for the use of the addressee(s) only. HUIZE LTD can’t take any liability and guarantee of the text of the email message and virus. On Sun, 19 Mar 2023 at 17:07, Grant Taylor via Bird-users < bird-users@network.cz> wrote:
On 3/19/23 4:06 AM, Brandon Zhi wrote:
Hi guys,
Hi,
I'm doing a "split routing table" on my router, and I'm importing routes into specific kernel routing tables via bird. So I'm doing some strange routing by using ip rule.
I've got systems doing similar.
[root@archlinux ~]# ip -4 rule 0: from all lookup local 32751: from 134.195.121.118 lookup 101 ... 32766: from all lookup main 32767: from all lookup default
As you could see, IPv4 from 134.195.121.118/32 will using table 101
ACK
When I am mtr to 8.8.8.8, it still uses the main route table NOT table 101 which does not follow the ip rule.
Have you tried using something other than `mtr`?
What happens if you simply try to use `ping` or `telnet` or `netcat`?
[root@archlinux ~]# mtr -a 134.195.121.118 8.8.8.8
[root@archlinux ~]# tcpdump -i any host 134.195.121.118 tcpdump: data link type LINUX_SLL2 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
Please tweak your tcpdump command line and re-run your test.
Please: - match the source (134.195.121.118) and destination (8.8.8.8) - disable name resolution (-n)
As you could see, the package still using the default route table not table 101.
Unfortunately I'm not far enough through my cup of coffee to use the information at hand to deduce anything.
What happens when you run the following command:
ip route get from 134.195.121.118 8.8.8.8
This happens on Debian at the same time, and I think it's a problem about Linux, not just ArchLinux. I also tested passing ip -6 rule add from some ipv6 address table 101. It can work normally, that is to say, there is a matching problem with ipv4 in ip rule, but this phenomenon does not exist in ipv6.
I feel like this isn't a BIRD specific issue.
I've had problems in the past with ip rules relying on the source address. I think /some/, but not all, of it has to do with when and / or how the program in question picks the source address. Though I do see that you specified the source (BIND) address for `mtr` via it's `-a` option. I've also seen some systems that PBR using source address doesn't work reliably and I never took the time to figure out why.
I think I'd re-send a copy of the message that I'm replying to off to the Linux Advanced Routing & Traffic Control (LARTC) mailing list if I were you as I think that's probably a better place than here. But I wouldn't hold my breath as that list has been very hit or miss for a while now.
-- Grant. . . . unix || die
participants (2)
-
Brandon Zhi -
Grant Taylor