<div dir="ltr">Dear Bjørn,<div><br></div><div>Thank you so much for your effort and detailed information on how to shorten out this issue.</div><div>I really appreciate this! </div><div><br></div><div>Apologies for the late reply.</div><div>We had a holiday here in Israel, and with the covid-19 situation (we in the middle of the second wave) I'm really losing my head.</div><div>The
situation here in Israel is really bad compared to the rest of the world :-(</div><div><br></div><div>Again, thank you so much! Your solution opened my mind to new things and ideas.<br></div><div><br></div><div>However, I tried out your solution, and It doesn't work on my setup.</div><div>I'm using a clean ubuntu 18.04 with vpp version 18.07-release without any luck.</div><div><br></div><div>I have compiled an asciinema of the entire process, maybe you can take a quick look?</div><div><br></div><div> <a href="goog_1496068446">https://asciinema.org/a/05VG1Xhl3kYOydE685wjphoqf<br></a></div><div><a href="https://asciinema.org/a/3sOWj8x5YBWnT5mNUgxiIHmUw"> https://asciinema.org/a/3sOWj8x5YBWnT5mNUgxiIHmUw</a><br></div><div><br></div><div><br></div><div>Here is also a quick figure of the topology</div><div><div><img src="cid:ii_kfp6hiss0" alt="image.png" width="514" height="381"><br></div></div><div><br></div><div> You were saying "There are a gazillion other ways to achieve the same" are you referring to the "two-way default routing"? </div><div>Or you mean the entire process.</div><div><br></div><div>As I said your solution is very creative - but even though I spent a long time on this - I didn't manage to come up with an alternative. </div><div><br></div><div>Even with your setup, I did try to do a long debug to it.</div><div>I used tcpdump on all interfaces (including tracing out the ethernet address "-e"), but nothing I can say will provide good feedback here.</div><div><br></div><div><br></div><div>Will be very thankful if someone can drop a comment here.</div><div><br></div><div>Thank you.</div><div>Sassy</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Sep 23, 2020 at 4:03 PM Bjørn Mork <<a href="mailto:bjorn@mork.no">bjorn@mork.no</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">FWIW, I just had to try out how much it takes to connect a host owned<br>
LTE netdev to VPP. That's actually pretty easy. But you end forwarding<br>
between the VPP host interface and the LTE netdev in Linux, which I<br>
guess is something you eventually might like to avoid. I assume this is<br>
pretty easy to implement as a VPP plugin or similar. It's not related<br>
to the actual functionality here anyway.<br>
<br>
My demo setup is a slight modification of the start of the VPP tutorial:<br>
<a href="https://fd.io/docs/vpp/v2005/gettingstarted/progressivevpp/interface.html" rel="noreferrer" target="_blank">https://fd.io/docs/vpp/v2005/gettingstarted/progressivevpp/interface.html</a><br>
<br>
<br>
What I did was - on the host:<br>
<br>
1) Create veth pair according to tutorial instructions:<br>
<br>
ip link add name vpp1out type veth peer name vpp1host<br>
<br>
<br>
2) Create a new sub-interface for MBIM session ID 3:<br>
<br>
ip link add wwan0.3 link wwan0 type vlan id 3<br>
<br>
<br>
3) Create a new network namespace and put both these interfaces there:<br>
<br>
ip netns add vpplte<br>
ip link set vpp1host netns vpplte<br>
ip link set wwan0.3 netns vpplte<br>
<br>
4) Start a shell in the network namespace and execute the remaining host<br>
commands there:<br>
<br>
ip netns exec vpplte /bin/bash<br>
<br>
<br>
5) Set all links up:<br>
<br>
ip link set lo up<br>
ip link set vpp1host up<br>
ip link set wwan0.3 up<br>
<br>
<br>
6) Enable proxy-arp and forwarding<br>
<br>
echo 1 > /proc/sys/net/ipv4/conf/vpp1host/proxy_arp<br>
echo 1 > /proc/sys/net/ipv4/conf/vpp1host/forwarding<br>
echo 1 > /proc/sys/net/ipv4/conf/wwan0.3/forwarding<br>
<br>
<br>
7) set up two-way default routing<br>
<br>
ip route add default dev wwan0.3<br>
ip route add default dev vpp1host table 3<br>
ip rule add pref 1000 iif wwan0.3 lookup 3<br>
<br>
8) Disable arp on the veth endpoint<br>
<br>
ip link set dev vpp1host arp off<br>
<br>
<br>
9) Connect the session, and note the assigned address<br>
<br>
mbimcli -p -d /dev/cdc-wdm0 --connect=apn=telenor.smart,session-id=3,ip-type=ipv4v6<br>
<br>
<br>
This is the output I got, to be used in the VPP shell below:<br>
<br>
[/dev/cdc-wdm0] Successfully connected<br>
<br>
[/dev/cdc-wdm0] Connection status:<br>
Session ID: '3'<br>
Activation state: 'activated'<br>
Voice call state: 'none'<br>
IP type: 'ipv4v6'<br>
Context type: 'internet'<br>
Network error: 'unknown'<br>
<br>
[/dev/cdc-wdm0] IPv4 configuration available: 'address, gateway, dns, mtu'<br>
IP [0]: '<a href="http://10.169.198.6/30" rel="noreferrer" target="_blank">10.169.198.6/30</a>'<br>
Gateway: '10.169.198.5'<br>
DNS [0]: '193.213.112.4'<br>
DNS [1]: '130.67.15.198'<br>
MTU: '1500'<br>
<br>
[/dev/cdc-wdm0] IPv6 configuration available: 'address, gateway, dns, mtu'<br>
IP [0]: '2a02:2121:2c0:e913:392d:3e46:cf98:4ca3/64'<br>
Gateway: '2a02:2121:2c0:e913:70a8:2a1c:dc62:2022'<br>
DNS [0]: '2001:4600:4:fff::52'<br>
DNS [1]: '2001:4600:4:1fff::52'<br>
MTU: '1540'<br>
<br>
<br>
<br>
In the VPP shell:<br>
<br>
10) Create host interface and set link up as instructed by the tutorial<br>
<br>
create host-interface name vpp1out<br>
set int state host-vpp1out up<br>
<br>
<br>
11) Assign the IP address from the mbimcli command above:<br>
<br>
<br>
set int ip address host-vpp1out <a href="http://10.169.198.6/30" rel="noreferrer" target="_blank">10.169.198.6/30</a><br>
<br>
<br>
12) Set a default route (or whatever you want) via some fake gateway<br>
address on the other end of that veth pair - the one suggested by<br>
the modem is fine:<br>
<br>
ip route add 0/0 via 10.169.198.5<br>
<br>
<br>
13) ping an address on the other side of the LTE link:<br>
<br>
<br>
vpp# ping 8.8.8.8 <br>
116 bytes from <a href="http://8.8.8.8" rel="noreferrer" target="_blank">8.8.8.8</a>: icmp_seq=1 ttl=114 time=68.0229 ms<br>
116 bytes from <a href="http://8.8.8.8" rel="noreferrer" target="_blank">8.8.8.8</a>: icmp_seq=2 ttl=114 time=48.9930 ms<br>
116 bytes from <a href="http://8.8.8.8" rel="noreferrer" target="_blank">8.8.8.8</a>: icmp_seq=3 ttl=114 time=52.8561 ms<br>
116 bytes from <a href="http://8.8.8.8" rel="noreferrer" target="_blank">8.8.8.8</a>: icmp_seq=4 ttl=114 time=51.5395 ms<br>
116 bytes from <a href="http://8.8.8.8" rel="noreferrer" target="_blank">8.8.8.8</a>: icmp_seq=5 ttl=114 time=99.0668 ms<br>
<br>
Statistics: 5 sent, 5 received, 0% packet loss<br>
<br>
<br>
<br>
<br>
This was a simple abuse of a dedicated network namespace in the Linux<br>
host, where it is easy to do two-way default routing without having to<br>
care about addressing and routes at all. There are a gazillion other<br>
ways to achieve the same,<br>
<br>
There is no way to know that there is anything special about the<br>
host-vpp1out interface from the VPP shell. It looks like any other host<br>
interface. But you'll obviously have to do something on the host if you<br>
are going to run anything fancy over it. But that's normally not the<br>
use case for an LTE connection anyway, I guess.<br>
<br>
<br>
vpp# show interface <br>
Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count <br>
host-vpp1out 1 up 9000/0/0/0 rx packets 135<br>
rx bytes 22334<br>
tx packets 61<br>
tx bytes 4874<br>
drops 146<br>
ip4 15<br>
ip6 32<br>
local0 0 down 0/0/0/0 <br>
vpp# show ip neighbor<br>
Time IP Flags Ethernet Interface <br>
4095.4097 10.169.198.5 D fe:cd:9c:57:a9:da host-vpp1out<br>
vpp# show ip fib <br>
ipv4-VRF:0, fib_index:0, flow hash:[src dst sport dport proto ] epoch:0 flags:none locks:[adjacency:1, recursive-resolution:1, default-route:1, nat-hi:2, ]<br>
<a href="http://0.0.0.0/0" rel="noreferrer" target="_blank">0.0.0.0/0</a><br>
unicast-ip4-chain<br>
[@0]: dpo-load-balance: [proto:ip4 index:1 buckets:1 uRPF:14 to:[60:5760]]<br>
[0] [@12]: dpo-load-balance: [proto:ip4 index:13 buckets:1 uRPF:12 to:[0:0] via:[60:5760]]<br>
[0] [@5]: ipv4 via 10.169.198.5 host-vpp1out: mtu:9000 next:3 fecd9c57a9da02fe5a6a76810800<br>
<a href="http://0.0.0.0/32" rel="noreferrer" target="_blank">0.0.0.0/32</a><br>
unicast-ip4-chain<br>
[@0]: dpo-load-balance: [proto:ip4 index:2 buckets:1 uRPF:1 to:[0:0]]<br>
[0] [@0]: dpo-drop ip4<br>
<a href="http://10.169.198.4/32" rel="noreferrer" target="_blank">10.169.198.4/32</a><br>
unicast-ip4-chain<br>
[@0]: dpo-load-balance: [proto:ip4 index:10 buckets:1 uRPF:9 to:[0:0]]<br>
[0] [@0]: dpo-drop ip4<br>
<a href="http://10.169.198.5/32" rel="noreferrer" target="_blank">10.169.198.5/32</a><br>
unicast-ip4-chain<br>
[@0]: dpo-load-balance: [proto:ip4 index:13 buckets:1 uRPF:12 to:[0:0] via:[60:5760]]<br>
[0] [@5]: ipv4 via 10.169.198.5 host-vpp1out: mtu:9000 next:3 fecd9c57a9da02fe5a6a76810800<br>
<a href="http://10.169.198.4/30" rel="noreferrer" target="_blank">10.169.198.4/30</a><br>
unicast-ip4-chain<br>
[@0]: dpo-load-balance: [proto:ip4 index:9 buckets:1 uRPF:8 to:[0:0]]<br>
[0] [@4]: ipv4-glean: host-vpp1out: mtu:9000 next:1 ffffffffffff02fe5a6a76810806<br>
<a href="http://10.169.198.6/32" rel="noreferrer" target="_blank">10.169.198.6/32</a><br>
unicast-ip4-chain<br>
[@0]: dpo-load-balance: [proto:ip4 index:12 buckets:1 uRPF:13 to:[15:1440]]<br>
[0] [@2]: dpo-receive: 10.169.198.6 on host-vpp1out<br>
<a href="http://10.169.198.7/32" rel="noreferrer" target="_blank">10.169.198.7/32</a><br>
unicast-ip4-chain<br>
[@0]: dpo-load-balance: [proto:ip4 index:11 buckets:1 uRPF:11 to:[0:0]]<br>
[0] [@0]: dpo-drop ip4<br>
<a href="http://224.0.0.0/4" rel="noreferrer" target="_blank">224.0.0.0/4</a><br>
unicast-ip4-chain<br>
[@0]: dpo-load-balance: [proto:ip4 index:4 buckets:1 uRPF:3 to:[0:0]]<br>
[0] [@0]: dpo-drop ip4<br>
<a href="http://240.0.0.0/4" rel="noreferrer" target="_blank">240.0.0.0/4</a><br>
unicast-ip4-chain<br>
[@0]: dpo-load-balance: [proto:ip4 index:3 buckets:1 uRPF:2 to:[0:0]]<br>
[0] [@0]: dpo-drop ip4<br>
<a href="http://255.255.255.255/32" rel="noreferrer" target="_blank">255.255.255.255/32</a><br>
unicast-ip4-chain<br>
[@0]: dpo-load-balance: [proto:ip4 index:5 buckets:1 uRPF:4 to:[0:0]]<br>
[0] [@0]: dpo-drop ip4<br>
<br>
<br>
<br>
And the Linux host view:<br>
<br>
root@miraculix:/tmp# ip link<br>
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000<br>
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br>
10: vpp1host@if11: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000<br>
link/ether fe:cd:9c:57:a9:da brd ff:ff:ff:ff:ff:ff link-netnsid 0<br>
12: wwan0.3@if3: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000<br>
link/ether 42:0a:0d:ab:b4:f5 brd ff:ff:ff:ff:ff:ff link-netnsid 0<br>
root@miraculix:/tmp# ip addr<br>
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000<br>
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br>
inet <a href="http://127.0.0.1/8" rel="noreferrer" target="_blank">127.0.0.1/8</a> scope host lo<br>
valid_lft forever preferred_lft forever<br>
inet6 ::1/128 scope host <br>
valid_lft forever preferred_lft forever<br>
10: vpp1host@if11: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000<br>
link/ether fe:cd:9c:57:a9:da brd ff:ff:ff:ff:ff:ff link-netnsid 0<br>
inet6 fe80::fccd:9cff:fe57:a9da/64 scope link <br>
valid_lft forever preferred_lft forever<br>
12: wwan0.3@if3: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000<br>
link/ether 42:0a:0d:ab:b4:f5 brd ff:ff:ff:ff:ff:ff link-netnsid 0<br>
inet6 2a02:2121:2c0:e913:400a:dff:feab:b4f5/64 scope global dynamic mngtmpaddr <br>
valid_lft forever preferred_lft forever<br>
inet6 fe80::400a:dff:feab:b4f5/64 scope link <br>
valid_lft forever preferred_lft forever<br>
root@miraculix:/tmp# ip route<br>
default dev wwan0.3 scope link <br>
root@miraculix:/tmp# ip route show table 3<br>
default dev vpp1host scope link <br>
root@miraculix:/tmp# ip rule<br>
0: from all lookup local<br>
1000: from all iif wwan0.3 lookup 3<br>
32766: from all lookup main<br>
32767: from all lookup default<br>
<br>
<br>
<br>
<br>
<br>
<br>
Bjørn<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr">Regards,<div><br></div><div>Sassy Natan</div><div>972-(0)54-2203702</div></div></div></div></div>