<div><div dir="ltr">Hi Again,<div><br></div><div>I am still fighting with this and didn't manage to get this working.</div><div><br></div><div dir="auto">Spend over 10 hours today and yesterday without any luck.</div><div dir="auto"><br></div><div dir="auto">Maybe someone can help?</div><div dir="auto"><br></div><div dir="auto">Thank you 🙏 </div></div></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Sep 30, 2020 at 12:28 PM Sassy Natan <<a href="mailto:sassyn@gmail.com" target="_blank">sassyn@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div dir="ltr">Dear Bjørn,<div><br></div><div>Thank you so much for your effort and detailed information on how to shorten out this issue.</div><div>I really appreciate this! </div><div><br></div><div>Apologies for the late reply.</div><div>We had a holiday here in Israel, and with the covid-19 situation (we in the middle of the second wave) I'm really losing my head.</div><div>The <br><br><br><br>situation here in Israel is really bad compared to the rest of the world :-(</div><div><br></div><div>Again, thank you so much! Your solution opened my mind to new things and ideas.<br></div><div><br></div><div>However, I tried out your solution, and It doesn't work on my setup.</div><div>I'm using a clean ubuntu 18.04 with vpp version 18.07-release without any luck.</div><div><br></div><div>I have compiled an asciinema of the entire process, maybe you can take a quick look?</div><div><br></div><div> <a href="http://goog_1496068446" target="_blank">https://asciinema.org/a/05VG1Xhl3kYOydE685wjphoqf<br></a></div><div><a href="https://asciinema.org/a/3sOWj8x5YBWnT5mNUgxiIHmUw" target="_blank"> https://asciinema.org/a/3sOWj8x5YBWnT5mNUgxiIHmUw</a><br></div><div><br></div><div><br></div><div>Here is also a quick figure of the topology</div><div><div><img src="https://mail.google.com/mail/?ui=2&ik=795e8a1046&attid=0.1&th=174ea11547f0517b&view=fimg&rm=174ea11547f0517b&sz=w1600-h1000&attbid=ANGjdJ95ZyFO2AaLAhod_aYKnYE8M_df7N8CKJNdyv3AeqZ8ZNzrsTlvo9xyciIAuaA93O_0nZYR0JIunrUcHZ7vW84QZSa38gr3pt3OmO-Pd1Cx7rwydiL8SID5xLA&disp=emb&realattid=ii_kfp6hiss0&zw" alt="image.png" style="width:812px;max-width:100%"><br></div></div><div><br></div><div> You were saying "There are a gazillion other ways to achieve the same" are you referring to the "two-way default routing"? </div><div>Or you mean the entire process.</div><div><br></div><div>As I said your solution is very creative - but even though I spent a long time on this - I didn't manage to come up with an alternative. </div><div><br></div><div>Even with your setup, I did try to do a long debug to it.</div><div>I used tcpdump on all interfaces (including tracing out the ethernet address "-e"), but nothing I can say will provide good feedback here.</div><div><br></div><div><br></div><div>Will be very thankful if someone can drop a comment here.</div><div><br></div><div>Thank you.</div><div>Sassy</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Sep 23, 2020 at 4:03 PM Bjørn Mork <<a href="mailto:bjorn@mork.no" target="_blank">bjorn@mork.no</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">FWIW, I just had to try out how much it takes to connect a host owned<br><br><br>LTE netdev to VPP. That's actually pretty easy. But you end forwarding<br><br><br>between the VPP host interface and the LTE netdev in Linux, which I<br><br><br>guess is something you eventually might like to avoid. I assume this is<br><br><br>pretty easy to implement as a VPP plugin or similar. It's not related<br><br><br>to the actual functionality here anyway.<br><br><br><br><br><br>My demo setup is a slight modification of the start of the VPP tutorial:<br><br><br><a href="https://fd.io/docs/vpp/v2005/gettingstarted/progressivevpp/interface.html" rel="noreferrer" target="_blank">https://fd.io/docs/vpp/v2005/gettingstarted/progressivevpp/interface.html</a><br><br><br><br><br><br><br><br><br>What I did was - on the host:<br><br><br><br><br><br>1) Create veth pair according to tutorial instructions:<br><br><br><br><br><br> ip link add name vpp1out type veth peer name vpp1host<br><br><br><br><br><br><br><br><br>2) Create a new sub-interface for MBIM session ID 3:<br><br><br><br><br><br> ip link add wwan0.3 link wwan0 type vlan id 3<br><br><br><br><br><br><br><br><br>3) Create a new network namespace and put both these interfaces there:<br><br><br><br><br><br> ip netns add vpplte<br><br><br> ip link set vpp1host netns vpplte<br><br><br> ip link set wwan0.3 netns vpplte<br><br><br><br><br><br>4) Start a shell in the network namespace and execute the remaining host<br><br><br> commands there:<br><br><br><br><br><br> ip netns exec vpplte /bin/bash<br><br><br><br><br><br><br><br><br>5) Set all links up:<br><br><br><br><br><br> ip link set lo up<br><br><br> ip link set vpp1host up<br><br><br> ip link set wwan0.3 up<br><br><br><br><br><br><br><br><br>6) Enable proxy-arp and forwarding<br><br><br><br><br><br> echo 1 > /proc/sys/net/ipv4/conf/vpp1host/proxy_arp<br><br><br> echo 1 > /proc/sys/net/ipv4/conf/vpp1host/forwarding<br><br><br> echo 1 > /proc/sys/net/ipv4/conf/wwan0.3/forwarding<br><br><br><br><br><br><br><br><br>7) set up two-way default routing<br><br><br><br><br><br> ip route add default dev wwan0.3<br><br><br> ip route add default dev vpp1host table 3<br><br><br> ip rule add pref 1000 iif wwan0.3 lookup 3<br><br><br><br><br><br>8) Disable arp on the veth endpoint<br><br><br><br><br><br> ip link set dev vpp1host arp off<br><br><br><br><br><br><br><br><br>9) Connect the session, and note the assigned address<br><br><br><br><br><br> mbimcli -p -d /dev/cdc-wdm0 --connect=apn=telenor.smart,session-id=3,ip-type=ipv4v6<br><br><br><br><br><br><br><br><br>This is the output I got, to be used in the VPP shell below:<br><br><br><br><br><br>[/dev/cdc-wdm0] Successfully connected<br><br><br><br><br><br>[/dev/cdc-wdm0] Connection status:<br><br><br> Session ID: '3'<br><br><br> Activation state: 'activated'<br><br><br> Voice call state: 'none'<br><br><br> IP type: 'ipv4v6'<br><br><br> Context type: 'internet'<br><br><br> Network error: 'unknown'<br><br><br><br><br><br>[/dev/cdc-wdm0] IPv4 configuration available: 'address, gateway, dns, mtu'<br><br><br> IP [0]: '<a href="http://10.169.198.6/30" rel="noreferrer" target="_blank">10.169.198.6/30</a>'<br><br><br> Gateway: '10.169.198.5'<br><br><br> DNS [0]: '193.213.112.4'<br><br><br> DNS [1]: '130.67.15.198'<br><br><br> MTU: '1500'<br><br><br><br><br><br>[/dev/cdc-wdm0] IPv6 configuration available: 'address, gateway, dns, mtu'<br><br><br> IP [0]: '2a02:2121:2c0:e913:392d:3e46:cf98:4ca3/64'<br><br><br> Gateway: '2a02:2121:2c0:e913:70a8:2a1c:dc62:2022'<br><br><br> DNS [0]: '2001:4600:4:fff::52'<br><br><br> DNS [1]: '2001:4600:4:1fff::52'<br><br><br> MTU: '1540'<br><br><br><br><br><br><br><br><br><br><br><br>In the VPP shell:<br><br><br><br><br><br>10) Create host interface and set link up as instructed by the tutorial<br><br><br><br><br><br> create host-interface name vpp1out<br><br><br> set int state host-vpp1out up<br><br><br><br><br><br><br><br><br>11) Assign the IP address from the mbimcli command above:<br><br><br><br><br><br><br><br><br> set int ip address host-vpp1out <a href="http://10.169.198.6/30" rel="noreferrer" target="_blank">10.169.198.6/30</a><br><br><br><br><br><br><br><br><br>12) Set a default route (or whatever you want) via some fake gateway<br><br><br> address on the other end of that veth pair - the one suggested by<br><br><br> the modem is fine:<br><br><br><br><br><br> ip route add 0/0 via 10.169.198.5<br><br><br><br><br><br><br><br><br>13) ping an address on the other side of the LTE link:<br><br><br><br><br><br><br><br><br> vpp# ping 8.8.8.8 <br><br><br> 116 bytes from <a href="http://8.8.8.8" rel="noreferrer" target="_blank">8.8.8.8</a>: icmp_seq=1 ttl=114 time=68.0229 ms<br><br><br> 116 bytes from <a href="http://8.8.8.8" rel="noreferrer" target="_blank">8.8.8.8</a>: icmp_seq=2 ttl=114 time=48.9930 ms<br><br><br> 116 bytes from <a href="http://8.8.8.8" rel="noreferrer" target="_blank">8.8.8.8</a>: icmp_seq=3 ttl=114 time=52.8561 ms<br><br><br> 116 bytes from <a href="http://8.8.8.8" rel="noreferrer" target="_blank">8.8.8.8</a>: icmp_seq=4 ttl=114 time=51.5395 ms<br><br><br> 116 bytes from <a href="http://8.8.8.8" rel="noreferrer" target="_blank">8.8.8.8</a>: icmp_seq=5 ttl=114 time=99.0668 ms<br><br><br><br><br><br> Statistics: 5 sent, 5 received, 0% packet loss<br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>This was a simple abuse of a dedicated network namespace in the Linux<br><br><br>host, where it is easy to do two-way default routing without having to<br><br><br>care about addressing and routes at all. There are a gazillion other<br><br><br>ways to achieve the same,<br><br><br><br><br><br>There is no way to know that there is anything special about the<br><br><br>host-vpp1out interface from the VPP shell. It looks like any other host<br><br><br>interface. But you'll obviously have to do something on the host if you<br><br><br>are going to run anything fancy over it. But that's normally not the<br><br><br>use case for an LTE connection anyway, I guess.<br><br><br><br><br><br><br><br><br>vpp# show interface <br><br><br> Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count <br><br><br>host-vpp1out 1 up 9000/0/0/0 rx packets 135<br><br><br> rx bytes 22334<br><br><br> tx packets 61<br><br><br> tx bytes 4874<br><br><br> drops 146<br><br><br> ip4 15<br><br><br> ip6 32<br><br><br>local0 0 down 0/0/0/0 <br><br><br>vpp# show ip neighbor<br><br><br> Time IP Flags Ethernet Interface <br><br><br> 4095.4097 10.169.198.5 D fe:cd:9c:57:a9:da host-vpp1out<br><br><br>vpp# show ip fib <br><br><br>ipv4-VRF:0, fib_index:0, flow hash:[src dst sport dport proto ] epoch:0 flags:none locks:[adjacency:1, recursive-resolution:1, default-route:1, nat-hi:2, ]<br><br><br><a href="http://0.0.0.0/0" rel="noreferrer" target="_blank">0.0.0.0/0</a><br><br><br> unicast-ip4-chain<br><br><br> [@0]: dpo-load-balance: [proto:ip4 index:1 buckets:1 uRPF:14 to:[60:5760]]<br><br><br> [0] [@12]: dpo-load-balance: [proto:ip4 index:13 buckets:1 uRPF:12 to:[0:0] via:[60:5760]]<br><br><br> [0] [@5]: ipv4 via 10.169.198.5 host-vpp1out: mtu:9000 next:3 fecd9c57a9da02fe5a6a76810800<br><br><br><a href="http://0.0.0.0/32" rel="noreferrer" target="_blank">0.0.0.0/32</a><br><br><br> unicast-ip4-chain<br><br><br> [@0]: dpo-load-balance: [proto:ip4 index:2 buckets:1 uRPF:1 to:[0:0]]<br><br><br> [0] [@0]: dpo-drop ip4<br><br><br><a href="http://10.169.198.4/32" rel="noreferrer" target="_blank">10.169.198.4/32</a><br><br><br> unicast-ip4-chain<br><br><br> [@0]: dpo-load-balance: [proto:ip4 index:10 buckets:1 uRPF:9 to:[0:0]]<br><br><br> [0] [@0]: dpo-drop ip4<br><br><br><a href="http://10.169.198.5/32" rel="noreferrer" target="_blank">10.169.198.5/32</a><br><br><br> unicast-ip4-chain<br><br><br> [@0]: dpo-load-balance: [proto:ip4 index:13 buckets:1 uRPF:12 to:[0:0] via:[60:5760]]<br><br><br> [0] [@5]: ipv4 via 10.169.198.5 host-vpp1out: mtu:9000 next:3 fecd9c57a9da02fe5a6a76810800<br><br><br><a href="http://10.169.198.4/30" rel="noreferrer" target="_blank">10.169.198.4/30</a><br><br><br> unicast-ip4-chain<br><br><br> [@0]: dpo-load-balance: [proto:ip4 index:9 buckets:1 uRPF:8 to:[0:0]]<br><br><br> [0] [@4]: ipv4-glean: host-vpp1out: mtu:9000 next:1 ffffffffffff02fe5a6a76810806<br><br><br><a href="http://10.169.198.6/32" rel="noreferrer" target="_blank">10.169.198.6/32</a><br><br><br> unicast-ip4-chain<br><br><br> [@0]: dpo-load-balance: [proto:ip4 index:12 buckets:1 uRPF:13 to:[15:1440]]<br><br><br> [0] [@2]: dpo-receive: 10.169.198.6 on host-vpp1out<br><br><br><a href="http://10.169.198.7/32" rel="noreferrer" target="_blank">10.169.198.7/32</a><br><br><br> unicast-ip4-chain<br><br><br> [@0]: dpo-load-balance: [proto:ip4 index:11 buckets:1 uRPF:11 to:[0:0]]<br><br><br> [0] [@0]: dpo-drop ip4<br><br><br><a href="http://224.0.0.0/4" rel="noreferrer" target="_blank">224.0.0.0/4</a><br><br><br> unicast-ip4-chain<br><br><br> [@0]: dpo-load-balance: [proto:ip4 index:4 buckets:1 uRPF:3 to:[0:0]]<br><br><br> [0] [@0]: dpo-drop ip4<br><br><br><a href="http://240.0.0.0/4" rel="noreferrer" target="_blank">240.0.0.0/4</a><br><br><br> unicast-ip4-chain<br><br><br> [@0]: dpo-load-balance: [proto:ip4 index:3 buckets:1 uRPF:2 to:[0:0]]<br><br><br> [0] [@0]: dpo-drop ip4<br><br><br><a href="http://255.255.255.255/32" rel="noreferrer" target="_blank">255.255.255.255/32</a><br><br><br> unicast-ip4-chain<br><br><br> [@0]: dpo-load-balance: [proto:ip4 index:5 buckets:1 uRPF:4 to:[0:0]]<br><br><br> [0] [@0]: dpo-drop ip4<br><br><br><br><br><br><br><br><br><br><br><br>And the Linux host view:<br><br><br><br><br><br>root@miraculix:/tmp# ip link<br><br><br>1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000<br><br><br> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br><br><br>10: vpp1host@if11: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000<br><br><br> link/ether fe:cd:9c:57:a9:da brd ff:ff:ff:ff:ff:ff link-netnsid 0<br><br><br>12: wwan0.3@if3: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000<br><br><br> link/ether 42:0a:0d:ab:b4:f5 brd ff:ff:ff:ff:ff:ff link-netnsid 0<br><br><br>root@miraculix:/tmp# ip addr<br><br><br>1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000<br><br><br> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br><br><br> inet <a href="http://127.0.0.1/8" rel="noreferrer" target="_blank">127.0.0.1/8</a> scope host lo<br><br><br> valid_lft forever preferred_lft forever<br><br><br> inet6 ::1/128 scope host <br><br><br> valid_lft forever preferred_lft forever<br><br><br>10: vpp1host@if11: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000<br><br><br> link/ether fe:cd:9c:57:a9:da brd ff:ff:ff:ff:ff:ff link-netnsid 0<br><br><br> inet6 fe80::fccd:9cff:fe57:a9da/64 scope link <br><br><br> valid_lft forever preferred_lft forever<br><br><br>12: wwan0.3@if3: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000<br><br><br> link/ether 42:0a:0d:ab:b4:f5 brd ff:ff:ff:ff:ff:ff link-netnsid 0<br><br><br> inet6 2a02:2121:2c0:e913:400a:dff:feab:b4f5/64 scope global dynamic mngtmpaddr <br><br><br> valid_lft forever preferred_lft forever<br><br><br> inet6 fe80::400a:dff:feab:b4f5/64 scope link <br><br><br> valid_lft forever preferred_lft forever<br><br><br>root@miraculix:/tmp# ip route<br><br><br>default dev wwan0.3 scope link <br><br><br>root@miraculix:/tmp# ip route show table 3<br><br><br>default dev vpp1host scope link <br><br><br>root@miraculix:/tmp# ip rule<br><br><br>0: from all lookup local<br><br><br>1000: from all iif wwan0.3 lookup 3<br><br><br>32766: from all lookup main<br><br><br>32767: from all lookup default<br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>Bjørn<br><br><br></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr">Regards,<div><br></div><div>Sassy Natan</div><div>972-(0)54-2203702</div></div></div></div></div><br><br></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr">Regards,<div><br></div><div>Sassy Natan</div><div>972-(0)54-2203702</div></div></div></div></div><br><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr">Regards,<div><br></div><div>Sassy Natan</div><div>972-(0)54-2203702</div></div></div></div></div>