[ENHANCEMENT] Jail VNET with Netgraph

[PetruGarstea]

Is your feature request related to a problem? Please describe.
N/A

Describe the solution you'd like
N/A

Describe alternatives you've considered
Any plans to implement VNET Jails with Netgraph ?

ref:

• /usr/share/examples/jails/jng
• https://wiki.freebsd.org/TomMarcoen/JailNetworking

Additional context
In my setup the current traffic accounting is done with ng_netflow, traffic passed from epair interfaces via if_bridge is not being counted, I am wondering if change the networking to jng would ng_netflow count the traffic originated from VNET jails.

[PetruGarstea]

Greetings,

I spent last week on exploring netgraph networking for jails and here what I want to share/propose for this ENHANCEMENT

1. Setting up netgraph or if_bridge networking should come from the main config file in the similar way how the user chooses the file system, between ZFS or UFS

2. In my POC tests ng_bridge and if_bridge should not co-exits together on the same host, that means user has only one option to choose and for the migrated jail it needs to be detected what type of networking on the host is running.

3. Netgraph ethernet interfaces for jails can be managed by jng script that is located under /usr/share/examples/jails/jng

4. Netgraph modules have to be loaded into kernel,

cat /boot/loader.conf

netgraph_load="YES"
ng_netflow_load="YES"
ng_ksocket_load="YES"
ng_ether_load="YES"
ng_bridge_load="YES"
ng_eiface_load="YES"
ng_socket_load="YES"

5. netgraph config, that creates bridge and exports netflow that are passes via main physical ethernet interface

# send from physical interface to netflow node
ngctl mkpeer lan: netflow lower iface0
# name netflow node as netflow
ngctl name lan:lower netflow
# return from netflow to physical interface
ngctl connect lan: netflow: upper out1
# connect netflow node with bridge node
ngctl mkpeer netflow: bridge out0 link0
# name bridge node as lanbridge
ngctl name netflow:out0 lanbridge
# connect bridge node with netflow node
ngctl connect lanbridge: netflow: link1 iface1
# create management interface for tests 
ngctl mkpeer lanbridge: eiface link2 ether
# name management interface
ngctl name lanbridge:link2 ngmgmt
# using netflow v9
ngctl mkpeer netflow: ksocket export9 inet/dgram/udp
# configure lan 
ngctl msg lan: setpromisc 1
ngctl msg lan: setautosrc 0
# collecting ingress and egress
ngctl msg netflow: setconfig {iface=0 conf=11}
ngctl msg netflow: setconfig {iface=1 conf=11}
# exporting netflow
ngctl msg netflow:export9 connect inet/${IP_COLLECTOR}:${PORT_COLLECTOR}

6. With netgraph it is possible to extend the use case, for instance to add VxLAN isolation for distributed workloads just by adding necessary nodes to existing graph.

PS: IMHO, netgraph is highly extendable :)

Thanks for reading.

Cheers,
Petru

[patmaddox]

I haven't used netgraph yet - still need to look into it - but I have used epair and bridge. My experience is that those scripts under /usr/share/examples/jails/ are good as examples and didn't work when I tried to call them from jail.conf. Much better is to take out the specific lines and adapt them. FYI for anyone looking into this.

I've found vnet networking to be really good, so I'll be happy to experiment with this in Bastille.

What are the advantages of netgraph over epair? I couldn't find much information on that. I vaguely remember seeing a post somewhere that said that epair is single-threaded for all instances which seemed like quite a limitation... but don't quote me on that.

[kmgolden]

@patmaddox regarding advantages: In my small lab testing, I found netgraph to significantly increase performance for running my database, web server, and pf jails on the same host. It is also more configurable than epair. See link below for more details: https://people.freebsd.org/~julian/netgraph.html

I don't have the specific data now, but I could probably set up some tests with enough time :)

[bmac2]

@PetruGarstea is this still a valid feature request?? I want to confirm before I put it on the list to work on it.

@cedwards any thoughts on this one?? Seems a little out of scope of the bastille project to me. A template for this I could see. NOT bastille code changes.

[bmac2]

Upon more reading on this, this request should be solved using a template. This would allow anyone with their jails setup as vnet jails to apply a template for netgraph to that jail(s).

Action item: build a netgraph template

[tschettervictor]

Agreed.

[bmac2]

per documentation, the kernel must be compiled with netgraph enabled in it.