NGAP Load Balancing with LoxiLB and free5GC (L7 load-balancer)

[williamlin0518]

I'm trying to implement SCTP load balancing for Free5GC AMF using LoxiLB, but encountering connection issues. Here's my setup:

this is how I implement in very detail NGAP Load Balancing with LoxiLB using free5GC this note open to edit!

welcome any suggestions for improving this setup, corrections to my configurations, or alternative approaches. Feel free to point out if I'm misunderstanding any

If you need any additional information or logs, I'll be happy to provide them.

Environment:

• Single Kubernetes cluster running Free5GC components
• LoxiLB running as standalone Docker container
• Two AMF instances
• UERANSIM for testing

Configuration:

1. LoxiLB VIP: 10.100.50.249
2. AMF1: 10.100.50.251
3. AMF2: 10.100.50.252
4. SCTP Port: 38412

Current Setup:

1. LoxiLB is deployed and running:

docker run -u root --cap-add SYS_ADMIN --restart unless-stopped --privileged -dit -v /dev/log:/dev/log --entrypoint=/root/loxilb-io/loxilb/loxilb --net=host --name loxilb ghcr.io/loxilb-io/loxilb:scp --proxyonlymode

Deploy kube-loxilb

1. loxiURL: the IP address of the machine running the LoxiLB data-plane and to talk to the external LoxiLB instance.
   Host network IP: 192.168.56.106 (enp0s8)

        args:
        - --loxiURL=http://192.168.56.106:11111
        - --cidrPools=defaultPool=10.100.50.249/32
        #- --monitor
        #- --setBGP=64512
        #- --extBGPPeers=50.50.50.1:65101,51.51.51.1:65102
        #- --setRoles
        - --setLBMode=2
        #- --config=/opt/loxilb/agent/kube-loxilb.conf

Modify the AMF service

Modify the AMF service to use LoadBalancer type and add LoxiLB annotations

free5gc/values.yaml

change:

  amf:
    n2if:  # NGAP
      ipAddress: 10.100.50.250
    service:
      ngap:
        enabled: false
        name: amf-n2
        port: 38412
        nodeport: 31412
        protocol: SCTP
        type: NodePort

to:

  amf:
    n2if:  # NGAP
      ipAddress: 10.100.50.251
    service:
      ngap:
        enabled: true
        name: amf-n2
        port: 38412
        protocol: SCTP
        type: LoadBalancer
        loadBalancerClass: "loxilb.io/loxilb"  # 
        annotations:
          loxilb.io/probetype: "none"
          loxilb.io/lbmode: "fullproxy"
          loxilb.io/epselect: "n2"
          loxilb.io/lbclass: "loxilb.io/loxilb" 
  amf2:
    n2if:  # NGAP
      ipAddress: 10.100.50.252
    service:
      ngap:
        enabled: true
        name: amf2-n2
        port: 38412
        protocol: SCTP
        type: LoadBalancer
        loadBalancerClass: "loxilb.io/loxilb"  # 
        annotations:
          loxilb.io/probetype: "none"
          loxilb.io/lbmode: "fullproxy"
          loxilb.io/epselect: "n2"
          loxilb.io/lbclass: "loxilb.io/loxilb" 

gnb can't find AMF

###have no ct

vboxuser@test1000:~/work/free5gc-helm/charts$ docker exec -it loxilb loxicmd get lb -o wide
|    EXT IP     | SEC IPS | HOST | PORT  | PROTO |                        NAME                         | MARK | SEL |   MODE    | ENDPOINT  | EPORT | WEIGHT | STATE | COUNTERS |
|---------------|---------|------|-------|-------|-----------------------------------------------------|------|-----|-----------|-----------|-------|--------|-------|----------|
| 10.100.50.249 |         |      | 38412 | sctp  | free5gc_free5gc-helm-free5gc-amf2-amf2-n2:llb-inst0 |    0 | n2  | fullproxy | 10.0.2.15 | 31413 |      1 | -     | 0:0      |


vboxuser@test1000:~/work/free5gc-helm/charts$ docker exec -it loxilb loxicmd get ct
| SERVICE NAME | DESTIP | SRCIP | DPORT | SPORT | PROTO | STATE | ACT | PACKETS | BYTES |
|--------------|--------|-------|-------|-------|-------|-------|-----|---------|-------|

so I create lb by my own and still can't establish connection

docker exec -it loxilb loxicmd create lb 10.100.50.249 --sctp=38412:38412 --endpoints=10.100.50.251:1,10.100.50.252:1 --mode=fullproxy

vboxuser@test1000:~/work/free5gc-helm$ docker exec -it loxilb loxicmd create lb 10.100.50.249 --sctp=38412:38412 --endpoints=10.100.50.251:1,10.100.50.252:1 --mode=fullproxy

ProtoPortpair: map[sctp:[38412:38412]]
Debug: response.StatusCode: 200
Success
vboxuser@test1000:~/work/free5gc-helm$ docker exec -it loxilb loxicmd get lb -o wide
|    EXT IP     | SEC IPS | HOST | PORT  | PROTO |                        NAME                         | MARK | SEL |   MODE    |   ENDPOINT    | EPORT | WEIGHT | STATE | COUNTERS |
|---------------|---------|------|-------|-------|-----------------------------------------------------|------|-----|-----------|---------------|-------|--------|-------|----------|
| 10.100.50.249 |         |      | 38412 | sctp  | free5gc_free5gc-helm-free5gc-amf2-amf2-n2:llb-inst0 |    0 | rr  | fullproxy | 10.100.50.251 | 38412 |      1 | -     | 0:0      |
|               |         |      |       |       |                                                     |      |     |           | 10.100.50.252 | 38412 |      1 | -     | 0:0      |
vboxuser@test1000:~/work/free5gc-helm$ 

Questions:

1. Is my LoxiLB configuration correct for SCTP load balancing?

2. Why isn't the SCTP connection being established through the VIP?

3. the way I duplicate amf may wrong? like use NodePort instead of LoadBalancer type for AMF services?

[backguynn]

Hello, I reviewed the document you posted about "NGAP Load Balancing with LoxiLB using Free5GC."
It seems that both the gNB and UE are also running as Kubernetes pods. Additionally, the IP addresses 10.100.50.251 and 10.100.50.252 assigned to the AMF are likely pod IPs. In this case, it could cause a potential issue.

LoxiLB is running independently outside of Kubernetes. In this case, Kubernetes likely doesn't have a route configuration for directing packets to the external IP 10.100.50.249 that you've specified. (The packets are probably being sent via the default route.)

In other words, I believe you need to add a route to the Kubernetes nodes to ensure that packets sent from the gNB are correctly delivered to LoxiLB.

[williamlin0518]

vboxuser@test1000:~/work/free5gc-helm/charts$ ip route | grep 10.100.50.249
10.100.50.249 via 192.168.56.106 dev enp0s8 
vboxuser@test1000:~/work/free5gc-helm/charts$ docker exec -it loxilb loxicmd get lb -o wide
|    EXT IP     | SEC IPS | HOST | PORT  | PROTO |                        NAME                         | MARK | SEL |   MODE    |   ENDPOINT    | EPORT | WEIGHT | STATE | COUNTERS |
|---------------|---------|------|-------|-------|-----------------------------------------------------|------|-----|-----------|---------------|-------|--------|-------|----------|
| 10.100.50.249 |         |      | 38412 | sctp  | free5gc_free5gc-helm-free5gc-amf2-amf2-n2:llb-inst0 |    0 | rr  | fullproxy | 10.100.50.251 | 38412 |      1 | -     | 0:0      |
|               |         |      |       |       |                                                     |      |     |           | 10.100.50.252 | 38412 |      1 | -     | 0:0      |
vboxuser@test1000:~/work/free5gc-helm/charts$ 

I can ping 10.100.50.249 now, but gnb can't not found amf.

vboxuser@test1000:~/work/free5gc-helm$ ping 10.100.50.249
PING 10.100.50.249 (10.100.50.249) 56(84) bytes of data.
64 bytes from 10.100.50.249: icmp_seq=1 ttl=64 time=0.037 ms
64 bytes from 10.100.50.249: icmp_seq=2 ttl=64 time=0.043 ms
^C
--- 10.100.50.249 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1012ms
rtt min/avg/max/mdev = 0.037/0.040/0.043/0.003 ms
vboxuser@test1000:~/work/free5gc-helm$ kubectl get services -n free5gc | grep amf
free5gc-helm-free5gc-amf-amf-n2     LoadBalancer   10.152.183.109   llb-10.100.50.249   38412:31412/SCTP   30m
free5gc-helm-free5gc-amf-service    ClusterIP      10.152.183.172   <none>              80/TCP             30m
free5gc-helm-free5gc-amf2-amf2-n2   LoadBalancer   10.152.183.46    llb-10.100.50.249   38412:31413/SCTP   30m
free5gc-helm-free5gc-amf2-service   ClusterIP      10.152.183.168   <none>              80/TCP             30m

Still have connectivity issues between gNB and AMF, maybe my method mix in-cluster and external cluster modes

[TrekkieCoder]

Kindly try creating the rules manually as follows (add "select=n2" ) as follows -

docker exec -it loxilb loxicmd create lb 10.100.50.249 --sctp=38412:38412 --endpoints=10.100.50.251:1,10.100.50.252:1 --mode=fullproxy --select=n2

Further I have some questions :

1. Can you ping to 10.100.50.251 and 10.100.50.252 from loxilb node ?
2. Can you ping to 10.100.50.249 from gNB ?
3. Are 10.100.50.251 and 10.100.50.252 the secondary IP of the AMF pods ?
4. Can Free5GC run multiple AMF instances in the same deployment (stateless mode) or need multiple instances of 5GC like Open5GS ?

Further we have some more enhancements in L7 NGAP support. Request to use the image ghcr.io/loxilb-io/loxilb:scp2

[williamlin0518]

1. no, They're specified values in Helm chart but not actual interfaces
2. Yes after add the VIP (10.100.50.249)on the interface ip addr add 10.100.50.249/32 dev enp0s8
3. No, these just configuration values in my Helm chart, not actual IP addresses of AMF pod IPs, so I should create endpoint with my pod ip?
4. I will check later!

thanks!

[TrekkieCoder]

Thanks @williamlin0518 @backguynn. The final blog is live here .