OVS Provider not visible under Network Service Providers for guest network

[shoaibjdev]

ISSUE TYPE

• Bug Report OVS Provider not visible under Network Service Providers for guest network

• Improvement Request

• Enhancement Request

• Feature Idea

• Documentation Report

• Other

COMPONENT NAME

UI

CLOUDSTACK VERSION

4.12

CONFIGURATION

OS / ENVIRONMENT

RHEL 7.6

SUMMARY

Create a Zone with Management + Storage network with Physical Network 1 and Guest network in Physical Network 2 with Open vSwitch GRE for network isolation.
Open vSwitch is installed on both KVM Hosts and cloudbr0 is bridged to physical interface 1 (management + storage) and cloudbr1 is bridged physical interface 2 (guest)

After Zone creation, when navigated to Zone --> Physical Network 2 ---> Network Service Provider ---> Configure ---> OVS provider is missing

EXPECTED RESULTS

OVS should be listed under Network Service Provider and should be enabled

ACTUAL RESULTS

OVS network service provider is missing

![image](https://user-images.githubusercontent.com/3480900/72224860-0265a500-3590-11ea-9549-a3f920c503ef.png)

![image](https://user-images.githubusercontent.com/3480900/72224862-05f92c00-3590-11ea-989b-ffcd51d45dbb.png)

[weizhouapache]

@shoaibjdev
I have a setup with GRE. Ovs is enabled by default.

(local) 🐵 > list physicalnetworks
{
  "count": 1,
  "physicalnetwork": [
    {
      "broadcastdomainrange": "ZONE",
      "id": "a00b2b4c-945a-41c9-bf62-454a49cefc47",
      "isolationmethods": "GRE",
      "name": "OVS",
      "state": "Enabled",
      "vlan": "2000-2100",
      "zoneid": "7ff7c5bd-c41f-40ca-ba31-3b7bde583d38"
    }
  ]
}

(local) 🐵 > list networkserviceproviders physicalnetworkid="a00b2b4c-945a-41c9-bf62-454a49cefc47" name=ovs
{
  "count": 1,
  "networkserviceprovider": [
    {
      "canenableindividualservice": true,
      "id": "c08f3e76-00c6-489d-9b37-f9cd7d304cc3",
      "name": "Ovs",
      "physicalnetworkid": "a00b2b4c-945a-41c9-bf62-454a49cefc47",
      "servicelist": [
        "Lb",
        "StaticNat",
        "PortForwarding"
      ],
      "state": "Enabled"
    }
  ]
}

[shoaibjdev]

@weizhouapache I can confirm from database, it's there but if you verify from ACS UI, it's not in the list of Network Providers for the Zone's physical network.

Kindly share screenshot of UI if it's visible on your end.

[shoaibjdev]

Also the reason I raised it as GRE doesn't seem to work, and as it doesn't appear under UI so I doubt if there is something broken.

If you have only 1 KVM node and it's the node where Virtual Router is running, then instances will get IP from the VNI specified during setup.
But if you add 2nd KVM node, no matter what you do, it will never get an IP from Virtual Router.
If now I migrate virtual router to this new KVM node, instances will get the IPs.

So basically, OVS GRE tunnel is not working on ACS 4.12, I have asked @andrijapanicsb to advise if GRE is still in use on ACS or should we move away from it and trying some other option for OVS tunnel to work, awaiting his response from that thread.

[weizhouapache]

@weizhouapache I can confirm from database, it's there but if you verify from ACS UI, it's not in the list of Network Providers for the Zone's physical network.

Kindly share screenshot of UI if it's visible on your end.

I confirm ovs is not visible in the list of network provider on UI.

[weizhouapache]

Also the reason I raised it as GRE doesn't seem to work, and as it doesn't appear under UI so I doubt if there is something broken.

If you have only 1 KVM node and it's the node where Virtual Router is running, then instances will get IP from the VNI specified during setup.
But if you add 2nd KVM node, no matter what you do, it will never get an IP from Virtual Router.
If now I migrate virtual router to this new KVM node, instances will get the IPs.

So basically, OVS GRE tunnel is not working on ACS 4.12, I have asked @andrijapanicsb to advise if GRE is still in use on ACS or should we move away from it and trying some other option for OVS tunnel to work, awaiting his response from that thread.

@shoaibjdev do you use default network offering or new offering created by following instruction http://docs.cloudstack.apache.org/projects/archived-cloudstack-getting-started/en/latest/networking/ovs-plugin.html#network-offerings ?

[shoaibjdev]

@weizhouapache - Really appreciate your help and assistance on this. We have created the custom offering and tried with it as well, but Virtual routers didn't came up. So we went with default network offering.

Would you be kind enough, to direct us in the right direction as per best practices in production what is in general is adopted by customers running ACS in production as of today?

Our requirement for networking is to let the tenants create their own private networks running on multiple Hypervisor nodes and all instances running on differents nodes should be able to reach each other as longs its under the same super CIDR they defined. And traffic from other tenants should not interfere with others.

We want to avoid using VLAN tags from the physical networks as it wouldn't make any sense for us to plug additional VLANs when we scale the hardware, and we do not want tenants to be part of the physical network ports.

So we decided to go with SDN solution and let tenants create their own networks identified by VNI's, either VXLAN or GRE from OVS works very well and mature solution for vanilla KVM and OVS setup without ACS. But we see some limitation on ACS as OVS doesn't work with VPC which would have solved this scenario.

Challenge we have is just getting the OVS tunnel to work across Hypervisor hosts. Please advise how we can achieve this on ACS.

[svenvogel]

@shoaibjdev we use OVS. we dont got GRE run. maybe its broken. i dont know.

[shoaibjdev]

@shoaibjdev we use OVS. we dont got GRE run. maybe its broken. i dont know.

@svenvogel Thanks for the confirmation. Could you please advise how did you manage to handle to handle the traffic across hypervisor hosts, if not GRE?

[weizhouapache]

@shoaibjdev are there some error logs in /var/log/cloudstack/agent/ovstunnel.log similar as following ?

2019-11-28 14:11:41,678 - Tunnel interface validated:['/usr/bin/ovs-vsctl', 'get', 'interface', '433982a7-cada-4855-a72b-4b30344f645d', 'options:remote_ip']
2019-11-28 14:11:41,678 - Executing:['/usr/bin/ovs-vsctl', 'get', 'interface', '433982a7-cada-4855-a72b-4b30344f645d', 'ofport']
2019-11-28 14:11:41,698 - Executing:['/usr/bin/ovs-vsctl', 'get', 'bridge', 'OVSTunnel2046', 'other_config:is-ovs-tun-network']
2019-11-28 14:11:41,704 - Executing:['/usr/bin/ovs-vsctl', 'get', 'bridge', 'OVSTunnel2046', 'other_config:is-ovs_vpc_distributed_vr_network']
2019-11-28 14:11:41,709 - The command exited with the error code: 1 (stderr output:ovs-vsctl: no key "is-ovs_vpc_distributed_vr_network" in Bridge record "OVSTunnel2046" column other_config
)
2019-11-28 14:11:41,709 - An unexpected error occured. Rolling back
2019-11-28 14:11:41,709 - Deleting GRE interface
2019-11-28 14:11:41,709 - Executing:['/usr/bin/ovs-vsctl', 'del-port', 'OVSTunnel2046', 't2046-4-3']
2019-11-28 14:12:22,157 - About to manually create the bridge:OVSTunnel2046

[yadvr]

@shoaibjdev can you test latest master with Primate?

[DaanHoogland]

@shoaibjdev are you still working on this issue? As suggested implicitly, can you try a cli tool like cloudmonkey to find the provider and does that work?

[DaanHoogland]

@shoaibjdev I'm closing this issue as it is not active. You can reopenif you wish.