Enhancement: create Shared networks and VPC private gateways by users by weizhouapache · Pull Request #5730 · apache/cloudstack

weizhouapache · 2021-11-29T14:13:16Z

Description

Currently shared networks and VPC private gateways can only be created from ROOT admin.
This PR contains the following enhancement

1. Create shared network offering with specifyvlan=false

specifyvlan MUST be true for shared network offering for now
specifyvlan can be true or false with this PR.

2. Create shared network by non-ROOT users

If specifyvlan=true in shared network offering

vlan is required, it can be created/removed only by ROOT admin

if specifyvlan=false in shared network offering

All users can create shared network with an associated network (isolated networks), it will use the same vlan of associated network.
If associated network is not specified, cloudstack will allocate a vlan from zone guest vlan ranges to the shared network.

ROOT admin and Domain admins can create shared network for domain (acltype=domain) and account (acltype=account)
Normal users can create shared network for account (acltype=account)
Associated network MUST be an Isolated network
Associated network MUST be owned by same account (if acltype=account) or in the same domain (if acltype=domain)
Associated network MUST be on same physical network as the new shared network.

3. Create VPC private gateway by non-ROOT users

ALL users can create VPC private gateway with an associated network. it will use the same vlan of associated network.

Associated network MUST be an Isolated network
Associated network MUST be owned by same account
Associated network MUST be on same physical network as the private gateway.
The private gateway will use the new system network offering : System-Private-Gateway-Network-Offering-Without-Vlan

ROOT admin can create VPC private gateway with a specified vlan. It can only be removed by root admin either.
The private gateway will use the new system network offering : System-Private-Gateway-Network-Offering

4. List guest vlans in zone

An API is introduced to list all guest vlans: listGuestVlans . It is only available by root admin.

Guest vlans can be listed by

id
zoneid
physical network id
vnet (vlan id)
allocatedonly (true/false)

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Screenshots (if appropriate):

How Has This Been Tested?

weizhouapache · 2021-11-29T14:13:32Z

@blueorangutan package

blueorangutan · 2021-11-29T14:14:03Z

@weizhouapache a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-11-29T14:54:01Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 1797

weizhouapache · 2021-11-29T15:31:00Z

@blueorangutan test

blueorangutan · 2021-11-29T15:32:04Z

@weizhouapache a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-11-30T02:16:30Z

Trillian test result (tid-2594)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 37410 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5730-t2594-kvm-centos7.zip
Smoke tests completed. 90 look OK, 3 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_create_pvlan_network	`Error`	0.03	test_pvlan.py
test_01_isolated_persistent_network	`Error`	31.07	test_persistent_network.py
test_dedicateGuestVlanRange	`Error`	1.10	test_guest_vlan_range.py
ContextSuite context=TestDedicateGuestVlanRange>:teardown	`Error`	2.16	test_guest_vlan_range.py

weizhouapache · 2021-11-30T10:03:19Z

@blueorangutan package

blueorangutan · 2021-11-30T10:04:03Z

@weizhouapache a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-11-30T10:50:13Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 1800

weizhouapache · 2021-11-30T11:02:54Z

@blueorangutan package

…y if other ways do not work

weizhouapache · 2021-11-30T12:01:49Z

@blueorangutan package

blueorangutan · 2021-11-30T12:02:03Z

@weizhouapache a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-11-30T12:41:49Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 1802

weizhouapache · 2021-11-30T13:21:22Z

@blueorangutan test keepEnv

blueorangutan · 2021-11-30T13:22:03Z

@weizhouapache a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-11-30T23:18:04Z

Trillian test result (tid-2595)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34444 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5730-t2595-kvm-centos7.zip
Smoke tests completed. 92 look OK, 1 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_create_pvlan_network	`Error`	0.04	test_pvlan.py

GutoVeronezi · 2021-12-06T16:39:04Z

+        if (detail != null) {
+            Long associatedNetworkId = Long.valueOf(detail.getValue());
+            NetworkVO associatedNetwork = ApiDBUtils.findNetworkById(associatedNetworkId);
+            if (associatedNetwork != null) {
+                response.setAssociatedNetworkId(associatedNetwork.getUuid());
+                response.setAssociatedNetworkName(associatedNetwork.getName());
+            }
+        }
+


We could invert this if statement and add a return, to reduce code indentation.

blueorangutan · 2021-12-06T18:21:19Z

Trillian test result (tid-2621)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 35059 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5730-t2621-kvm-centos7.zip
Smoke tests completed. 89 look OK, 4 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_add_primary_storage_disabled_host	`Error`	0.75	test_primary_storage.py
test_01_primary_storage_nfs	`Error`	0.11	test_primary_storage.py
ContextSuite context=TestStorageTags>:setup	`Error`	0.19	test_primary_storage.py
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	32.95	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	`Failure`	55.48	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	`Failure`	40.28	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	0.04	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	73.81	test_kubernetes_clusters.py
test_02_list_snapshots_with_removed_data_store	`Error`	9.50	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	`Error`	9.50	test_snapshots.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	307.13	test_hostha_kvm.py

weizhouapache · 2021-12-07T12:13:22Z

@blueorangutan package

blueorangutan · 2021-12-07T12:14:04Z

@weizhouapache a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-12-07T13:03:28Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 1853

weizhouapache · 2021-12-07T13:12:14Z

@blueorangutan test keepEnv

blueorangutan · 2021-12-07T13:13:03Z

@weizhouapache a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-12-07T23:33:21Z

Trillian test result (tid-2634)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 35900 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5730-t2634-kvm-centos7.zip
Smoke tests completed. 93 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

…orks

weizhouapache · 2021-12-10T21:44:44Z

@blueorangutan package

blueorangutan · 2021-12-10T21:45:03Z

@weizhouapache a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-12-10T22:31:39Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 1871

weizhouapache · 2021-12-11T09:16:03Z

@blueorangutan test keepEnv

blueorangutan · 2021-12-11T09:17:03Z

@weizhouapache a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-12-11T20:30:03Z

Trillian test result (tid-2645)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 39063 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5730-t2645-kvm-centos7.zip
Smoke tests completed. 93 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

…orks

weizhouapache · 2021-12-15T07:42:36Z

#5769 contains all commits in this PR.
please review this PR and test with #5769.

…ther users

…same domain (#5769) * Enhancement: create Shared networks and VPC private gateways by users * UI bug fix: pass correct domainid in CreateSharedNetworkForm * Update #5730: fix test failure with test_guest_vlan_range.py * Update #5730: fix test failure with test_persistent_network.py * Update #5730: Add since to new API commands and API parameters * Update #5730: Get first physical network for VPC private gateway if other ways do not work * Update #5730: code optimization (return !offering.isSpecifyVlan()) * Update #5730: fix hard-coded network offering id in test_pvlan.py * Update #5730: skip access check on the network owner if the owner is ROOT/system * Update #5730: overlap check on cidr/startip/endip * Update #5730: add methods to get accountid/domainid of shared networks * Update #5730: improve integration tests * Update #5730: update as per GutoVeronezi's comments * Network Sharing: give network access permission to other accounts within a domain * network: update ip in lb/pf/dnat tables when update vm nic ip * Update #5757: create 3 separated methods for DNAT/LB/PF update * travis: install python3-setuptools * Network Sharing: update integration test * Update #5769: Remove NetworkPermission.Ops * Update #5769: Update as per Daan's comments * Update #5769: Update as per Suresh's comments * Update #5769: fix UI bug that accounts/projects are not listed * Update #5769: fix domain admin can deploy vm on L2 network of other users * Update #5769: Remove method listPermittedNetworkIdsByDomains in NetworkPermissionDao * Update #5769: Skip network operation permissions check for root admin * UI: fix create Isolated/L2 network form * Update #5730: fix create Shared network form * Update #5769: fix domain admin can deploy vm on L2 network of other users * test: fix test_storage_policy.py * Update #5769: fix remove_nic in test_network_permissions.py * Update #5769: extract some codes to a method * Update #5769: fix add/remove nic by domain admin * Update #5769: allow domain admin to enable/disable static nat and create port forwarding rules * Update #5769: update integration test * Update #5769: fix unit test AssignLoadBalancerTest.java * Update #5769: allow normal users to share network permission to other users on UI * Update #5769: fix small UI bug with label * Update #5769: Support L2 network as associated network * test: sleep 30s after restarting mgt server in test_kubernetes_supported_versions.py to fix test failures with test_secondary_storage.py * Update #5784: revert part of changes in #2420 * Update #5757: invert if condition to reduce code indentation * Update #5769: fix regular user cannot create L2 network * Update #5769: Add associated nework id and name in private gateway response * Update #5769: list networks by networkfilter=Account on UI * Update #5769: fix ui issue when list private gateways or create shared network if no isolated networks * Update #5769: fix vue ui warnings * Update #5679: add BaseResponseWithAssociatedNetwork and extract method setResponseAssociatedNetworkInformation * Update #5679: extract some methods in VpcManagerImpl.java * Update #5679: Update smoke tests as per Daan's comments * Update #5769: fix vpc with private gateways cannot be removed when remove an acount * Update #5769: fix unit test failures after merging latest main * Update #5769: fix schema-41610to41700.sql * Update #5769: fix Request failed due to empty network offering list on UI * Update #5769: Throw exception when account is not found by name * Update #5769: display a warning message if network offering list is empty * Update #5769: fix an UI bug caused by previous commit b286cb7 * Update #5769: fix UI bugs due to vue3 merge * Update #5769: fix issue due to account type refactoring * Update #5769: fix ui bugs due to vue3 * Update #5769: fix issue due to vue3 upgrade * Update #5769: fix issue due to vue3 upgrade part 2 * Update #5769: fix issue due to vue3 upgrade part 3 * Update #5769: highlight default scope when create shared network on UI * Update #5769: fix domain list is not loaded on UI * Update #5769: fix restart/delete shared network by normal users * Update #5769: fix restart domain-scope shared network by domain admin * Update #5769: fix 3 UI bugs (1) double networks in list; (2) icon of first items in list; (3) account/project autoselect * Update #5769: fix 2 ui bugs; (1) selected project is not changed when change domain; (2) no network should be selected by default * Update #5769: fix update shared networks by domain admin/regular user * Update #5769: fix Flicking warning message about the empty network offerings * Update #5769: display associated network name in shared network info card * Update #5769: fix create private gateway form * Update #5769: fix network lists in project view * Update #5769: fix duplicated networks in network dropdown * Update #5769: fix failed to create shared network if associated L2 network is Setup * Update #5769: check AccessType.OperateEntry on network in its implementation * Revert "Update #5769: check AccessType.OperateEntry on network in its implementation" This reverts commit c42c489. * Update #5769: fix keyword search in list guest vlans

weizhouapache added 2 commits November 29, 2021 14:20

Enhancement: create Shared networks and VPC private gateways by users

77dc299

UI bug fix: pass correct domainid in CreateSharedNetworkForm

5f58791

weizhouapache added 2 commits November 30, 2021 11:02

Update apache#5730: fix test failure with test_guest_vlan_range.py

d571e1c

Update apache#5730: fix test failure with test_persistent_network.py

3951120

sureshanaparti reviewed Nov 30, 2021

View reviewed changes

Comment thread ...rc/main/java/org/apache/cloudstack/api/command/admin/vpc/CreatePrivateGatewayByAdminCmd.java

sureshanaparti reviewed Nov 30, 2021

View reviewed changes

Comment thread api/src/main/java/org/apache/cloudstack/api/command/user/network/CreateNetworkCmd.java

Update apache#5730: Add since to new API commands and API parameters

8e14d03

weizhouapache requested review from DaanHoogland and yadvr November 30, 2021 11:03

ravening reviewed Nov 30, 2021

View reviewed changes

Comment thread ...hestration/src/main/java/org/apache/cloudstack/engine/orchestration/NetworkOrchestrator.java Outdated

weizhouapache added 2 commits November 30, 2021 12:22

Update apache#5730: Get first physical network for VPC private gatewa…

65a82df

…y if other ways do not work

Update apache#5730: code optimization (return !offering.isSpecifyVlan())

48b4908

Update apache#5730: fix hard-coded network offering id in test_pvlan.py

c47b691

weizhouapache marked this pull request as ready for review December 1, 2021 08:41

GutoVeronezi reviewed Dec 6, 2021

View reviewed changes

Update apache#5730: update as per GutoVeronezi's comments

0a12ed7

Merge remote-tracking branch 'apache/main' into 4.17-user-shared-netw…

1dbc3ca

…orks

weizhouapache mentioned this pull request Dec 13, 2021

New feature: give access permission of networks to other accounts in same domain #5769

Merged

2 of 12 tasks

weizhouapache closed this Dec 14, 2021

weizhouapache reopened this Dec 14, 2021

Merge remote-tracking branch 'apache/main' into 4.17-user-shared-netw…

9dc349c

…orks

weizhouapache marked this pull request as draft December 15, 2021 07:40

weizhouapache added 4 commits December 15, 2021 13:57

UI: fix create Isolated/L2 network form

f12b3f1

Update apache#5730: fix create Shared network form

cd5d6f4

Update apache#5769: fix domain admin can deploy vm on L2 network of o…

76be448

…ther users

test: fix test_storage_policy.py

2e149c2

weizhouapache closed this Dec 23, 2021

yadvr removed this from the 4.17.0.0 milestone Mar 24, 2022

weizhouapache deleted the 4.17-user-shared-networks branch December 9, 2022 10:37

Conversation

weizhouapache commented Nov 29, 2021

Description

1. Create shared network offering with specifyvlan=false

2. Create shared network by non-ROOT users

If specifyvlan=true in shared network offering

if specifyvlan=false in shared network offering

3. Create VPC private gateway by non-ROOT users

4. List guest vlans in zone

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

weizhouapache commented Nov 29, 2021

Uh oh!

blueorangutan commented Nov 29, 2021

Uh oh!

blueorangutan commented Nov 29, 2021

Uh oh!

weizhouapache commented Nov 29, 2021

Uh oh!

blueorangutan commented Nov 29, 2021

Uh oh!

blueorangutan commented Nov 30, 2021

Uh oh!

weizhouapache commented Nov 30, 2021

Uh oh!

blueorangutan commented Nov 30, 2021

Uh oh!

Uh oh!

Uh oh!

blueorangutan commented Nov 30, 2021

Uh oh!

weizhouapache commented Nov 30, 2021

Uh oh!

Uh oh!

weizhouapache commented Nov 30, 2021

Uh oh!

blueorangutan commented Nov 30, 2021

Uh oh!

blueorangutan commented Nov 30, 2021

Uh oh!

weizhouapache commented Nov 30, 2021

Uh oh!

blueorangutan commented Nov 30, 2021

Uh oh!

blueorangutan commented Nov 30, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GutoVeronezi Dec 6, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

blueorangutan commented Dec 6, 2021

Uh oh!

weizhouapache commented Dec 7, 2021

Uh oh!

blueorangutan commented Dec 7, 2021

Uh oh!

blueorangutan commented Dec 7, 2021

Uh oh!

weizhouapache commented Dec 7, 2021

Uh oh!

blueorangutan commented Dec 7, 2021

Uh oh!

blueorangutan commented Dec 7, 2021

Uh oh!

weizhouapache commented Dec 10, 2021

Uh oh!

blueorangutan commented Dec 10, 2021

Uh oh!

blueorangutan commented Dec 10, 2021

Uh oh!