Skip to content

Updates to HPE-Primera and Pure FlashArray Drivers to use Host-based VLUN Assignments - #8889

Merged
sureshanaparti merged 51 commits into
apache:4.19from
rg9975:4.19
Jun 25, 2024
Merged

Updates to HPE-Primera and Pure FlashArray Drivers to use Host-based VLUN Assignments#8889
sureshanaparti merged 51 commits into
apache:4.19from
rg9975:4.19

Conversation

@rg9975

@rg9975 rg9975 commented Apr 6, 2024

Copy link
Copy Markdown

Description

This PR makes a number of bug fixes to the Primera and Pure storage array discovered in working environments after the release of 4.19.0.0.

The following items are addressed:

  1. Switched to a host-centric rather than hostgroup-centric vlun assignment model. hostgroup-centric works better for shared filesystems but the direct volume attach method for these drivers is better suited to per-host vlun assignments. This required changes to how vlun id's are stored as well as a number of changes in the storage framework to add missing grantAccess/revokeAccess calls or reorder those calls so data is properly available in the Disk objects sent to agents.
  2. Improved error handling and logging in the storage portion of the agent when copying volumes.
  3. Fix to properly handle storage tag to storage provider mapping when migrating disks to a different storage provider and selecting a new disk offering.
  4. Fix an issue related to template removal on primary storage for Pure when all volumes using the template are deleted. Pure storage 'saves' the volume in a Deleted area for some period of time after deletion so any recreation of the volume on subsequent volume creations will trigger an error. The fix renames the volume prior to deletion by appending a timestamp.
  5. Small issues with oAuth login discovered during testing when incorrect user password is entered at the login prompt.
  6. Several small logging cleanups in a few modules that were producing unnecessary noise at INFO level.
  7. Various exception/error handling improvements in storage classes to improve triage/debugging of storage/volume issues.
  8. Fix for issue FibreChannelAdaptor disconnectPhysicalDiskByPath should not always return true #8789 .

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

The standard test suite at plugins/storage/volume/adaptive/README.md was run against a local installation of Pure Flasharray and HPE Primera.

How did you try to break this feature and the system with this change?

Test cases include using the built-in NFS driver in combination with both Flasharray and Primera, moving volumes back and forth between all 3 file systems, and testing standard functions of NFS with these drivers in place.

Glover, Rene (rg9975) and others added 24 commits March 31, 2024 01:33
Glover, Rene (rg9975)
Updates to change PUre and Primera to host-centric vlun assignments; …
1524363 
…various small bug fixes
Glover, Rene (rg9975)
Updates to change PUre and Primera to host-centric vlun assignments; …
41ef2ee 
…various small bug fixes
Glover, Rene (rg9975)
Updates to change PUre and Primera to host-centric vlun assignments; …
cb1f0fa 
…various small bug fixes
Glover, Rene (rg9975)
Updates to change PUre and Primera to host-centric vlun assignments; …
e5c61bc 
…various small bug fixes
Glover, Rene (rg9975)
update to add timestamp when deleting pure volumes to avoid future co…
7c12b06 
…nflicts
Glover, Rene (rg9975)
update to migrate to properly check disk offering is valid for the ta…
5bd6909 
…rget storage pool
Glover, Rene (rg9975)
update to migrate to properly check disk offering is valid for the ta…
7d041c4 
…rget storage pool
Glover, Rene (rg9975)
update to migrate to properly check disk offering is valid for the ta…
c761869 
…rget storage pool
Glover, Rene (rg9975)
update to migrate to properly check disk offering is valid for the ta…
3e8cb96 
…rget storage pool
Glover, Rene (rg9975)
Updates to change PUre and Primera to host-centric vlun assignments; …
73a6fd9 
…various small bug fixes
Glover, Rene (rg9975)
Updates to change PUre and Primera to host-centric vlun assignments; …
c622afe 
…various small bug fixes
Glover, Rene (rg9975)
Updates to change PUre and Primera to host-centric vlun assignments; …
d9f1016 
…various small bug fixes
Glover, Rene (rg9975)
Updates to change PUre and Primera to host-centric vlun assignments; …
94cf706 
…various small bug fixes
Glover, Rene (rg9975)
update to add timestamp when deleting pure volumes to avoid future co…
2dcfdb9 
…nflicts
Glover, Rene (rg9975)
update to migrate to properly check disk offering is valid for the ta…
35b76c9 
…rget storage pool
Glover, Rene (rg9975)
update to migrate to properly check disk offering is valid for the ta…
638a30f 
…rget storage pool
Glover, Rene (rg9975)
update to migrate to properly check disk offering is valid for the ta…
71cb21d 
…rget storage pool
Glover, Rene (rg9975)
update to migrate to properly check disk offering is valid for the ta…
ad331ff 
…rget storage pool
GLOVER RENE
Merge branch '4.19' of https://github.com/rg9975/cloudstack into 4.19
2916954
Glover, Rene (rg9975)
improve error handling when copying volumes to add precision to which…
1151ddf 
… step failed
Glover, Rene (rg9975)
rename pure volume before delete to avoid conflicts if the same name …
eb72474 
…is used before its expunged on the array
Glover, Rene (rg9975)
rename pure volume before delete to avoid conflicts if the same name …
98225b7 
…is used before its expunged on the array
Glover, Rene (rg9975)
remove dead code in AdaptiveDataStoreLifeCycleImpl.java
da6eb37
Glover, Rene (rg9975)
remove dead code in AdaptiveDataStoreLifeCycleImpl.java
bb0c3ee
@DaanHoogland

DaanHoogland commented Apr 8, 2024

Copy link
Copy Markdown
Contributor

@rg9975 the github actions show 1 succesfull check out of 22 👀 probably not a structural issue with your code, but please have a look.

@DaanHoogland

DaanHoogland commented Jun 20, 2024

Copy link
Copy Markdown
Contributor

@rg9975 can you look at @sureshanaparti 's comments, please?

@yadvr

yadvr commented Jun 21, 2024

Copy link
Copy Markdown
Member

@blueorangutan package

@blueorangutan

blueorangutan commented Jun 21, 2024

Copy link
Copy Markdown

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

blueorangutan commented Jun 21, 2024

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10051

Rene Glover and others added 4 commits June 21, 2024 09:55
Rene Glover @sureshanaparti
Update engine/storage/datamotion/src/main/java/org/apache/cloudstack/…
89d7a2c 
…storage/motion/StorageSystemDataMotionStrategy.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
Rene Glover @sureshanaparti
Update framework/spring/module/src/main/java/org/apache/cloudstack/sp…
c55dfc9 
…ring/module/model/impl/DefaultModuleDefinitionSet.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
Glover, Rene (rg9975)
address PR comments from @sureshanaparti
42780e7
Glover, Rene (rg9975)
address PR comments from @sureshanaparti
018e556
@sureshanaparti

sureshanaparti commented Jun 21, 2024

Copy link
Copy Markdown
Contributor

@blueorangutan package

@blueorangutan

blueorangutan commented Jun 21, 2024

Copy link
Copy Markdown

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

blueorangutan commented Jun 21, 2024

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10062

@sureshanaparti

sureshanaparti commented Jun 21, 2024

Copy link
Copy Markdown
Contributor

@blueorangutan test

@blueorangutan

blueorangutan commented Jun 21, 2024

Copy link
Copy Markdown

@sureshanaparti a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan

blueorangutan commented Jun 22, 2024

Copy link
Copy Markdown

[SF] Trillian test result (tid-10565)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 52661 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8889-t10565-kvm-centos7.zip
Smoke tests completed. 130 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_02_redundant_VPC_default_routes Failure 1347.23 test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers Failure 431.99 test_vpc_redundant.py

@yadvr yadvr closed this Jun 24, 2024
@yadvr yadvr reopened this Jun 24, 2024
@yadvr

yadvr commented Jun 24, 2024

Copy link
Copy Markdown
Member

@blueorangutan package

@blueorangutan

blueorangutan commented Jun 24, 2024

Copy link
Copy Markdown

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

blueorangutan commented Jun 24, 2024

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10087

@sureshanaparti

sureshanaparti commented Jun 24, 2024

Copy link
Copy Markdown
Contributor

@blueorangutan test

@blueorangutan

blueorangutan commented Jun 24, 2024

Copy link
Copy Markdown

@sureshanaparti a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@yadvr

yadvr commented Jun 24, 2024

Copy link
Copy Markdown
Member

@blueorangutan test matrix

@blueorangutan

blueorangutan commented Jun 24, 2024

Copy link
Copy Markdown

@rohityadavcloud a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

@blueorangutan

blueorangutan commented Jun 24, 2024

Copy link
Copy Markdown

[SF] Trillian test result (tid-10582)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 45585 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8889-t10582-kvm-centos7.zip
Smoke tests completed. 130 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestClusterDRS>:setup Error 0.00 test_cluster_drs.py

@blueorangutan

blueorangutan commented Jun 25, 2024

Copy link
Copy Markdown

[SF] Trillian test result (tid-10595)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43302 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8889-t10595-xenserver-71.zip
Smoke tests completed. 131 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File

@blueorangutan

blueorangutan commented Jun 25, 2024

Copy link
Copy Markdown

[SF] Trillian test result (tid-10597)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 46257 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8889-t10597-kvm-centos7.zip
Smoke tests completed. 131 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File

@sureshanaparti
sureshanaparti merged commit 6ee6603 into apache:4.19 Jun 25, 2024
@blueorangutan

blueorangutan commented Jun 25, 2024

Copy link
Copy Markdown

[SF] Trillian test result (tid-10596)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 47496 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8889-t10596-vmware-67u3.zip
Smoke tests completed. 127 look OK, 4 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_DeployVmAntiAffinityGroup_in_project Error 110.38 test_affinity_groups_projects.py
test_DeployVmAntiAffinityGroup Error 47.13 test_affinity_groups.py
test_03_deploy_and_scale_kubernetes_cluster Failure 49.46 test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster Failure 0.09 test_kubernetes_clusters.py
test_01_non_strict_host_anti_affinity Failure 147.37 test_nonstrict_affinity_group.py
test_02_non_strict_host_affinity Error 108.45 test_nonstrict_affinity_group.py

dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Jul 2, 2024
Rene Glover GLOVER RENE
@sureshanaparti @dhslove
Updates to HPE-Primera and Pure FlashArray Drivers to use Host-based …
0e09335 
…VLUN Assignments (apache#8889)

* Updates to change PUre and Primera to host-centric vlun assignments; various small bug fixes

* update to add timestamp when deleting pure volumes to avoid future conflicts

* update to migrate to properly check disk offering is valid for the target storage pool

* Updates to change PUre and Primera to host-centric vlun assignments; various small bug fixes

* update to add timestamp when deleting pure volumes to avoid future conflicts

* update to migrate to properly check disk offering is valid for the target storage pool

* improve error handling when copying volumes to add precision to which step failed

* rename pure volume before delete to avoid conflicts if the same name is used before its expunged on the array

* remove dead code in AdaptiveDataStoreLifeCycleImpl.java

* Fix issues found in PR checks

* fix session refresh TTL logic

* updates from PR comments

* logic to delete by path ONLY on supported OUI

* fix to StorageSystemDataMotionStrategy compile error

* change noisy debug message to trace message

* fix double callback call in handleVolumeMigrationFromNonManagedStorageToManagedStorage

* fix for flash array delete error

* fix typo in StorageSystemDataMotionStrategy

* change copyVolume to use writeback to speed up copy ops

* remove returning PrimaryStorageDownloadAnswer when connectPhysicalDisk returns false during KVMStorageProcessor template copy

* remove change to only set UUID on snapshot if it is a vmSnapshot

* reverting change to UserVmManagerImpl.configureCustomRootDiskSize

* add error checking/simplification per comments from @slavkap

* Update engine/storage/datamotion/src/main/java/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* address PR comments from @sureshanaparti

---------

Co-authored-by: GLOVER RENE <rg9975@cs419-mgmtserver.rg9975nprd.app.ecp.att.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rp- rp- rp- left review comments
@DaanHoogland DaanHoogland DaanHoogland left review comments
@sureshanaparti sureshanaparti sureshanaparti left review comments
@harikrishna-patnala harikrishna-patnala harikrishna-patnala approved these changes
@slavkap slavkap slavkap approved these changes

Assignees

No one assigned

Labels

component:kvm component:orchestration component:primary-storage component:projects component:storage

Projects

No open projects
Apache CloudStack 4.19.1
Status: Done

Milestone

4.19.1.0

Development

Successfully merging this pull request may close these issues.

10 participants

@rg9975 @DaanHoogland @weizhouapache @codecov-commenter @blueorangutan @slavkap @rp- @yadvr @sureshanaparti @harikrishna-patnala