Skip to content

Cloudstack 4.18 HTTP 503 Error (MySQL 5.6/5.7) #7358

Closed
Closed
Cloudstack 4.18 HTTP 503 Error (MySQL 5.6/5.7)#7358
Milestone
unplanned

Description

@kenneth0595
kenneth0595
opened on Mar 22, 2023

I installed Cloudstack 4.18 new version and access to ui with IP:8080 but I couldn’t see cloudstack ui because of HTTP 503 Error.
Please, how to resolve this issue.

Activity

  1. kiwiflyer commented on Mar 22, 2023

    @kiwiflyer
    kiwiflyer
    on Mar 22, 2023
    Contributor

    @kenneth0595 Can you provide us more info, such as the operating system you installed it on, how you installed it (packages or from source), and the steps you have taken to set up the system?

    Have you followed the installation guide? It's available here for reference -
    https://docs.cloudstack.apache.org/en/latest/installguide/index.html

  2. kenneth0595 commented on Mar 22, 2023

    @kenneth0595
    kenneth0595
    on Mar 22, 2023
    Author

    @kiwiflyer

    OS: CentOS 7.9
    I just followed the quick installation quide on Cloudstack document.
    After installing the os, I just update package with yum update.

  3. kiwiflyer commented on Mar 22, 2023

    @kiwiflyer
    kiwiflyer
    on Mar 22, 2023
    Contributor

    Can you take a look in /var/log/cloudstack/management/management-server.log and /var/log/cloudstack/management/access.log?

    Look for errors and exceptions being thrown.

  4. weizhouapache commented on Mar 22, 2023

    @weizhouapache
    weizhouapache
    on Mar 22, 2023
    Member

    @kenneth0595
    Try http://ip:8080/client

  5. kenneth0595 commented on Mar 22, 2023

    @kenneth0595
    kenneth0595
    on Mar 22, 2023
    Author

    @kiwiflyer
    [root@Cloud-Host01 ~]# grep -i -E 'error|exception' /var/log/cloudstack/management/management-server.log
    2023-03-22 12:32:20,130 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CREATE TABLE cloud.s3 ( id bigint unsigned NOT NULL AUTO_INCREMENT COMMENT 'id', uuid varchar(40), access_key varchar(20) NOT NULL COMMENT ' The S3 access key', secret_key varchar(40) NOT NULL COMMENT ' The S3 secret key', end_point varchar(1024) COMMENT ' The S3 host', bucket varchar(63) NOT NULL COMMENT ' The S3 host', https tinyint unsigned DEFAULT NULL COMMENT ' Flag indicating whether or not to connect over HTTPS', connection_timeout integer COMMENT ' The amount of time to wait (in milliseconds) when initially establishing a connection before giving up and timing out.', max_error_retry integer COMMENT ' The maximum number of retry attempts for failed retryable requests (ex: 5xx error responses from services).', socket_timeout integer COMMENT ' The amount of time to wait (in milliseconds) for data to be transferred over an established, open connection before the connection times out and is closed.', created datetime COMMENT 'date the s3 first signed on', PRIMARY KEY (id), CONSTRAINT uc_s3__uuid UNIQUE (uuid) ) ENGINE=InnoDB DEFAULT CHARSET=utf8
    2023-03-22 12:32:21,031 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CREATE VIEW cloud.volume_view AS select volumes.id, volumes.uuid, volumes.name, volumes.device_id, volumes.volume_type, volumes.size, volumes.created, volumes.state, volumes.attached, volumes.removed, volumes.pod_id, account.id account_id, account.uuid account_uuid, account.account_name account_name, account.type account_type, domain.id domain_id, domain.uuid domain_uuid, domain.name domain_name, domain.path domain_path, projects.id project_id, projects.uuid project_uuid, projects.name project_name, data_center.id data_center_id, data_center.uuid data_center_uuid, data_center.name data_center_name, vm_instance.id vm_id, vm_instance.uuid vm_uuid, vm_instance.name vm_name, vm_instance.state vm_state, vm_instance.vm_type, user_vm.display_name vm_display_name, volume_host_ref.size volume_host_size, volume_host_ref.created volume_host_created, volume_host_ref.format, volume_host_ref.download_pct, volume_host_ref.download_state, volume_host_ref.error_str, disk_offering.id disk_offering_id, disk_offering.uuid disk_offering_uuid, disk_offering.name disk_offering_name, disk_offering.display_text disk_offering_display_text, disk_offering.use_local_storage, disk_offering.system_use, storage_pool.id pool_id, storage_pool.uuid pool_uuid, storage_pool.name pool_name, cluster.hypervisor_type, vm_template.id template_id, vm_template.uuid template_uuid, vm_template.extractable, vm_template.type template_type, resource_tags.id tag_id, resource_tags.uuid tag_uuid, resource_tags.key tag_key, resource_tags.value tag_value, resource_tags.domain_id tag_domain_id, resource_tags.account_id tag_account_id, resource_tags.resource_id tag_resource_id, resource_tags.resource_uuid tag_resource_uuid, resource_tags.resource_type tag_resource_type, resource_tags.customer tag_customer, async_job.id job_id, async_job.uuid job_uuid, async_job.job_status job_status, async_job.account_id job_account_id from cloud.volumes inner join cloud.account ON volumes.account_id = account.id inner join cloud.domain ON volumes.domain_id = domain.id left join cloud.projects ON projects.project_account_id = account.id left join cloud.data_center ON volumes.data_center_id = data_center.id left join cloud.vm_instance ON volumes.instance_id = vm_instance.id left join cloud.user_vm ON user_vm.id = vm_instance.id left join cloud.volume_host_ref ON volumes.id = volume_host_ref.volume_id and volumes.data_center_id = volume_host_ref.zone_id left join cloud.disk_offering ON volumes.disk_offering_id = disk_offering.id left join cloud.storage_pool ON volumes.pool_id = storage_pool.id left join cloud.cluster ON storage_pool.cluster_id = cluster.id left join cloud.vm_template ON volumes.template_id = vm_template.id left join cloud.resource_tags ON resource_tags.resource_id = volumes.id and resource_tags.resource_type = 'Volume' left join cloud.async_job ON async_job.instance_id = volumes.id and async_job.instance_type = 'Volume' and async_job.job_status = 0
    2023-03-22 12:32:21,502 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CREATE TABLE cloud.object_datastore_ref ( id bigint unsigned NOT NULL auto_increment, datastore_uuid varchar(255) NOT NULL, datastore_role varchar(255) NOT NULL, object_uuid varchar(255) NOT NULL, object_type varchar(255) NOT NULL, created DATETIME NOT NULL, last_updated DATETIME, job_id varchar(255), download_pct int(10) unsigned, download_state varchar(255), url varchar(255), format varchar(255), checksum varchar(255), error_str varchar(255), local_path varchar(255), install_path varchar(255), size bigint unsigned COMMENT 'the size of the template on the pool', state varchar(255) NOT NULL, update_count bigint unsigned NOT NULL, updated DATETIME, PRIMARY KEY (id) ) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8
    2023-03-22 12:32:21,553 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CREATE TABLE cloud.template_store_ref ( id bigint unsigned NOT NULL auto_increment, store_id bigint unsigned, template_id bigint unsigned NOT NULL, created DATETIME NOT NULL, last_updated DATETIME, job_id varchar(255), download_pct int(10) unsigned, size bigint unsigned, store_role varchar(255), physical_size bigint unsigned DEFAULT 0, download_state varchar(255), error_str varchar(255), local_path varchar(255), install_path varchar(255), url varchar(255), state varchar(255) NOT NULL, destroyed tinyint(1) COMMENT 'indicates whether the template_store entry was destroyed by the user or not', is_copy tinyint(1) NOT NULL DEFAULT 0 COMMENT 'indicates whether this was copied ', update_count bigint unsigned, ref_cnt bigint unsigned DEFAULT 0, updated datetime, PRIMARY KEY (id), INDEX i_template_store_ref__store_id(store_id), CONSTRAINT fk_template_store_ref__template_id FOREIGN KEY fk_template_store_ref__template_id (template_id) REFERENCES vm_template (id), INDEX i_template_store_ref__template_id(template_id) ) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8
    2023-03-22 12:32:21,583 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CREATE TABLE cloud.volume_store_ref ( id bigint unsigned NOT NULL auto_increment, store_id bigint unsigned NOT NULL, volume_id bigint unsigned NOT NULL, zone_id bigint unsigned NOT NULL, created DATETIME NOT NULL, last_updated DATETIME, job_id varchar(255), download_pct int(10) unsigned, size bigint unsigned, physical_size bigint unsigned DEFAULT 0, download_state varchar(255), checksum varchar(255) COMMENT 'checksum for the data disk', error_str varchar(255), local_path varchar(255), install_path varchar(255), url varchar(255), download_url varchar(255), state varchar(255) NOT NULL, destroyed tinyint(1) COMMENT 'indicates whether the volume_host entry was destroyed by the user or not', update_count bigint unsigned, ref_cnt bigint unsigned, updated datetime, PRIMARY KEY (id), CONSTRAINT fk_volume_store_ref__store_id FOREIGN KEY fk_volume_store_ref__store_id (store_id) REFERENCES image_store (id) ON DELETE CASCADE, INDEX i_volume_store_ref__store_id(store_id), CONSTRAINT fk_volume_store_ref__volume_id FOREIGN KEY fk_volume_store_ref__volume_id (volume_id) REFERENCES volumes (id), INDEX i_volume_store_ref__volume_id(volume_id) ) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8
    2023-03-22 12:32:23,049 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CREATE VIEW cloud.volume_view AS select volumes.id, volumes.uuid, volumes.name, volumes.device_id, volumes.volume_type, volumes.size, volumes.min_iops, volumes.max_iops, volumes.created, volumes.state, volumes.attached, volumes.removed, volumes.pod_id, volumes.display_volume, volumes.format, account.id account_id, account.uuid account_uuid, account.account_name account_name, account.type account_type, domain.id domain_id, domain.uuid domain_uuid, domain.name domain_name, domain.path domain_path, projects.id project_id, projects.uuid project_uuid, projects.name project_name, data_center.id data_center_id, data_center.uuid data_center_uuid, data_center.name data_center_name, data_center.networktype data_center_type, vm_instance.id vm_id, vm_instance.uuid vm_uuid, vm_instance.name vm_name, vm_instance.state vm_state, vm_instance.vm_type, user_vm.display_name vm_display_name, volume_store_ref.size volume_store_size, volume_store_ref.download_pct, volume_store_ref.download_state, volume_store_ref.error_str, volume_store_ref.created created_on_store, disk_offering.id disk_offering_id, disk_offering.uuid disk_offering_uuid, disk_offering.name disk_offering_name, disk_offering.display_text disk_offering_display_text, disk_offering.use_local_storage, disk_offering.system_use, disk_offering.bytes_read_rate, disk_offering.bytes_write_rate, disk_offering.iops_read_rate, disk_offering.iops_write_rate, storage_pool.id pool_id, storage_pool.uuid pool_uuid, storage_pool.name pool_name, cluster.hypervisor_type, vm_template.id template_id, vm_template.uuid template_uuid, vm_template.extractable, vm_template.type template_type, resource_tags.id tag_id, resource_tags.uuid tag_uuid, resource_tags.key tag_key, resource_tags.value tag_value, resource_tags.domain_id tag_domain_id, resource_tags.account_id tag_account_id, resource_tags.resource_id tag_resource_id, resource_tags.resource_uuid tag_resource_uuid, resource_tags.resource_type tag_resource_type, resource_tags.customer tag_customer, async_job.id job_id, async_job.uuid job_uuid, async_job.job_status job_status, async_job.account_id job_account_id from cloud.volumes inner join cloud.account ON volumes.account_id = account.id inner join cloud.domain ON volumes.domain_id = domain.id left join cloud.projects ON projects.project_account_id = account.id left join cloud.data_center ON volumes.data_center_id = data_center.id left join cloud.vm_instance ON volumes.instance_id = vm_instance.id left join cloud.user_vm ON user_vm.id = vm_instance.id left join cloud.volume_store_ref ON volumes.id = volume_store_ref.volume_id left join cloud.disk_offering ON volumes.disk_offering_id = disk_offering.id left join cloud.storage_pool ON volumes.pool_id = storage_pool.id left join cloud.cluster ON storage_pool.cluster_id = cluster.id left join cloud.vm_template ON volumes.template_id = vm_template.id OR volumes.iso_id = vm_template.id left join cloud.resource_tags ON resource_tags.resource_id = volumes.id and resource_tags.resource_type = 'Volume' left join cloud.async_job ON async_job.instance_id = volumes.id and async_job.instance_type = 'Volume' and async_job.job_status = 0
    2023-03-22 12:32:23,109 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CREATE VIEW cloud.template_view AS select vm_template.id, vm_template.uuid, vm_template.unique_name, vm_template.name, vm_template.public, vm_template.featured, vm_template.type, vm_template.hvm, vm_template.bits, vm_template.url, vm_template.format, vm_template.created, vm_template.checksum, vm_template.display_text, vm_template.enable_password, vm_template.dynamically_scalable, vm_template.guest_os_id, guest_os.uuid guest_os_uuid, guest_os.display_name guest_os_name, vm_template.bootable, vm_template.prepopulate, vm_template.cross_zones, vm_template.hypervisor_type, vm_template.extractable, vm_template.template_tag, vm_template.sort_key, vm_template.removed, vm_template.enable_sshkey, source_template.id source_template_id, source_template.uuid source_template_uuid, account.id account_id, account.uuid account_uuid, account.account_name account_name, account.type account_type, domain.id domain_id, domain.uuid domain_uuid, domain.name domain_name, domain.path domain_path, projects.id project_id, projects.uuid project_uuid, projects.name project_name, data_center.id data_center_id, data_center.uuid data_center_uuid, data_center.name data_center_name, launch_permission.account_id lp_account_id, template_store_ref.store_id, image_store.scope as store_scope, template_store_ref.state, template_store_ref.download_state, template_store_ref.download_pct, template_store_ref.error_str, template_store_ref.size, template_store_ref.destroyed, template_store_ref.created created_on_store, vm_template_details.name detail_name, vm_template_details.value detail_value, resource_tags.id tag_id, resource_tags.uuid tag_uuid, resource_tags.key tag_key, resource_tags.value tag_value, resource_tags.domain_id tag_domain_id, resource_tags.account_id tag_account_id, resource_tags.resource_id tag_resource_id, resource_tags.resource_uuid tag_resource_uuid, resource_tags.resource_type tag_resource_type, resource_tags.customer tag_customer, CONCAT(vm_template.id, '', IFNULL(data_center.id, 0)) as temp_zone_pair from cloud.vm_template inner join cloud.guest_os ON guest_os.id = vm_template.guest_os_id inner join cloud.account ON account.id = vm_template.account_id inner join cloud.domain ON domain.id = account.domain_id left join cloud.projects ON projects.project_account_id = account.id left join cloud.vm_template_details ON vm_template_details.template_id = vm_template.id left join cloud.vm_template source_template ON source_template.id = vm_template.source_template_id left join cloud.template_store_ref ON template_store_ref.template_id = vm_template.id and template_store_ref.store_role = 'Image' left join cloud.image_store ON image_store.removed is NULL AND template_store_ref.store_id is not NULL AND image_store.id = template_store_ref.store_id left join cloud.template_zone_ref ON template_zone_ref.template_id = vm_template.id AND template_store_ref.store_id is NULL AND template_zone_ref.removed is null left join cloud.data_center ON (image_store.data_center_id = data_center.id OR template_zone_ref.zone_id = data_center.id) left join cloud.launch_permission ON launch_permission.template_id = vm_template.id left join cloud.resource_tags ON resource_tags.resource_id = vm_template.id and (resource_tags.resource_type = 'Template' or resource_tags.resource_type='ISO')
    2023-03-22 12:32:23,627 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Ignored SQL Exception when trying to drop key last_sent on table alert exception: Can't DROP 'last_sent'; check that column/key exists
    2023-03-22 12:32:23,627 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Ignored SQL Exception when trying to drop key i_alert__last_sent on table alert exception: Can't DROP 'i_alert__last_sent'; check that column/key exists
    2023-03-22 12:32:23,639 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Ignored SQL Exception when trying to drop foreign key fk_external_dhcp_devices_nsp_id on table baremetal_dhcp_devices exception: Can't DROP 'fk_external_dhcp_devices_nsp_id'; check that column/key exists
    2023-03-22 12:32:23,640 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Ignored SQL Exception when trying to drop foreign key fk_external_dhcp_devices_host_id on table baremetal_dhcp_devices exception: Can't DROP 'fk_external_dhcp_devices_host_id'; check that column/key exists
    2023-03-22 12:32:23,641 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Ignored SQL Exception when trying to drop foreign key fk_external_dhcp_devices_pod_id on table baremetal_dhcp_devices exception: Can't DROP 'fk_external_dhcp_devices_pod_id'; check that column/key exists
    2023-03-22 12:32:23,643 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Ignored SQL Exception when trying to drop foreign key fk_external_dhcp_devices_physical_network_id on table baremetal_dhcp_devices exception: Can't DROP 'fk_external_dhcp_devices_physical_network_id'; check that column/key exists
    2023-03-22 12:32:23,644 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Ignored SQL Exception when trying to drop foreign key fk_external_dhcp_devices_nsp_id on table baremetal_pxe_devices exception: Can't DROP 'fk_external_dhcp_devices_nsp_id'; check that column/key exists
    2023-03-22 12:32:23,645 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Ignored SQL Exception when trying to drop foreign key fk_external_dhcp_devices_host_id on table baremetal_pxe_devices exception: Can't DROP 'fk_external_dhcp_devices_host_id'; check that column/key exists
    2023-03-22 12:32:23,648 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Ignored SQL Exception when trying to drop foreign key fk_external_dhcp_devices_pod_id on table baremetal_pxe_devices exception: Can't DROP 'fk_external_dhcp_devices_pod_id'; check that column/key exists
    2023-03-22 12:32:23,651 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Ignored SQL Exception when trying to drop foreign key fk_external_dhcp_devices_physical_network_id on table baremetal_pxe_devices exception: Can't DROP 'fk_external_dhcp_devices_physical_network_id'; check that column/key exists
    2023-03-22 12:32:23,652 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Ignored SQL Exception when trying to drop foreign key fk_external_pxe_devices_nsp_id on table baremetal_pxe_devices exception: Can't DROP 'fk_external_pxe_devices_nsp_id'; check that column/key exists
    2023-03-22 12:32:23,654 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Ignored SQL Exception when trying to drop foreign key fk_external_pxe_devices_host_id on table baremetal_pxe_devices exception: Can't DROP 'fk_external_pxe_devices_host_id'; check that column/key exists
    2023-03-22 12:32:23,655 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Ignored SQL Exception when trying to drop foreign key fk_external_pxe_devices_physical_network_id on table baremetal_pxe_devices exception: Can't DROP 'fk_external_pxe_devices_physical_network_id'; check that column/key exists
    2023-03-22 12:32:23,876 INFO [c.c.u.d.Upgrade410to420] (main:null) (logid:) migrateDatafromIsoIdInVolumesTable: ignoring Exception: Unknown column 'iso_id1' in 'field list'
    2023-03-22 12:32:24,903 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CREATE VIEW cloud.template_view AS select vm_template.id, vm_template.uuid, vm_template.unique_name, vm_template.name, vm_template.public, vm_template.featured, vm_template.type, vm_template.hvm, vm_template.bits, vm_template.url, vm_template.format, vm_template.created, vm_template.checksum, vm_template.display_text, vm_template.enable_password, vm_template.dynamically_scalable, vm_template.state template_state, vm_template.guest_os_id, guest_os.uuid guest_os_uuid, guest_os.display_name guest_os_name, vm_template.bootable, vm_template.prepopulate, vm_template.cross_zones, vm_template.hypervisor_type, vm_template.extractable, vm_template.template_tag, vm_template.sort_key, vm_template.removed, vm_template.enable_sshkey, source_template.id source_template_id, source_template.uuid source_template_uuid, account.id account_id, account.uuid account_uuid, account.account_name account_name, account.type account_type, domain.id domain_id, domain.uuid domain_uuid, domain.name domain_name, domain.path domain_path, projects.id project_id, projects.uuid project_uuid, projects.name project_name, data_center.id data_center_id, data_center.uuid data_center_uuid, data_center.name data_center_name, launch_permission.account_id lp_account_id, template_store_ref.store_id, image_store.scope as store_scope, template_store_ref.state, template_store_ref.download_state, template_store_ref.download_pct, template_store_ref.error_str, template_store_ref.size, template_store_ref.destroyed, template_store_ref.created created_on_store, vm_template_details.name detail_name, vm_template_details.value detail_value, resource_tags.id tag_id, resource_tags.uuid tag_uuid, resource_tags.key tag_key, resource_tags.value tag_value, resource_tags.domain_id tag_domain_id, resource_tags.account_id tag_account_id, resource_tags.resource_id tag_resource_id, resource_tags.resource_uuid tag_resource_uuid, resource_tags.resource_type tag_resource_type, resource_tags.customer tag_customer, CONCAT(vm_template.id, '    ', IFNULL(data_center.id, 0)) as temp_zone_pair from cloud.vm_template inner join cloud.guest_os ON guest_os.id = vm_template.guest_os_id inner join cloud.account ON account.id = vm_template.account_id inner join cloud.domain ON domain.id = account.domain_id left join cloud.projects ON projects.project_account_id = account.id left join cloud.vm_template_details ON vm_template_details.template_id = vm_template.id left join cloud.vm_template source_template ON source_template.id = vm_template.source_template_id left join cloud.template_store_ref ON template_store_ref.template_id = vm_template.id and template_store_ref.store_role = 'Image' and template_store_ref.destroyed=0 left join cloud.image_store ON image_store.removed is NULL AND template_store_ref.store_id is not NULL AND image_store.id = template_store_ref.store_id left join cloud.template_zone_ref ON template_zone_ref.template_id = vm_template.id AND template_store_ref.store_id is NULL AND template_zone_ref.removed is null left join cloud.data_center ON (image_store.data_center_id = data_center.id OR template_zone_ref.zone_id = data_center.id) left join cloud.launch_permission ON launch_permission.template_id = vm_template.id left join cloud.resource_tags ON resource_tags.resource_id = vm_template.id and (resource_tags.resource_type = 'Template' or resource_tags.resource_type='ISO')
    2023-03-22 12:32:24,906 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CREATE VIEW cloud.volume_view AS select volumes.id, volumes.uuid, volumes.name, volumes.device_id, volumes.volume_type, volumes.size, volumes.min_iops, volumes.max_iops, volumes.created, volumes.state, volumes.attached, volumes.removed, volumes.pod_id, volumes.display_volume, volumes.format, volumes.path, account.id account_id, account.uuid account_uuid, account.account_name account_name, account.type account_type, domain.id domain_id, domain.uuid domain_uuid, domain.name domain_name, domain.path domain_path, projects.id project_id, projects.uuid project_uuid, projects.name project_name, data_center.id data_center_id, data_center.uuid data_center_uuid, data_center.name data_center_name, data_center.networktype data_center_type, vm_instance.id vm_id, vm_instance.uuid vm_uuid, vm_instance.name vm_name, vm_instance.state vm_state, vm_instance.vm_type, user_vm.display_name vm_display_name, volume_store_ref.size volume_store_size, volume_store_ref.download_pct, volume_store_ref.download_state, volume_store_ref.error_str, volume_store_ref.created created_on_store, disk_offering.id disk_offering_id, disk_offering.uuid disk_offering_uuid, disk_offering.name disk_offering_name, disk_offering.display_text disk_offering_display_text, disk_offering.use_local_storage, disk_offering.system_use, disk_offering.bytes_read_rate, disk_offering.bytes_write_rate, disk_offering.iops_read_rate, disk_offering.iops_write_rate, storage_pool.id pool_id, storage_pool.uuid pool_uuid, storage_pool.name pool_name, cluster.hypervisor_type, vm_template.id template_id, vm_template.uuid template_uuid, vm_template.extractable, vm_template.type template_type, resource_tags.id tag_id, resource_tags.uuid tag_uuid, resource_tags.key tag_key, resource_tags.value tag_value, resource_tags.domain_id tag_domain_id, resource_tags.account_id tag_account_id, resource_tags.resource_id tag_resource_id, resource_tags.resource_uuid tag_resource_uuid, resource_tags.resource_type tag_resource_type, resource_tags.customer tag_customer, async_job.id job_id, async_job.uuid job_uuid, async_job.job_status job_status, async_job.account_id job_account_id from cloud.volumes inner join cloud.account ON volumes.account_id = account.id inner join cloud.domain ON volumes.domain_id = domain.id left join cloud.projects ON projects.project_account_id = account.id left join cloud.data_center ON volumes.data_center_id = data_center.id left join cloud.vm_instance ON volumes.instance_id = vm_instance.id left join cloud.user_vm ON user_vm.id = vm_instance.id left join cloud.volume_store_ref ON volumes.id = volume_store_ref.volume_id left join cloud.disk_offering ON volumes.disk_offering_id = disk_offering.id left join cloud.storage_pool ON volumes.pool_id = storage_pool.id left join cloud.cluster ON storage_pool.cluster_id = cluster.id left join cloud.vm_template ON volumes.template_id = vm_template.id OR volumes.iso_id = vm_template.id left join cloud.resource_tags ON resource_tags.resource_id = volumes.id and resource_tags.resource_type = 'Volume' left join cloud.async_job ON async_job.instance_id = volumes.id and async_job.instance_type = 'Volume' and async_job.job_status = 0
    2023-03-22 12:32:25,490 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CREATE VIEW cloud.volume_view AS select volumes.id, volumes.uuid, volumes.name, volumes.device_id, volumes.volume_type, volumes.size, volumes.min_iops, volumes.max_iops, volumes.created, volumes.state, volumes.attached, volumes.removed, volumes.pod_id, volumes.display_volume, volumes.format, volumes.path, volumes.chain_info, account.id account_id, account.uuid account_uuid, account.account_name account_name, account.type account_type, domain.id domain_id, domain.uuid domain_uuid, domain.name domain_name, domain.path domain_path, projects.id project_id, projects.uuid project_uuid, projects.name project_name, data_center.id data_center_id, data_center.uuid data_center_uuid, data_center.name data_center_name, data_center.networktype data_center_type, vm_instance.id vm_id, vm_instance.uuid vm_uuid, vm_instance.name vm_name, vm_instance.state vm_state, vm_instance.vm_type, user_vm.display_name vm_display_name, volume_store_ref.size volume_store_size, volume_store_ref.download_pct, volume_store_ref.download_state, volume_store_ref.error_str, volume_store_ref.created created_on_store, disk_offering.id disk_offering_id, disk_offering.uuid disk_offering_uuid, disk_offering.name disk_offering_name, disk_offering.display_text disk_offering_display_text, disk_offering.use_local_storage, disk_offering.system_use, disk_offering.bytes_read_rate, disk_offering.bytes_write_rate, disk_offering.iops_read_rate, disk_offering.iops_write_rate, disk_offering.cache_mode, storage_pool.id pool_id, storage_pool.uuid pool_uuid, storage_pool.name pool_name, cluster.hypervisor_type, vm_template.id template_id, vm_template.uuid template_uuid, vm_template.extractable, vm_template.type template_type, vm_template.name template_name, vm_template.display_text template_display_text, iso.id iso_id, iso.uuid iso_uuid, iso.name iso_name, iso.display_text iso_display_text, resource_tags.id tag_id, resource_tags.uuid tag_uuid, resource_tags.key tag_key, resource_tags.value tag_value, resource_tags.domain_id tag_domain_id, resource_tags.account_id tag_account_id, resource_tags.resource_id tag_resource_id, resource_tags.resource_uuid tag_resource_uuid, resource_tags.resource_type tag_resource_type, resource_tags.customer tag_customer, async_job.id job_id, async_job.uuid job_uuid, async_job.job_status job_status, async_job.account_id job_account_id from cloud.volumes inner join cloud.account ON volumes.account_id = account.id inner join cloud.domain ON volumes.domain_id = domain.id left join cloud.projects ON projects.project_account_id = account.id left join cloud.data_center ON volumes.data_center_id = data_center.id left join cloud.vm_instance ON volumes.instance_id = vm_instance.id left join cloud.user_vm ON user_vm.id = vm_instance.id left join cloud.volume_store_ref ON volumes.id = volume_store_ref.volume_id left join cloud.disk_offering ON volumes.disk_offering_id = disk_offering.id left join cloud.storage_pool ON volumes.pool_id = storage_pool.id left join cloud.cluster ON storage_pool.cluster_id = cluster.id left join cloud.vm_template ON volumes.template_id = vm_template.id left join cloud.vm_template iso ON iso.id = volumes.iso_id left join cloud.resource_tags ON resource_tags.resource_id = volumes.id and resource_tags.resource_type = 'Volume' left join cloud.async_job ON async_job.instance_id = volumes.id and async_job.instance_type = 'Volume' and async_job.job_status = 0
    2023-03-22 12:32:27,551 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CREATE VIEW cloud.volume_view AS select volumes.id, volumes.uuid, volumes.name, volumes.device_id, volumes.volume_type, volumes.provisioning_type, volumes.size, volumes.min_iops, volumes.max_iops, volumes.created, volumes.state, volumes.attached, volumes.removed, volumes.pod_id, volumes.display_volume, volumes.format, volumes.path, volumes.chain_info, account.id account_id, account.uuid account_uuid, account.account_name account_name, account.type account_type, domain.id domain_id, domain.uuid domain_uuid, domain.name domain_name, domain.path domain_path, projects.id project_id, projects.uuid project_uuid, projects.name project_name, data_center.id data_center_id, data_center.uuid data_center_uuid, data_center.name data_center_name, data_center.networktype data_center_type, vm_instance.id vm_id, vm_instance.uuid vm_uuid, vm_instance.name vm_name, vm_instance.state vm_state, vm_instance.vm_type, user_vm.display_name vm_display_name, volume_store_ref.size volume_store_size, volume_store_ref.download_pct, volume_store_ref.download_state, volume_store_ref.error_str, volume_store_ref.created created_on_store, disk_offering.id disk_offering_id, disk_offering.uuid disk_offering_uuid, disk_offering.name disk_offering_name, disk_offering.display_text disk_offering_display_text, disk_offering.use_local_storage, disk_offering.system_use, disk_offering.bytes_read_rate, disk_offering.bytes_write_rate, disk_offering.iops_read_rate, disk_offering.iops_write_rate, disk_offering.cache_mode, storage_pool.id pool_id, storage_pool.uuid pool_uuid, storage_pool.name pool_name, cluster.hypervisor_type, vm_template.id template_id, vm_template.uuid template_uuid, vm_template.extractable, vm_template.type template_type, vm_template.name template_name, vm_template.display_text template_display_text, iso.id iso_id, iso.uuid iso_uuid, iso.name iso_name, iso.display_text iso_display_text, resource_tags.id tag_id, resource_tags.uuid tag_uuid, resource_tags.key tag_key, resource_tags.value tag_value, resource_tags.domain_id tag_domain_id, resource_tags.account_id tag_account_id, resource_tags.resource_id tag_resource_id, resource_tags.resource_uuid tag_resource_uuid, resource_tags.resource_type tag_resource_type, resource_tags.customer tag_customer, async_job.id job_id, async_job.uuid job_uuid, async_job.job_status job_status, async_job.account_id job_account_id from cloud.volumes inner join cloud.account ON volumes.account_id = account.id inner join cloud.domain ON volumes.domain_id = domain.id left join cloud.projects ON projects.project_account_id = account.id left join cloud.data_center ON volumes.data_center_id = data_center.id left join cloud.vm_instance ON volumes.instance_id = vm_instance.id left join cloud.user_vm ON user_vm.id = vm_instance.id left join cloud.volume_store_ref ON volumes.id = volume_store_ref.volume_id left join cloud.disk_offering ON volumes.disk_offering_id = disk_offering.id left join cloud.storage_pool ON volumes.pool_id = storage_pool.id left join cloud.cluster ON storage_pool.cluster_id = cluster.id left join cloud.vm_template ON volumes.template_id = vm_template.id left join cloud.vm_template iso ON iso.id = volumes.iso_id left join cloud.resource_tags ON resource_tags.resource_id = volumes.id and resource_tags.resource_type = 'Volume' left join cloud.async_job ON async_job.instance_id = volumes.id and async_job.instance_type = 'Volume' and async_job.job_status = 0
    2023-03-22 12:32:27,607 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CREATE VIEW cloud.template_view AS select vm_template.id, vm_template.uuid, vm_template.unique_name, vm_template.name, vm_template.public, vm_template.featured, vm_template.type, vm_template.hvm, vm_template.bits, vm_template.url, vm_template.format, vm_template.created, vm_template.checksum, vm_template.display_text, vm_template.enable_password, vm_template.dynamically_scalable, vm_template.state template_state, vm_template.guest_os_id, guest_os.uuid guest_os_uuid, guest_os.display_name guest_os_name, vm_template.bootable, vm_template.prepopulate, vm_template.cross_zones, vm_template.hypervisor_type, vm_template.extractable, vm_template.template_tag, vm_template.sort_key, vm_template.removed, vm_template.enable_sshkey, source_template.id source_template_id, source_template.uuid source_template_uuid, account.id account_id, account.uuid account_uuid, account.account_name account_name, account.type account_type, domain.id domain_id, domain.uuid domain_uuid, domain.name domain_name, domain.path domain_path, projects.id project_id, projects.uuid project_uuid, projects.name project_name, data_center.id data_center_id, data_center.uuid data_center_uuid, data_center.name data_center_name, launch_permission.account_id lp_account_id, template_store_ref.store_id, image_store.scope as store_scope, template_store_ref.state, template_store_ref.download_state, template_store_ref.download_pct, template_store_ref.error_str, template_store_ref.size, template_store_ref.destroyed, template_store_ref.created created_on_store, vm_template_details.name detail_name, vm_template_details.value detail_value, resource_tags.id tag_id, resource_tags.uuid tag_uuid, resource_tags.key tag_key, resource_tags.value tag_value, resource_tags.domain_id tag_domain_id, resource_tags.account_id tag_account_id, resource_tags.resource_id tag_resource_id, resource_tags.resource_uuid tag_resource_uuid, resource_tags.resource_type tag_resource_type, resource_tags.customer tag_customer, CONCAT(vm_template.id, '', IFNULL(data_center.id, 0)) as temp_zone_pair from cloud.vm_template inner join cloud.guest_os ON guest_os.id = vm_template.guest_os_id inner join cloud.account ON account.id = vm_template.account_id inner join cloud.domain ON domain.id = account.domain_id left join cloud.projects ON projects.project_account_id = account.id left join cloud.vm_template_details ON vm_template_details.template_id = vm_template.id left join cloud.vm_template source_template ON source_template.id = vm_template.source_template_id left join cloud.template_store_ref ON template_store_ref.template_id = vm_template.id and template_store_ref.store_role = 'Image' and template_store_ref.destroyed=0 left join cloud.image_store ON image_store.removed is NULL AND template_store_ref.store_id is not NULL AND image_store.id = template_store_ref.store_id left join cloud.template_zone_ref ON template_zone_ref.template_id = vm_template.id AND template_store_ref.store_id is NULL AND template_zone_ref.removed is null left join cloud.data_center ON (image_store.data_center_id = data_center.id OR template_zone_ref.zone_id = data_center.id) left join cloud.launch_permission ON launch_permission.template_id = vm_template.id left join cloud.resource_tags ON resource_tags.resource_id = vm_template.id and (resource_tags.resource_type = 'Template' or resource_tags.resource_type='ISO')
    2023-03-22 12:32:30,396 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Ignored SQL Exception when trying to drop key i_vm_instance__instance_name on table vm_instance exception: Can't DROP 'i_vm_instance__instance_name'; check that column/key exists
    2023-03-22 12:32:30,733 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CREATE VIEW template_view AS SELECT vm_template.id AS id, vm_template.uuid AS uuid, vm_template.unique_name AS unique_name, vm_template.name AS name, vm_template.public AS public, vm_template.featured AS featured, vm_template.type AS type, vm_template.hvm AS hvm, vm_template.bits AS bits, vm_template.url AS url, vm_template.format AS format, vm_template.created AS created, vm_template.checksum AS checksum, vm_template.display_text AS display_text, vm_template.enable_password AS enable_password, vm_template.dynamically_scalable AS dynamically_scalable, vm_template.state AS template_state, vm_template.guest_os_id AS guest_os_id, guest_os.uuid AS guest_os_uuid, guest_os.display_name AS guest_os_name, vm_template.bootable AS bootable, vm_template.prepopulate AS prepopulate, vm_template.cross_zones AS cross_zones, vm_template.hypervisor_type AS hypervisor_type, vm_template.extractable AS extractable, vm_template.template_tag AS template_tag, vm_template.sort_key AS sort_key, vm_template.removed AS removed, vm_template.enable_sshkey AS enable_sshkey, source_template.id AS source_template_id, source_template.uuid AS source_template_uuid, account.id AS account_id, account.uuid AS account_uuid, account.account_name AS account_name, account.type AS account_type, domain.id AS domain_id, domain.uuid AS domain_uuid, domain.name AS domain_name, domain.path AS domain_path, projects.id AS project_id, projects.uuid AS project_uuid, projects.name AS project_name, data_center.id AS data_center_id, data_center.uuid AS data_center_uuid, data_center.name AS data_center_name, launch_permission.account_id AS lp_account_id, template_store_ref.store_id AS store_id, image_store.scope AS store_scope, template_store_ref.state AS state, template_store_ref.download_state AS download_state, template_store_ref.download_pct AS download_pct, template_store_ref.error_str AS error_str, template_store_ref.size AS size, template_store_ref.destroyed AS destroyed, template_store_ref.created AS created_on_store, vm_template_details.name AS detail_name, vm_template_details.value AS detail_value, resource_tags.id AS tag_id, resource_tags.uuid AS tag_uuid, resource_tags.key AS tag_key, resource_tags.value AS tag_value, resource_tags.domain_id AS tag_domain_id, domain.uuid AS tag_domain_uuid, domain.name AS tag_domain_name, resource_tags.account_id AS tag_account_id, account.account_name AS tag_account_name, resource_tags.resource_id AS tag_resource_id, resource_tags.resource_uuid AS tag_resource_uuid, resource_tags.resource_type AS tag_resource_type, resource_tags.customer AS tag_customer, CONCAT(vm_template.id, '    ', IFNULL(data_center.id, 0)) AS temp_zone_pair FROM ((((((((((((vm_template JOIN guest_os ON ((guest_os.id = vm_template.guest_os_id))) JOIN account ON ((account.id = vm_template.account_id))) JOIN domain ON ((domain.id = account.domain_id))) LEFT JOIN projects ON ((projects.project_account_id = account.id))) LEFT JOIN vm_template_details ON ((vm_template_details.template_id = vm_template.id))) LEFT JOIN vm_template source_template ON ((source_template.id = vm_template.source_template_id))) LEFT JOIN template_store_ref ON (((template_store_ref.template_id = vm_template.id) AND (template_store_ref.store_role = 'Image') AND (template_store_ref.destroyed = 0)))) LEFT JOIN image_store ON ((ISNULL(image_store.removed) AND (template_store_ref.store_id IS NOT NULL) AND (image_store.id = template_store_ref.store_id)))) LEFT JOIN template_zone_ref ON (((template_zone_ref.template_id = vm_template.id) AND ISNULL(template_store_ref.store_id) AND ISNULL(template_zone_ref.removed)))) LEFT JOIN data_center ON (((image_store.data_center_id = data_center.id) OR (template_zone_ref.zone_id = data_center.id)))) LEFT JOIN launch_permission ON ((launch_permission.template_id = vm_template.id))) LEFT JOIN resource_tags ON (((resource_tags.resource_id = vm_template.id) AND ((resource_tags.resource_type = 'Template') OR (resource_tags.resource_type = 'ISO')))))
    2023-03-22 12:32:30,736 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CREATE VIEW volume_view AS SELECT volumes.id AS id, volumes.uuid AS uuid, volumes.name AS name, volumes.device_id AS device_id, volumes.volume_type AS volume_type, volumes.provisioning_type AS provisioning_type, volumes.size AS size, volumes.min_iops AS min_iops, volumes.max_iops AS max_iops, volumes.created AS created, volumes.state AS state, volumes.attached AS attached, volumes.removed AS removed, volumes.pod_id AS pod_id, volumes.display_volume AS display_volume, volumes.format AS format, volumes.path AS path, volumes.chain_info AS chain_info, account.id AS account_id, account.uuid AS account_uuid, account.account_name AS account_name, account.type AS account_type, domain.id AS domain_id, domain.uuid AS domain_uuid, domain.name AS domain_name, domain.path AS domain_path, projects.id AS project_id, projects.uuid AS project_uuid, projects.name AS project_name, data_center.id AS data_center_id, data_center.uuid AS data_center_uuid, data_center.name AS data_center_name, data_center.networktype AS data_center_type, vm_instance.id AS vm_id, vm_instance.uuid AS vm_uuid, vm_instance.name AS vm_name, vm_instance.state AS vm_state, vm_instance.vm_type AS vm_type, user_vm.display_name AS vm_display_name, volume_store_ref.size AS volume_store_size, volume_store_ref.download_pct AS download_pct, volume_store_ref.download_state AS download_state, volume_store_ref.error_str AS error_str, volume_store_ref.created AS created_on_store, disk_offering.id AS disk_offering_id, disk_offering.uuid AS disk_offering_uuid, disk_offering.name AS disk_offering_name, disk_offering.display_text AS disk_offering_display_text, disk_offering.use_local_storage AS use_local_storage, disk_offering.system_use AS system_use, disk_offering.bytes_read_rate AS bytes_read_rate, disk_offering.bytes_write_rate AS bytes_write_rate, disk_offering.iops_read_rate AS iops_read_rate, disk_offering.iops_write_rate AS iops_write_rate, disk_offering.cache_mode AS cache_mode, storage_pool.id AS pool_id, storage_pool.uuid AS pool_uuid, storage_pool.name AS pool_name, cluster.hypervisor_type AS hypervisor_type, vm_template.id AS template_id, vm_template.uuid AS template_uuid, vm_template.extractable AS extractable, vm_template.type AS template_type, vm_template.name AS template_name, vm_template.display_text AS template_display_text, iso.id AS iso_id, iso.uuid AS iso_uuid, iso.name AS iso_name, iso.display_text AS iso_display_text, resource_tags.id AS tag_id, resource_tags.uuid AS tag_uuid, resource_tags.key AS tag_key, resource_tags.value AS tag_value, resource_tags.domain_id AS tag_domain_id, domain.uuid AS tag_domain_uuid, domain.name AS tag_domain_name, resource_tags.account_id AS tag_account_id, account.account_name AS tag_account_name, resource_tags.resource_id AS tag_resource_id, resource_tags.resource_uuid AS tag_resource_uuid, resource_tags.resource_type AS tag_resource_type, resource_tags.customer AS tag_customer, async_job.id AS job_id, async_job.uuid AS job_uuid, async_job.job_status AS job_status, async_job.account_id AS job_account_id FROM ((((((((((((((volumes JOIN account ON ((volumes.account_id = account.id))) JOIN domain ON ((volumes.domain_id = domain.id))) LEFT JOIN projects ON ((projects.project_account_id = account.id))) LEFT JOIN data_center ON ((volumes.data_center_id = data_center.id))) LEFT JOIN vm_instance ON ((volumes.instance_id = vm_instance.id))) LEFT JOIN user_vm ON ((user_vm.id = vm_instance.id))) LEFT JOIN volume_store_ref ON ((volumes.id = volume_store_ref.volume_id))) LEFT JOIN disk_offering ON ((volumes.disk_offering_id = disk_offering.id))) LEFT JOIN storage_pool ON ((volumes.pool_id = storage_pool.id))) LEFT JOIN cluster ON ((storage_pool.cluster_id = cluster.id))) LEFT JOIN vm_template ON ((volumes.template_id = vm_template.id))) LEFT JOIN vm_template iso ON ((iso.id = volumes.iso_id))) LEFT JOIN resource_tags ON (((resource_tags.resource_id = volumes.id) AND (resource_tags.resource_type = 'Volume')))) LEFT JOIN async_job ON (((async_job.instance_id = volumes.id) AND (async_job.instance_type = 'Volume') AND (async_job.job_status = 0))))
    2023-03-22 12:32:31,954 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CREATE VIEW cloud.volume_view AS SELECT volumes.id, volumes.uuid, volumes.name, volumes.device_id, volumes.volume_type, volumes.provisioning_type, volumes.size, volumes.min_iops, volumes.max_iops, volumes.created, volumes.state, volumes.attached, volumes.removed, volumes.display_volume, volumes.format, volumes.path, volumes.chain_info, account.id account_id, account.uuid account_uuid, account.account_name account_name, account.type account_type, domain.id domain_id, domain.uuid domain_uuid, domain.name domain_name, domain.path domain_path, projects.id project_id, projects.uuid project_uuid, projects.name project_name, data_center.id data_center_id, data_center.uuid data_center_uuid, data_center.name data_center_name, data_center.networktype data_center_type, vm_instance.id vm_id, vm_instance.uuid vm_uuid, vm_instance.name vm_name, vm_instance.state vm_state, vm_instance.vm_type, user_vm.display_name vm_display_name, volume_store_ref.size volume_store_size, volume_store_ref.download_pct, volume_store_ref.download_state, volume_store_ref.error_str, volume_store_ref.created created_on_store, disk_offering.id disk_offering_id, disk_offering.uuid disk_offering_uuid, disk_offering.name disk_offering_name, disk_offering.display_text disk_offering_display_text, disk_offering.use_local_storage, disk_offering.system_use, disk_offering.bytes_read_rate, disk_offering.bytes_write_rate, disk_offering.iops_read_rate, disk_offering.iops_write_rate, disk_offering.cache_mode, storage_pool.id pool_id, storage_pool.uuid pool_uuid, storage_pool.name pool_name, cluster.id cluster_id, cluster.name cluster_name, cluster.uuid cluster_uuid, cluster.hypervisor_type, vm_template.id template_id, vm_template.uuid template_uuid, vm_template.extractable, vm_template.type template_type, vm_template.name template_name, vm_template.display_text template_display_text, iso.id iso_id, iso.uuid iso_uuid, iso.name iso_name, iso.display_text iso_display_text, resource_tags.id tag_id, resource_tags.uuid tag_uuid, resource_tags.key tag_key, resource_tags.value tag_value, resource_tags.domain_id tag_domain_id, resource_tags.account_id tag_account_id, resource_tags.resource_id tag_resource_id, resource_tags.resource_uuid tag_resource_uuid, resource_tags.resource_type tag_resource_type, resource_tags.customer tag_customer, async_job.id job_id, async_job.uuid job_uuid, async_job.job_status job_status, async_job.account_id job_account_id, host_pod_ref.id pod_id, host_pod_ref.uuid pod_uuid, host_pod_ref.name pod_name, resource_tag_account.account_name tag_account_name, resource_tag_domain.uuid tag_domain_uuid, resource_tag_domain.name tag_domain_name from cloud.volumes inner join cloud.account ON volumes.account_id = account.id inner join cloud.domain ON volumes.domain_id = domain.id left join cloud.projects ON projects.project_account_id = account.id left join cloud.data_center ON volumes.data_center_id = data_center.id left join cloud.vm_instance ON volumes.instance_id = vm_instance.id left join cloud.user_vm ON user_vm.id = vm_instance.id left join cloud.volume_store_ref ON volumes.id = volume_store_ref.volume_id left join cloud.disk_offering ON volumes.disk_offering_id = disk_offering.id left join cloud.storage_pool ON volumes.pool_id = storage_pool.id left join cloud.host_pod_ref ON storage_pool.pod_id = host_pod_ref.id left join cloud.cluster ON storage_pool.cluster_id = cluster.id left join cloud.vm_template ON volumes.template_id = vm_template.id left join cloud.vm_template iso ON iso.id = volumes.iso_id left join cloud.resource_tags ON resource_tags.resource_id = volumes.id and resource_tags.resource_type = 'Volume' left join cloud.async_job ON async_job.instance_id = volumes.id and async_job.instance_type = 'Volume' and async_job.job_status = 0 left join cloud.account resource_tag_account ON resource_tag_account.id = resource_tags.account_id left join cloud.domain resource_tag_domain ON resource_tag_domain.id = resource_tags.domain_id
    2023-03-22 12:32:32,147 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CREATE VIEW cloud.template_view AS SELECT vm_template.id AS id, vm_template.uuid AS uuid, vm_template.unique_name AS unique_name, vm_template.name AS name, vm_template.public AS public, vm_template.featured AS featured, vm_template.type AS type, vm_template.hvm AS hvm, vm_template.bits AS bits, vm_template.url AS url, vm_template.format AS format, vm_template.created AS created, vm_template.checksum AS checksum, vm_template.display_text AS display_text, vm_template.enable_password AS enable_password, vm_template.dynamically_scalable AS dynamically_scalable, vm_template.state AS template_state, vm_template.guest_os_id AS guest_os_id, guest_os.uuid AS guest_os_uuid, guest_os.display_name AS guest_os_name, vm_template.bootable AS bootable, vm_template.prepopulate AS prepopulate, vm_template.cross_zones AS cross_zones, vm_template.hypervisor_type AS hypervisor_type, vm_template.extractable AS extractable, vm_template.template_tag AS template_tag, vm_template.sort_key AS sort_key, vm_template.removed AS removed, vm_template.enable_sshkey AS enable_sshkey, parent_template.id AS parent_template_id, parent_template.uuid AS parent_template_uuid, source_template.id AS source_template_id, source_template.uuid AS source_template_uuid, account.id AS account_id, account.uuid AS account_uuid, account.account_name AS account_name, account.type AS account_type, domain.id AS domain_id, domain.uuid AS domain_uuid, domain.name AS domain_name, domain.path AS domain_path, projects.id AS project_id, projects.uuid AS project_uuid, projects.name AS project_name, data_center.id AS data_center_id, data_center.uuid AS data_center_uuid, data_center.name AS data_center_name, launch_permission.account_id AS lp_account_id, template_store_ref.store_id AS store_id, image_store.scope AS store_scope, template_store_ref.state AS state, template_store_ref.download_state AS download_state, template_store_ref.download_pct AS download_pct, template_store_ref.error_str AS error_str, template_store_ref.size AS size, template_store_ref.physical_size AS physical_size, template_store_ref.destroyed AS destroyed, template_store_ref.created AS created_on_store, vm_template_details.name AS detail_name, vm_template_details.value AS detail_value, resource_tags.id AS tag_id, resource_tags.uuid AS tag_uuid, resource_tags.key AS tag_key, resource_tags.value AS tag_value, resource_tags.domain_id AS tag_domain_id, domain.uuid AS tag_domain_uuid, domain.name AS tag_domain_name, resource_tags.account_id AS tag_account_id, account.account_name AS tag_account_name, resource_tags.resource_id AS tag_resource_id, resource_tags.resource_uuid AS tag_resource_uuid, resource_tags.resource_type AS tag_resource_type, resource_tags.customer AS tag_customer, CONCAT(vm_template.id, '', IFNULL(data_center.id, 0)) AS temp_zone_pair, vm_template.direct_download AS direct_download FROM (((((((((((((vm_template JOIN guest_os ON ((guest_os.id = vm_template.guest_os_id))) JOIN account ON ((account.id = vm_template.account_id))) JOIN domain ON ((domain.id = account.domain_id))) LEFT JOIN projects ON ((projects.project_account_id = account.id))) LEFT JOIN vm_template_details ON ((vm_template_details.template_id = vm_template.id))) LEFT JOIN vm_template source_template ON ((source_template.id = vm_template.source_template_id))) LEFT JOIN template_store_ref ON (((template_store_ref.template_id = vm_template.id) AND (template_store_ref.store_role = 'Image') AND (template_store_ref.destroyed = 0)))) LEFT JOIN vm_template parent_template ON ((parent_template.id = vm_template.parent_template_id))) LEFT JOIN image_store ON ((ISNULL(image_store.removed) AND (template_store_ref.store_id IS NOT NULL) AND (image_store.id = template_store_ref.store_id)))) LEFT JOIN template_zone_ref ON (((template_zone_ref.template_id = vm_template.id) AND ISNULL(template_store_ref.store_id) AND ISNULL(template_zone_ref.removed)))) LEFT JOIN data_center ON (((image_store.data_center_id = data_center.id) OR (template_zone_ref.zone_id = data_center.id)))) LEFT JOIN launch_permission ON ((launch_permission.template_id = vm_template.id))) LEFT JOIN resource_tags ON (((resource_tags.resource_id = vm_template.id) AND ((resource_tags.resource_type = 'Template') OR (resource_tags.resource_type = 'ISO')))))
    2023-03-22 12:32:32,301 ERROR [c.c.u.PropertiesUtil] (main:null) (logid:) Unable to find properties file: commands.properties
    2023-03-22 12:32:32,447 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- changed fingerprint type to TEXT, it avoids db exception when creating the certificate issue #3123
    2023-03-22 12:32:33,368 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CREATE VIEW cloud.template_view AS SELECT vm_template.id AS id, vm_template.uuid AS uuid, vm_template.unique_name AS unique_name, vm_template.name AS name, vm_template.public AS public, vm_template.featured AS featured, vm_template.type AS type, vm_template.hvm AS hvm, vm_template.bits AS bits, vm_template.url AS url, vm_template.format AS format, vm_template.created AS created, vm_template.checksum AS checksum, vm_template.display_text AS display_text, vm_template.enable_password AS enable_password, vm_template.dynamically_scalable AS dynamically_scalable, vm_template.state AS template_state, vm_template.guest_os_id AS guest_os_id, guest_os.uuid AS guest_os_uuid, guest_os.display_name AS guest_os_name, vm_template.bootable AS bootable, vm_template.prepopulate AS prepopulate, vm_template.cross_zones AS cross_zones, vm_template.hypervisor_type AS hypervisor_type, vm_template.extractable AS extractable, vm_template.template_tag AS template_tag, vm_template.sort_key AS sort_key, vm_template.removed AS removed, vm_template.enable_sshkey AS enable_sshkey, parent_template.id AS parent_template_id, parent_template.uuid AS parent_template_uuid, source_template.id AS source_template_id, source_template.uuid AS source_template_uuid, account.id AS account_id, account.uuid AS account_uuid, account.account_name AS account_name, account.type AS account_type, domain.id AS domain_id, domain.uuid AS domain_uuid, domain.name AS domain_name, domain.path AS domain_path, projects.id AS project_id, projects.uuid AS project_uuid, projects.name AS project_name, data_center.id AS data_center_id, data_center.uuid AS data_center_uuid, data_center.name AS data_center_name, launch_permission.account_id AS lp_account_id, template_store_ref.store_id AS store_id, image_store.scope AS store_scope, template_store_ref.state AS state, template_store_ref.download_state AS download_state, template_store_ref.download_pct AS download_pct, template_store_ref.error_str AS error_str, template_store_ref.size AS size, template_store_ref.physical_size AS physical_size, template_store_ref.destroyed AS destroyed, template_store_ref.created AS created_on_store, vm_template_details.name AS detail_name, vm_template_details.value AS detail_value, resource_tags.id AS tag_id, resource_tags.uuid AS tag_uuid, resource_tags.key AS tag_key, resource_tags.value AS tag_value, resource_tags.domain_id AS tag_domain_id, domain.uuid AS tag_domain_uuid, domain.name AS tag_domain_name, resource_tags.account_id AS tag_account_id, account.account_name AS tag_account_name, resource_tags.resource_id AS tag_resource_id, resource_tags.resource_uuid AS tag_resource_uuid, resource_tags.resource_type AS tag_resource_type, resource_tags.customer AS tag_customer, CONCAT(vm_template.id, '    ', IFNULL(data_center.id, 0)) AS temp_zone_pair, vm_template.direct_download AS direct_download, vm_template.deploy_as_is AS deploy_as_is FROM (((((((((((((vm_template JOIN guest_os ON ((guest_os.id = vm_template.guest_os_id))) JOIN account ON ((account.id = vm_template.account_id))) JOIN domain ON ((domain.id = account.domain_id))) LEFT JOIN projects ON ((projects.project_account_id = account.id))) LEFT JOIN vm_template_details ON ((vm_template_details.template_id = vm_template.id))) LEFT JOIN vm_template source_template ON ((source_template.id = vm_template.source_template_id))) LEFT JOIN template_store_ref ON (((template_store_ref.template_id = vm_template.id) AND (template_store_ref.store_role = 'Image') AND (template_store_ref.destroyed = 0)))) LEFT JOIN vm_template parent_template ON ((parent_template.id = vm_template.parent_template_id))) LEFT JOIN image_store ON ((ISNULL(image_store.removed) AND (template_store_ref.store_id IS NOT NULL) AND (image_store.id = template_store_ref.store_id)))) LEFT JOIN template_zone_ref ON (((template_zone_ref.template_id = vm_template.id) AND ISNULL(template_store_ref.store_id) AND ISNULL(template_zone_ref.removed)))) LEFT JOIN data_center ON (((image_store.data_center_id = data_center.id) OR (template_zone_ref.zone_id = data_center.id)))) LEFT JOIN launch_permission ON ((launch_permission.template_id = vm_template.id))) LEFT JOIN resource_tags ON (((resource_tags.resource_id = vm_template.id) AND ((resource_tags.resource_type = 'Template') OR (resource_tags.resource_type = 'ISO')))))
    2023-03-22 12:32:34,847 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CREATE VIEW cloud.volume_view AS SELECT volumes.id, volumes.uuid, volumes.name, volumes.device_id, volumes.volume_type, volumes.provisioning_type, volumes.size, volumes.min_iops, volumes.max_iops, volumes.created, volumes.state, volumes.attached, volumes.removed, volumes.display_volume, volumes.format, volumes.path, volumes.chain_info, volumes.external_uuid, account.id account_id, account.uuid account_uuid, account.account_name account_name, account.type account_type, domain.id domain_id, domain.uuid domain_uuid, domain.name domain_name, domain.path domain_path, projects.id project_id, projects.uuid project_uuid, projects.name project_name, data_center.id data_center_id, data_center.uuid data_center_uuid, data_center.name data_center_name, data_center.networktype data_center_type, vm_instance.id vm_id, vm_instance.uuid vm_uuid, vm_instance.name vm_name, vm_instance.state vm_state, vm_instance.vm_type, user_vm.display_name vm_display_name, volume_store_ref.size volume_store_size, volume_store_ref.download_pct, volume_store_ref.download_state, volume_store_ref.error_str, volume_store_ref.created created_on_store, disk_offering.id disk_offering_id, disk_offering.uuid disk_offering_uuid, disk_offering.name disk_offering_name, disk_offering.display_text disk_offering_display_text, disk_offering.use_local_storage, service_offering.system_use, disk_offering.bytes_read_rate, disk_offering.bytes_write_rate, disk_offering.iops_read_rate, disk_offering.iops_write_rate, disk_offering.cache_mode, storage_pool.id pool_id, storage_pool.uuid pool_uuid, storage_pool.name pool_name, cluster.id cluster_id, cluster.name cluster_name, cluster.uuid cluster_uuid, cluster.hypervisor_type, vm_template.id template_id, vm_template.uuid template_uuid, vm_template.extractable, vm_template.type template_type, vm_template.name template_name, vm_template.display_text template_display_text, iso.id iso_id, iso.uuid iso_uuid, iso.name iso_name, iso.display_text iso_display_text, resource_tags.id tag_id, resource_tags.uuid tag_uuid, resource_tags.key tag_key, resource_tags.value tag_value, resource_tags.domain_id tag_domain_id, resource_tags.account_id tag_account_id, resource_tags.resource_id tag_resource_id, resource_tags.resource_uuid tag_resource_uuid, resource_tags.resource_type tag_resource_type, resource_tags.customer tag_customer, async_job.id job_id, async_job.uuid job_uuid, async_job.job_status job_status, async_job.account_id job_account_id, host_pod_ref.id pod_id, host_pod_ref.uuid pod_uuid, host_pod_ref.name pod_name, resource_tag_account.account_name tag_account_name, resource_tag_domain.uuid tag_domain_uuid, resource_tag_domain.name tag_domain_name from cloud.volumes inner join cloud.account ON volumes.account_id = account.id inner join cloud.domain ON volumes.domain_id = domain.id left join cloud.projects ON projects.project_account_id = account.id left join cloud.data_center ON volumes.data_center_id = data_center.id left join cloud.vm_instance ON volumes.instance_id = vm_instance.id left join cloud.user_vm ON user_vm.id = vm_instance.id left join cloud.volume_store_ref ON volumes.id = volume_store_ref.volume_id left join cloud.service_offering ON vm_instance.service_offering_id = service_offering.id left join cloud.disk_offering ON volumes.disk_offering_id = disk_offering.id left join cloud.storage_pool ON volumes.pool_id = storage_pool.id left join cloud.host_pod_ref ON storage_pool.pod_id = host_pod_ref.id left join cloud.cluster ON storage_pool.cluster_id = cluster.id left join cloud.vm_template ON volumes.template_id = vm_template.id left join cloud.vm_template iso ON iso.id = volumes.iso_id left join cloud.resource_tags ON resource_tags.resource_id = volumes.id and resource_tags.resource_type = 'Volume' left join cloud.async_job ON async_job.instance_id = volumes.id and async_job.instance_type = 'Volume' and async_job.job_status = 0 left join cloud.account resource_tag_account ON resource_tag_account.id = resource_tags.account_id left join cloud.domain resource_tag_domain ON resource_tag_domain.id = resource_tags.domain_id
    2023-03-22 12:32:35,820 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) Error executing: CALL cloud.IDEMPOTENT_ADD_FOREIGN_KEY('cloud.vm_template', 'user_data', 'id')
    2023-03-22 12:32:35,820 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) java.sql.SQLIntegrityConstraintViolationException: Can't write; duplicate key in table '#sql-3755_b'
    2023-03-22 12:32:35,821 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to execute upgrade script
    java.sql.SQLIntegrityConstraintViolationException: Can't write; duplicate key in table '#sql-3755_b'
    2023-03-22 12:32:35,824 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to upgrade the database
    com.cloud.utils.exception.CloudRuntimeException: Unable to execute upgrade script
    Caused by: java.sql.SQLIntegrityConstraintViolationException: Can't write; duplicate key in table '#sql-3755_b'
    2023-03-22 12:54:31,236 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) Error executing: ALTER TABLE cloud.networks ADD COLUMN public_mtu bigint unsigned comment "MTU for VR public interface"
    2023-03-22 12:54:31,237 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
    2023-03-22 12:54:31,238 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to execute upgrade script
    java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
    2023-03-22 12:54:31,241 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to upgrade the database
    com.cloud.utils.exception.CloudRuntimeException: Unable to execute upgrade script
    Caused by: java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
    2023-03-22 12:59:09,842 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) Error executing: ALTER TABLE cloud.networks ADD COLUMN public_mtu bigint unsigned comment "MTU for VR public interface"
    2023-03-22 12:59:09,842 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
    2023-03-22 12:59:09,843 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to execute upgrade script
    java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
    2023-03-22 12:59:09,846 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to upgrade the database
    com.cloud.utils.exception.CloudRuntimeException: Unable to execute upgrade script
    Caused by: java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
    2023-03-22 13:35:43,354 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) Error executing: ALTER TABLE cloud.networks ADD COLUMN public_mtu bigint unsigned comment "MTU for VR public interface"
    2023-03-22 13:35:43,354 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
    2023-03-22 13:35:43,355 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to execute upgrade script
    java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
    2023-03-22 13:35:43,358 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to upgrade the database
    com.cloud.utils.exception.CloudRuntimeException: Unable to execute upgrade script
    Caused by: java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
    2023-03-22 14:29:42,394 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) Error executing: ALTER TABLE cloud.networks ADD COLUMN public_mtu bigint unsigned comment "MTU for VR public interface"
    2023-03-22 14:29:42,395 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
    2023-03-22 14:29:42,396 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to execute upgrade script
    java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
    2023-03-22 14:29:42,399 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to upgrade the database
    com.cloud.utils.exception.CloudRuntimeException: Unable to execute upgrade script
    Caused by: java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
    2023-03-23 07:20:40,100 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) Error executing: ALTER TABLE cloud.networks ADD COLUMN public_mtu bigint unsigned comment "MTU for VR public interface"
    2023-03-23 07:20:40,101 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
    2023-03-23 07:20:40,102 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to execute upgrade script
    java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
    2023-03-23 07:20:40,105 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to upgrade the database
    com.cloud.utils.exception.CloudRuntimeException: Unable to execute upgrade script
    Caused by: java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'

    Access log

    45.128.232.112 - - [22/Mar/2023:04:11:22 +0000] "GET / HTTP/1.1" 301 0 "-" "Linux Gnu (cow)" 47
    205.210.31.31 - - [22/Mar/2023:04:23:47 +0000] "GET / HTTP/1.0" 301 0 "-" "Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com" 3
    43.158.217.52 - - [22/Mar/2023:04:25:22 +0000] "GET / HTTP/1.1" 301 0 "-" "'Mozilla/5.0" 2
    192.168.219.252 - - [22/Mar/2023:04:26:07 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111.0.1661.44" 19
    192.168.219.252 - - [22/Mar/2023:04:26:07 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111.0.1661.44" 3
    192.168.219.252 - - [22/Mar/2023:04:26:21 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111.0.1661.44" 3
    192.168.219.252 - - [22/Mar/2023:04:26:21 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111.0.1661.44" 2
    192.168.219.252 - - [22/Mar/2023:04:27:57 +0000] "GET /client/api/?command=listIdps&response=json HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 4
    192.168.219.252 - - [22/Mar/2023:04:28:13 +0000] "GET /client/api/?command=logout&response=json HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:28:13 +0000] "POST /client/api/ HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 3
    192.168.219.252 - - [22/Mar/2023:04:28:45 +0000] "POST /client/api/ HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:28:45 +0000] "GET /client/api/?command=logout&response=json HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:28:55 +0000] "GET /client/api/?command=logout&response=json HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:28:55 +0000] "POST /client/api/ HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:29:23 +0000] "GET /client/api/?command=listIdps&response=json HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:29:34 +0000] "GET /client/api/?command=logout&response=json HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 4
    192.168.219.252 - - [22/Mar/2023:04:29:34 +0000] "POST /client/api/ HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:04:30:36 +0000] "GET /client/api/?command=listIdps&response=json HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:30:54 +0000] "POST /client/api/ HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:30:54 +0000] "GET /client/api/?command=logout&response=json HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:31:07 +0000] "POST /client/api/ HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:31:07 +0000] "GET /client/api/?command=logout&response=json HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:32:04 +0000] "POST /client/api/ HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 3
    192.168.219.252 - - [22/Mar/2023:04:32:04 +0000] "GET /client/api/?command=logout&response=json HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 3
    192.168.219.252 - - [22/Mar/2023:04:32:10 +0000] "GET /client/api/?command=logout&response=json HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 3
    192.168.219.252 - - [22/Mar/2023:04:32:10 +0000] "POST /client/api/ HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 3
    192.168.219.252 - - [22/Mar/2023:04:32:16 +0000] "POST /client/api/ HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:04:32:16 +0000] "GET /client/api/?command=logout&response=json HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:32:33 +0000] "POST /client/api/ HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:32:33 +0000] "GET /client/api/?command=logout&response=json HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:32:39 +0000] "POST /client/api/ HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:04:32:39 +0000] "GET /client/api/?command=logout&response=json HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:32:45 +0000] "POST /client/api/ HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:04:32:45 +0000] "GET /client/api/?command=logout&response=json HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:04:33:02 +0000] "POST /client/api/ HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:33:02 +0000] "GET /client/api/?command=logout&response=json HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:04:50:23 +0000] "GET /client/api/?command=listIdps&response=json HTTP/1.1" 503 73 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 59
    192.168.219.252 - - [22/Mar/2023:04:51:00 +0000] "GET / HTTP/1.1" 301 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 3
    192.168.219.252 - - [22/Mar/2023:04:51:00 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:51:00 +0000] "GET /favicon.ico HTTP/1.1" 301 0 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:04:51:00 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:04:51:06 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:51:07 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:51:07 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:04:51:17 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 3
    192.168.219.252 - - [22/Mar/2023:04:51:24 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:51:24 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    211.226.15.92 - - [22/Mar/2023:04:51:48 +0000] "GET / HTTP/1.1" 301 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    211.226.15.92 - - [22/Mar/2023:04:51:48 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    211.226.15.92 - - [22/Mar/2023:04:51:49 +0000] "GET /favicon.ico HTTP/1.1" 301 0 "http://www.kitrocloud.com/client/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    211.226.15.92 - - [22/Mar/2023:04:51:49 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://www.kitrocloud.com/client/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    211.226.15.92 - - [22/Mar/2023:04:52:03 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 3
    211.226.15.92 - - [22/Mar/2023:04:52:03 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://www.kitrocloud.com/client/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    211.226.15.92 - - [22/Mar/2023:04:57:34 +0000] "GET / HTTP/1.1" 301 0 "-" "curl/7.86.0" 1
    127.0.0.1 - - [22/Mar/2023:04:58:26 +0000] "GET / HTTP/1.1" 301 0 "-" "curl/7.29.0" 1
    192.168.219.253 - - [22/Mar/2023:04:58:48 +0000] "GET / HTTP/1.1" 301 0 "-" "curl/7.29.0" 2
    192.168.219.252 - - [22/Mar/2023:04:58:59 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:58:59 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:04:59:42 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:59:42 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:59:43 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:04:59:43 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:59:44 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:04:59:44 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:00:23 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:00:23 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:00:30 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:05:00:30 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.254 - - [22/Mar/2023:05:04:40 +0000] "GET / HTTP/1.1" 301 0 "-" "curl/7.29.0" 2
    192.168.219.254 - - [22/Mar/2023:05:04:49 +0000] "GET /client HTTP/1.1" 302 0 "-" "curl/7.29.0" 3
    192.168.219.252 - - [22/Mar/2023:05:05:08 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:05:08 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:05:52 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:05:52 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:06:04 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:05:06:04 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:05:07:24 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:07:24 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:07:29 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 0
    192.168.219.252 - - [22/Mar/2023:05:07:29 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 0
    193.164.194.23 - - [22/Mar/2023:05:07:40 +0000] "GET / HTTP/1.1" 301 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:05:10:02 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:10:02 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:11:01 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 3
    192.168.219.252 - - [22/Mar/2023:05:11:01 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:11:03 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:11:03 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:05:11:04 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:05:11:04 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:05:11:05 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:11:05 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:11:06 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:05:11:06 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:05:11:07 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:05:11:07 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    46.101.78.241 - - [22/Mar/2023:05:14:37 +0000] "GET / HTTP/1.1" 301 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" 1
    46.101.78.241 - - [22/Mar/2023:05:14:37 +0000] "GET /client/ HTTP/1.1" 503 394 "http://220.74.34.204/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" 2
    46.101.78.241 - - [22/Mar/2023:05:14:38 +0000] "GET /client/get_targets HTTP/1.1" 503 405 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" 1
    46.101.78.241 - - [22/Mar/2023:05:14:38 +0000] "GET /upl.php HTTP/1.1" 301 0 "-" "Mozilla/5.0" 1
    46.101.78.241 - - [22/Mar/2023:05:14:49 +0000] "GET /geoip/ HTTP/1.1" 301 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:05:15:39 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:15:39 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:15:41 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:05:15:41 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:05:15:42 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:15:42 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:05:15:43 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:05:15:43 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:05:15:43 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:15:43 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    192.168.219.252 - - [22/Mar/2023:05:15:45 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:15:45 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    211.226.15.92 - - [22/Mar/2023:05:16:03 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    211.226.15.92 - - [22/Mar/2023:05:16:03 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://www.kitrocloud.com/client/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    211.226.15.92 - - [22/Mar/2023:05:19:50 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    211.226.15.92 - - [22/Mar/2023:05:19:50 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://www.kitrocloud.com/client/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 3
    211.226.15.92 - - [22/Mar/2023:05:21:11 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    211.226.15.92 - - [22/Mar/2023:05:21:11 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://www.kitrocloud.com/client/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    211.226.15.92 - - [22/Mar/2023:05:21:13 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 1
    211.226.15.92 - - [22/Mar/2023:05:21:13 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://www.kitrocloud.com/client/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2
    192.168.219.252 - - [22/Mar/2023:05:21:30 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111.0.1661.44" 2
    192.168.219.252 - - [22/Mar/2023:05:21:30 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111.0.1661.44" 1
    192.168.219.252 - - [22/Mar/2023:05:21:36 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111.0.1661.44" 1
    192.168.219.252 - - [22/Mar/2023:05:21:36 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111.0.1661.44" 1
    192.168.219.252 - - [22/Mar/2023:05:21:51 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111.0.1661.44" 3
    192.168.219.252 - - [22/Mar/2023:05:21:52 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111.0.1661.44" 1
    192.168.219.252 - - [22/Mar/2023:05:22:07 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111.0.1661.44" 2
    192.168.219.252 - - [22/Mar/2023:05:22:07 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111.0.1661.44" 2
    179.43.177.242 - - [22/Mar/2023:05:44:16 +0000] "GET / HTTP/1.1" 301 0 "-" "Hello World" 40
    34.122.208.38 - - [22/Mar/2023:06:48:29 +0000] "CONNECT www.google.com:443 HTTP/1.1" 301 0 "-" "Go-http-client/1.1" 3
    149.129.50.37 - - [22/Mar/2023:06:52:37 +0000] "GET http://www.1ucn.com/proxychecker/index.php HTTP/1.1" 301 0 "-" "Mozilla/1.22 (compatible; MSIE 5.01; PalmOS 3.0) EudoraWeb 2" 2
    149.129.50.37 - - [22/Mar/2023:06:52:37 +0000] "GET http://www.1ucn.com/client/proxychecker/index.php HTTP/1.1" 503 416 "-" "Mozilla/1.22 (compatible; MSIE 5.01; PalmOS 3.0) EudoraWeb 2" 21
    182.43.148.98 - - [22/Mar/2023:07:14:54 +0000] "GET / HTTP/1.1" 301 0 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1" 3
    182.43.148.98 - - [22/Mar/2023:07:14:55 +0000] "GET /client/ HTTP/1.1" 503 394 "http://220.74.34.204" "Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1" 1
    179.43.177.242 - - [22/Mar/2023:07:16:50 +0000] "GET / HTTP/1.1" 301 0 "-" "Hello World" 2
    192.241.223.20 - - [22/Mar/2023:08:00:59 +0000] "GET / HTTP/1.1" 301 0 "-" "Mozilla/5.0 zgrab/0.x" 3
    117.199.144.224 - - [22/Mar/2023:08:58:22 +0000] "GET / HTTP/1.1" 301 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" 2
    117.199.144.224 - - [22/Mar/2023:08:58:24 +0000] "GET / HTTP/1.1" 301 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36" 1
    43.230.156.235 - - [22/Mar/2023:09:06:39 +0000] "GET / HTTP/1.1" 301 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" 1
    10.254.1.134 - - [22/Mar/2023:22:26:24 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 51
    10.254.1.134 - - [22/Mar/2023:22:26:24 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 3
    10.254.1.134 - - [22/Mar/2023:22:26:29 +0000] "GET /client/ HTTP/1.1" 503 394 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 3
    10.254.1.134 - - [22/Mar/2023:22:26:29 +0000] "GET /client/favicon.ico HTTP/1.1" 503 405 "http://192.168.219.253:8080/client/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2

  6. DaanHoogland commented on Mar 23, 2023

    @DaanHoogland
    DaanHoogland
    on Mar 23, 2023
    Contributor

    @kenneth0595 the essential error message is

    2023-03-22 12:32:35,820 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) Error executing: CALL cloud.IDEMPOTENT_ADD_FOREIGN_KEY('cloud.vm_template', 'user_data', 'id')
2023-03-22 12:32:35,820 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) java.sql.SQLIntegrityConstraintViolationException: Can't write; duplicate key in table '#sql-3755_b'

    This has been extensively tested so it shouldn't happen in your setup.

    We need to find more info somehow. you say you used yum update, but on a clean install of the os. Can you try with yum install instead? and with a clean DB of course.

  7. weizhouapache commented on Mar 23, 2023

    @weizhouapache
    weizhouapache
    on Mar 23, 2023
    Member

    @DaanHoogland
    it is the same issue as Sanjay said in the users mailing list.
    it might be caused by different mysql/mariadb versions.

    @kenneth0595
    can you share some information of your database ?

  8. weizhouapache commented on Mar 23, 2023

    @weizhouapache
    weizhouapache
    on Mar 23, 2023
    Member

    Sanjay Kumar has updated in the users mailing list

    I was using mysql 5.7.41 and now we've upgraded the mysql version and it's working now.

  9. kenneth0595 commented on Mar 23, 2023

    @kenneth0595
    kenneth0595
    on Mar 23, 2023
    Author

    @weizhouapache
    mysql> select version();
    +------------+
    | version() |
    +------------+
    | 5.6.51-log |
    +------------+

  10. DaanHoogland commented on Mar 24, 2023

    @DaanHoogland
    DaanHoogland
    on Mar 24, 2023
    Contributor

    that is even lower than the 5.7.41 reported by Sanjay, Can you upgrade @kenneth0595 ?

  11. kenneth0595 commented on Mar 24, 2023

    @kenneth0595
    kenneth0595
    on Mar 24, 2023
    Author

    @DaanHoogland
    I've just downgraded to version 4.17.2 because it is production that shoud host services.
    4.17.2 has no any problem.

  12. kenneth0595
    closed this as completedon Mar 24, 2023

  13. mail2lawi commented on Mar 24, 2023

    @mail2lawi
    mail2lawi
    on Mar 24, 2023

    Can you take a look in /var/log/cloudstack/management/management-server.log and /var/log/cloudstack/management/access.log?

    Look for errors and exceptions being thrown.

    Just for FYI this is the error on MySQL 5.7.1

    Mar 24 12:11:54 mgmt1.svr.co java[2802]: INFO  [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) init = null
Mar 24 12:11:54 mgmt1.svr.co java[2802]: INFO  [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) DB version = 4.17.2.0 Code Version = 4.18.0.0
Mar 24 12:11:54 mgmt1.svr.co java[2802]: INFO  [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Database upgrade must be performed from 4.17.2.0 to 4.18.0.0
Mar 24 12:11:54 mgmt1.svr.co java[2802]: ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) Error executing: ALTER TABLE `cloud`.`networks` ADD COLUMN `public_mtu` bigint unsigned comment "MTU for VR public interface"
Mar 24 12:11:54 mgmt1.svr.co java[2802]: ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
Mar 24 12:11:54 mgmt1.svr.co java[2802]: ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to execute upgrade script
Mar 24 12:11:54 mgmt1.svr.co java[2802]: java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:185)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:87)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at com.cloud.upgrade.DatabaseUpgradeChecker.runScript(DatabaseUpgradeChecker.java:226)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:310)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:401)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:64)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:54)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at java.base/java.lang.Iterable.forEach(Iterable.java:75)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:935)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:144)

    Let me downgrade to 5.5 and retry

    Just to update, the error is triggered in the last step when setting up the management server, i.e. when one runs the cloudstack-setup-management command

  14. mail2lawi commented on Mar 24, 2023

    @mail2lawi
    mail2lawi
    on Mar 24, 2023

    Second update: Judging from the error I suspected the issue is not with the DB version, rather the script.
    So I installed cloudstack 4.17.2 instead of 4.18 and the install worked fine on the same MySQL 5.7.1 version

    I am suspecting the issue is with the 4.17.2 - 4.18 upgrade script

  15. weizhouapache commented on Mar 24, 2023

    @weizhouapache
    weizhouapache
    on Mar 24, 2023
    Member

    Can you take a look in /var/log/cloudstack/management/management-server.log and /var/log/cloudstack/management/access.log?
    Look for errors and exceptions being thrown.

    Just for FYI this is the error on MySQL 5.7.1

    Mar 24 12:11:54 mgmt1.svr.co java[2802]: INFO  [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) init = null
Mar 24 12:11:54 mgmt1.svr.co java[2802]: INFO  [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) DB version = 4.17.2.0 Code Version = 4.18.0.0
Mar 24 12:11:54 mgmt1.svr.co java[2802]: INFO  [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Database upgrade must be performed from 4.17.2.0 to 4.18.0.0
Mar 24 12:11:54 mgmt1.svr.co java[2802]: ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) Error executing: ALTER TABLE `cloud`.`networks` ADD COLUMN `public_mtu` bigint unsigned comment "MTU for VR public interface"
Mar 24 12:11:54 mgmt1.svr.co java[2802]: ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
Mar 24 12:11:54 mgmt1.svr.co java[2802]: ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to execute upgrade script
Mar 24 12:11:54 mgmt1.svr.co java[2802]: java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:185)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:87)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at com.cloud.upgrade.DatabaseUpgradeChecker.runScript(DatabaseUpgradeChecker.java:226)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:310)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:401)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:64)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:54)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at java.base/java.lang.Iterable.forEach(Iterable.java:75)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:935)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:144)

    Let me downgrade to 5.5 and retry

    Just to update, the error is triggered in the last step when setting up the management server, i.e. when one runs the cloudstack-setup-management command

    If you failed to upgrade, some tables have already been changed, so you need to roll back to your database backup/snapshot before retry.

  16. 1 remaining item

  17. xiongzwen commented on Mar 27, 2023

    @xiongzwen
    xiongzwen
    on Mar 27, 2023

    我使用mysql(版本5.6.51)遇到同样的问题，并且测试过MySQL版本5.7.41。当使用mysql5.5(版本5.5.62)就没有这个问题

  18. weizhouapache
    changed the title [-]Cloudstack 4.18 HTTP 503 Error[/-] [+]Cloudstack 4.18 HTTP 503 Error (MySQL 5.6/5.7)[/+] on Mar 27, 2023

  19. cc20110101 commented on May 4, 2023

    @cc20110101
    cc20110101
    on May 4, 2023

    How to solve this? Will it be fixed

  20. weizhouapache commented on May 4, 2023

    @weizhouapache
    weizhouapache
    on May 4, 2023
    Member

    How to solve this? Will it be fixed

    @cc20110101
    this will be fixed in 4.18.1.0
    please keep eye on #7381

    as a workaround, you can use other mysql versions, e.g. mysql 5.8/8.0 or mariadb

  21. kenneth0595 commented on May 7, 2023

    @kenneth0595
    kenneth0595
    on May 7, 2023
    Author

    I did upgrade mysql version 5.6.51 to 8.0.33 today but it didn't work.
    The same 503 error still appears when I access to browser.

  22. kenneth0595 commented on May 7, 2023

    @kenneth0595
    kenneth0595
    on May 7, 2023
    Author

    I did successfully access to web ui with 4.18.
    Thank you!

  23. xuanyuanaosheng commented on Jun 28, 2023

    @xuanyuanaosheng
    xuanyuanaosheng
    on Jun 28, 2023

    @DaanHoogland @weizhouapache @kenneth0595 @kiwiflyer @cc20110101 @xiongzwen

    my env:

    • os version: CentOS Linux 7 (Core)
    • cloudstack version: 4.17.2.0
    • mysql version: 5.7.34

    when I upgrade cloudstack from 4.17.2.0 to 4.18.0.0
    The error log is same as this issue, through this issue I upgrade mysql version from 5.7.34 to 8.0.33, then I restart cloudstack-management, But it still have this error:

    2023-06-28 17:03:42,210 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) UPDATE `cloud`.`service_offering` so SET so.limit_cpu_use = 1 WHERE so.default_use = 1 AND so.vm_type IN ('domainrouter', 'secondarystoragevm', 'consoleproxy', 'internalloadbalancervm', 'elasticloadbalancervm') 
2023-06-28 17:03:42,214 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) ALTER TABLE `cloud`.`networks` ADD COLUMN `public_mtu` bigint unsigned comment "MTU for VR public interface"  
2023-06-28 17:03:42,265 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) Error executing: ALTER TABLE `cloud`.`networks` ADD COLUMN `public_mtu` bigint unsigned comment "MTU for VR public interface"  
2023-06-28 17:03:42,265 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
2023-06-28 17:03:42,269 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to execute upgrade script
java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
	at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:185)
	at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:87)
	at com.cloud.upgrade.DatabaseUpgradeChecker.runScript(DatabaseUpgradeChecker.java:226)
	at com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:310)
	at com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:401)
	at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:64)
	at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:54)
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178)
	at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54)
	at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155)
	at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123)
	at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:935)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:144)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:121)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:244)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:249)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:232)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:116)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:78)
	at org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
	at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:70)
	at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:57)
	at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:61)
	at org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52)
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
	at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:765)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.server.Server.start(Server.java:423)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.server.Server.doStart(Server.java:387)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:192)
	at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:107)
2023-06-28 17:03:42,276 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to upgrade the database
com.cloud.utils.exception.CloudRuntimeException: Unable to execute upgrade script
	at com.cloud.upgrade.DatabaseUpgradeChecker.runScript(DatabaseUpgradeChecker.java:232)
	at com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:310)
	at com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:401)
	at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:64)
	at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:54)
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178)
	at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54)
	at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155)
	at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123)
	at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:935)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:144)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:121)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:244)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:249)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:232)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:116)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:78)
	at org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
	at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:70)
	at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:57)
	at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:61)
	at org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52)
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
	at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:765)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.server.Server.start(Server.java:423)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.server.Server.doStart(Server.java:387)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:192)
	at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:107)
Caused by: java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
	at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:185)
	at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:87)
	at com.cloud.upgrade.DatabaseUpgradeChecker.runScript(DatabaseUpgradeChecker.java:226)
	... 52 more
2023-06-28 17:03:42,312 DEBUG [c.c.u.d.T.Transaction] (main:null) (logid:) Rolling back the transaction: Time = 260 Name =  Upgrade; called by -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-DatabaseUpgradeChecker.upgrade:325-DatabaseUpgradeChecker.check:401-CloudStackExtendedLifeCycle.checkIntegrity:64-CloudStackExtendedLifeCycle.start:54-DefaultLifecycleProcessor.doStart:178-DefaultLifecycleProcessor.access$200:54-DefaultLifecycleProcessor$LifecycleGroup.start:356-Iterable.forEach:75-DefaultLifecycleProcessor.startBeans:155

    How to repair this?

  24. xuanyuanaosheng commented on Jun 28, 2023

    @xuanyuanaosheng
    xuanyuanaosheng
    on Jun 28, 2023

    I did successfully access to web ui with 4.18. Thank you!

    how to fix this?

  25. xuanyuanaosheng commented on Jun 28, 2023

    @xuanyuanaosheng
    xuanyuanaosheng
    on Jun 28, 2023

    Can you take a look in /var/log/cloudstack/management/management-server.log and /var/log/cloudstack/management/access.log?
    Look for errors and exceptions being thrown.

    Just for FYI this is the error on MySQL 5.7.1

    Mar 24 12:11:54 mgmt1.svr.co java[2802]: INFO  [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) init = null
Mar 24 12:11:54 mgmt1.svr.co java[2802]: INFO  [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) DB version = 4.17.2.0 Code Version = 4.18.0.0
Mar 24 12:11:54 mgmt1.svr.co java[2802]: INFO  [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Database upgrade must be performed from 4.17.2.0 to 4.18.0.0
Mar 24 12:11:54 mgmt1.svr.co java[2802]: ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) Error executing: ALTER TABLE `cloud`.`networks` ADD COLUMN `public_mtu` bigint unsigned comment "MTU for VR public interface"
Mar 24 12:11:54 mgmt1.svr.co java[2802]: ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
Mar 24 12:11:54 mgmt1.svr.co java[2802]: ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to execute upgrade script
Mar 24 12:11:54 mgmt1.svr.co java[2802]: java.sql.SQLSyntaxErrorException: Duplicate column name 'public_mtu'
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:185)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:87)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at com.cloud.upgrade.DatabaseUpgradeChecker.runScript(DatabaseUpgradeChecker.java:226)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:310)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:401)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:64)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:54)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at java.base/java.lang.Iterable.forEach(Iterable.java:75)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:935)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586)
Mar 24 12:11:54 mgmt1.svr.co java[2802]: at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:144)

    Let me downgrade to 5.5 and retry
    Just to update, the error is triggered in the last step when setting up the management server, i.e. when one runs the cloudstack-setup-management command

    If you failed to upgrade, some tables have already been changed, so you need to roll back to your database backup/snapshot before retry.

    If the database has no backup, how to fix this issue ?

  26. weizhouapache commented on Jun 28, 2023

    @weizhouapache
    weizhouapache
    on Jun 28, 2023
    Member

    If the database has no backup, how to fix this issue ?

    @xuanyuanaosheng
    you need to rollback your database to 4.17.2.0, upgrade mysql and then try to upgrade to 4.18.0.0 again

    try again after dropping the following columns

    ALTER TABLE `cloud`.`networks` DROP COLUMN `public_mtu`;
ALTER TABLE `cloud`.`networks` DROP COLUMN `private_mtu` ;
ALTER TABLE `cloud`.`vpc` DROP COLUMN `public_mtu` ;
ALTER TABLE `cloud`.`nics` DROP COLUMN `mtu`;

  27. xuanyuanaosheng commented on Jun 29, 2023

    @xuanyuanaosheng
    xuanyuanaosheng
    on Jun 29, 2023

    @weizhouapache @DaanHoogland @cc20110101 @mail2lawi @kiwiflyer

    I did the test as your guide, But cloudstack new error is

    2023-06-29 16:29:48,085 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CALL `cloud`.`IDEMPOTENT_ADD_COLUMN`('cloud.volumes', 'passphrase_id', 'bigint unsigned DEFAULT NULL COMMENT "encryption passphrase id" ') 
2023-06-29 16:29:48,088 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CALL `cloud`.`IDEMPOTENT_ADD_FOREIGN_KEY`('cloud.volumes', 'passphrase', 'id') 
2023-06-29 16:29:48,249 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) Error executing: CALL `cloud`.`IDEMPOTENT_ADD_FOREIGN_KEY`('cloud.volumes', 'passphrase', 'id') 
2023-06-29 16:29:48,250 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Incorrect datetime value: '0014-10-00 00:00:00' for column 'created' at row 1
2023-06-29 16:29:48,253 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to execute upgrade script
com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Incorrect datetime value: '0014-10-00 00:00:00' for column 'created' at row 1
	at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:185)
	at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:87)
	at com.cloud.upgrade.DatabaseUpgradeChecker.runScript(DatabaseUpgradeChecker.java:226)
	at com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:310)
	at com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:401)
	at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:64)
	at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:54)
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178)

    I don't know why the 'volumes' table column 'created' is always 0014-10-00 00:00:00
    image

    how to fix this issue?

  28. weizhouapache commented on Jun 29, 2023

    @weizhouapache
    weizhouapache
    on Jun 29, 2023
    Member

    @xuanyuanaosheng
    it seems your database is broken. many fields in the volumes table have incorrect values.
    do you have any backup ?

  29. xuanyuanaosheng commented on Jun 29, 2023

    @xuanyuanaosheng
    xuanyuanaosheng
    on Jun 29, 2023

    @xuanyuanaosheng it seems your database is broken. many fields in the volumes table have incorrect values. do you have any backup ?

    @weizhouapache @DaanHoogland @cc20110101 @mail2lawi @kiwiflyer

    I think it is because the database upgrade problem.

    Yes, I suddenly remembered to recover data from the backup database file, it is under testing.

    As checked, It is indeed because a database upgrade problem. I recover data from the backup database file and upgrate to CloudStack cloudstack-management 4.18.0.0 successfully.

    # rpm -qa | grep cloudstack
cloudstack-management-4.18.0.0-1.el7.x86_64
cloudstack-common-4.18.0.0-1.x86_64

    But when I upgrade the cloudstack-agent, the connection between the management and agent node is failed.

    The cloudstack agent status

    # systemctl status cloudstack-agent
● cloudstack-agent.service - CloudStack Agent
   Loaded: loaded (/usr/lib/systemd/system/cloudstack-agent.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Fri 2023-06-30 12:16:48 CST; 3s ago
     Docs: http://www.cloudstack.org/
  Process: 113054 ExecStart=/usr/bin/java $JAVA_OPTS $JAVA_DEBUG -cp $CLASSPATH $JAVA_CLASS (code=exited, status=1/FAILURE)
 Main PID: 113054 (code=exited, status=1/FAILURE)

    and the error log is

    

2023-06-30 12:17:05,780 INFO  [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Process agent startup answer, agent id = 0
2023-06-30 12:17:05,780 INFO  [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Set agent id 0
2023-06-30 12:17:05,784 INFO  [cloud.agent.Agent] (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
2023-06-30 12:17:05,790 INFO  [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Startup Response Received: agent id = 0
2023-06-30 12:17:17,418 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-06-30 12:17:17,421 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-06-30 12:17:17,423 INFO  [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-06-30 12:17:17,454 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage
2023-06-30 12:17:17,454 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm
2023-06-30 12:17:17,483 INFO  [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2023-06-30 12:17:17,483 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Using default Java settings for IPv6 preference for agent connection
2023-06-30 12:17:17,547 INFO  [cloud.agent.Agent] (main:null) (logid:) id is 0
2023-06-30 12:17:17,553 INFO  [kvm.resource.LibvirtComputingResource] (main:null) (logid:) uefi.properties file found at /etc/cloudstack/agent/uefi.properties
2023-06-30 12:17:17,553 INFO  [kvm.resource.LibvirtComputingResource] (main:null) (logid:) guest.nvram.template.legacy = /usr/share/edk2/ovmf/OVMF_VARS.fd
2023-06-30 12:17:17,553 INFO  [kvm.resource.LibvirtComputingResource] (main:null) (logid:) guest.loader.legacy = /usr/share/edk2/ovmf/OVMF_CODE.secboot.fd
2023-06-30 12:17:17,553 INFO  [kvm.resource.LibvirtComputingResource] (main:null) (logid:) guest.nvram.template.secure = /usr/share/edk2/ovmf/OVMF_VARS.secboot.fd
2023-06-30 12:17:17,553 INFO  [kvm.resource.LibvirtComputingResource] (main:null) (logid:) guest.loader.secure =/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd
2023-06-30 12:17:17,553 INFO  [kvm.resource.LibvirtComputingResource] (main:null) (logid:) guest.nvram.path = /var/lib/libvirt/qumu/nvram/
2023-06-30 12:17:17,587 INFO  [kvm.resource.LibvirtComputingResource] (main:null) (logid:) Failed to find passphrase for keystore: cloud.jks
2023-06-30 12:17:17,590 INFO  [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one
2023-06-30 12:17:17,785 INFO  [kvm.resource.LibvirtComputingResource] (main:null) (logid:) IO uring driver for Qemu: disabled
2023-06-30 12:17:17,849 WARN  [kvm.storage.KVMStoragePoolManager] (main:null) (logid:) Duplicate StorageAdaptor type PowerFlex, not loading com.cloud.hypervisor.kvm.storage.ScaleIOStorageAdaptor
2023-06-30 12:17:17,850 INFO  [kvm.resource.LibvirtComputingResource] (main:null) (logid:) No libvirt.vif.driver specified. Defaults to BridgeVifDriver.
2023-06-30 12:17:18,072 INFO  [cloud.serializer.GsonHelper] (main:null) (logid:) Default Builder inited.
2023-06-30 12:17:18,073 INFO  [kvm.resource.LibvirtComputingResource] (main:null) (logid:) iscsi session clean up is disabled
2023-06-30 12:17:18,075 INFO  [kvm.resource.LibvirtComputingResource] (main:null) (logid:) Skipping the memory balloon stats period setting, since there are no VMs (active Libvirt domains) on this host.
2023-06-30 12:17:18,075 INFO  [kvm.resource.LibvirtComputingResource] (main:null) (logid:) The [vm.memballoon.stats.period] property is set to '0', this prevents memory statistics from being displayed correctly. Adjust (increase) the value of this parameter to correct this.
2023-06-30 12:17:18,083 INFO  [cloud.agent.Agent] (main:null) (logid:) Agent [id = 0 : type = LibvirtComputingResource : zone = 11 : pod = 10 : workers = 5 : host = 10.25.2.173 : port = 8250
2023-06-30 12:17:18,089 INFO  [utils.nio.NioClient] (main:null) (logid:) Connecting to 10.25.2.173:8250
2023-06-30 12:17:18,150 INFO  [utils.nio.Link] (main:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2023-06-30 12:17:18,486 INFO  [utils.nio.NioClient] (main:null) (logid:) SSL: Handshake done
2023-06-30 12:17:18,486 INFO  [utils.nio.NioClient] (main:null) (logid:) Connected to 10.25.2.173:8250
2023-06-30 12:17:18,517 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:null) (logid:) Fetching CPU speed from command "lscpu".
2023-06-30 12:17:18,546 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:null) (logid:) Command [lscpu | grep -i 'Model name' | head -n 1 | egrep -o '[[:digit:]].[[:digit:]]+GHz' | sed 's/GHz//g'] resulted in the value [3200] for CPU speed.
2023-06-30 12:17:18,654 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion
2023-06-30 12:17:18,677 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool 572e5d2c-32a0-4223-915c-502fe565eb8d (Filesystem) in libvirt
2023-06-30 12:17:18,679 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Found existing defined storage pool 572e5d2c-32a0-4223-915c-502fe565eb8d, using it.
2023-06-30 12:17:18,680 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Trying to fetch storage pool 572e5d2c-32a0-4223-915c-502fe565eb8d from libvirt
2023-06-30 12:17:18,860 INFO  [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Process agent startup answer, agent id = 0
2023-06-30 12:17:18,861 INFO  [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Set agent id 0
2023-06-30 12:17:18,869 INFO  [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Startup Response Received: agent id = 0
2023-06-30 12:17:18,879 INFO  [cloud.agent.Agent] (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill

    2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Looking for keystore-setup in /usr/share/cloudstack-agent/lib/scripts/keystore-setup
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Looking for keystore-setup in /usr/share/cloudstack-agent/lib/keystore-setup
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Looking for keystore-setup in /usr/share/cloudstack-agent/keystore-setup
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Looking for keystore-setup in /usr/share/keystore-setup
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Looking for keystore-setup in /usr/keystore-setup
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Looking for keystore-setup in /keystore-setup
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Searching in environment.properties
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Looking for keystore-setup in /usr/share/cloudstack-common/scripts/util/keystore-setup
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Looking for keystore-cert-import in the classpath
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) System resource: null
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Classpath resource: null
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Looking for keystore-cert-import
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Current binaries reside at /usr/share/cloudstack-agent/lib
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Looking for keystore-cert-import in /usr/share/cloudstack-agent/lib/scripts/util/keystore-cert-import
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Looking for keystore-cert-import in /usr/share/cloudstack-agent/lib/scripts/keystore-cert-import
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Looking for keystore-cert-import in /usr/share/cloudstack-agent/lib/keystore-cert-import
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Looking for keystore-cert-import in /usr/share/cloudstack-agent/keystore-cert-import
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Looking for keystore-cert-import in /usr/share/keystore-cert-import
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Looking for keystore-cert-import in /usr/keystore-cert-import
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Looking for keystore-cert-import in /keystore-cert-import
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Searching in environment.properties
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common
2023-06-30 15:07:48,579 DEBUG [utils.script.Script] (main:null) (logid:) Looking for keystore-cert-import in /usr/share/cloudstack-common/scripts/util/keystore-cert-import
2023-06-30 15:07:48,584 INFO  [utils.nio.NioClient] (main:null) (logid:) Connecting to 10.25.2.173:8250
2023-06-30 15:07:48,608 INFO  [utils.nio.Link] (main:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2023-06-30 15:07:48,959 INFO  [utils.nio.NioClient] (main:null) (logid:) SSL: Handshake done
2023-06-30 15:07:48,959 INFO  [utils.nio.NioClient] (main:null) (logid:) Connected to 10.25.2.173:8250
2023-06-30 15:07:48,965 DEBUG [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Looking for libvirtd connection at: qemu:///system
2023-06-30 15:07:48,977 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:null) (logid:) Fetching CPU speed from command "lscpu".
2023-06-30 15:07:48,978 DEBUG [utils.script.Script] (Agent-Handler-1:null) (logid:) Executing: /bin/bash -c lscpu | grep -i 'Model name' | head -n 1 | egrep -o '[[:digit:]].[[:digit:]]+GHz' | sed 's/GHz//g' 
2023-06-30 15:07:48,979 DEBUG [utils.script.Script] (Agent-Handler-1:null) (logid:) Executing while with timeout : 3600000
2023-06-30 15:07:49,003 DEBUG [utils.script.Script] (Agent-Handler-1:null) (logid:) Execution is successful.
2023-06-30 15:07:49,005 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:null) (logid:) Command [lscpu | grep -i 'Model name' | head -n 1 | egrep -o '[[:digit:]].[[:digit:]]+GHz' | sed 's/GHz//g'] resulted in the value [3200] for CPU speed.
2023-06-30 15:07:49,018 DEBUG [utils.net.NetUtils] (Agent-Handler-1:null) (logid:) Retrieving network params of NIC [name:cloudbr0 (cloudbr0)].
2023-06-30 15:07:49,019 DEBUG [utils.net.NetUtils] (Agent-Handler-1:null) (logid:) Retrieving params of address [/10.67.128.5/24 [/10.67.128.255]] of NIC [name:cloudbr0 (cloudbr0)].
2023-06-30 15:07:49,023 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:null) (logid:) Parameters for private nic: 10.67.128.5 - dc:f4:01:dd:a9:d0-255.255.255.0
2023-06-30 15:07:49,023 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:null) (logid:) Parameters for storage nic: 10.67.128.5 - dc:f4:01:dd:a9:d0-255.255.255.0
2023-06-30 15:07:49,023 DEBUG [utils.net.NetUtils] (Agent-Handler-1:null) (logid:) Retrieving network params of NIC [name:cloudbr0 (cloudbr0)].
2023-06-30 15:07:49,024 DEBUG [utils.net.NetUtils] (Agent-Handler-1:null) (logid:) Retrieving params of address [/10.67.128.5/24 [/10.67.128.255]] of NIC [name:cloudbr0 (cloudbr0)].
2023-06-30 15:07:49,024 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:null) (logid:) Parameters for public nic: 10.67.128.5 - dc:f4:01:dd:a9:d0-255.255.255.0
2023-06-30 15:07:49,024 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) (logid:) Executing: /usr/share/cloudstack-common/scripts/vm/hypervisor/versions.sh 
2023-06-30 15:07:49,026 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) (logid:) Executing while with timeout : 1800000
2023-06-30 15:07:49,069 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) (logid:) Execution is successful.
2023-06-30 15:07:49,082 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) (logid:) Executing: sudo grep InitiatorName= /etc/iscsi/initiatorname.iscsi 
2023-06-30 15:07:49,084 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) (logid:) Executing while with timeout : 3600000
2023-06-30 15:07:49,116 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) (logid:) Execution is successful.
2023-06-30 15:07:49,119 DEBUG [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Looking for libvirtd connection at: qemu:///system
2023-06-30 15:07:49,120 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion
2023-06-30 15:07:49,120 DEBUG [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Opening a new libvirtd connection to: qemu:///system
2023-06-30 15:07:49,123 DEBUG [utils.script.Script] (Agent-Handler-1:null) (logid:) Executing: qemu-img --help 
2023-06-30 15:07:49,125 DEBUG [utils.script.Script] (Agent-Handler-1:null) (logid:) Executing while with timeout : 3600000
2023-06-30 15:07:49,132 DEBUG [utils.script.Script] (Agent-Handler-1:null) (logid:) Execution is successful.
2023-06-30 15:07:49,139 DEBUG [utils.script.Script] (Agent-Handler-1:null) (logid:) Executing: cryptsetup --usage 
2023-06-30 15:07:49,140 DEBUG [utils.script.Script] (Agent-Handler-1:null) (logid:) Executing while with timeout : 3600000
2023-06-30 15:07:49,144 DEBUG [utils.script.Script] (Agent-Handler-1:null) (logid:) Execution is successful.
2023-06-30 15:07:49,144 DEBUG [utils.script.Script] (Agent-Handler-1:null) (logid:) Usage: cryptsetup [-?Vvyrq] [-?|--help] [--usage] [-V|--version]
        [-v|--verbose] [--debug] [--debug-json] [-c|--cipher=STRING]
        [-h|--hash=STRING] [-y|--verify-passphrase] [-d|--key-file=STRING]
        [--master-key-file=STRING] [--dump-master-key] [-s|--key-size=BITS]
        [-l|--keyfile-size=bytes] [--keyfile-offset=bytes]
        [--new-keyfile-size=bytes] [--new-keyfile-offset=bytes]
        [-S|--key-slot=INT] [-b|--size=SECTORS] [--device-size=bytes]
        [-o|--offset=SECTORS] [-p|--skip=SECTORS] [-r|--readonly]
        [-q|--batch-mode] [-t|--timeout=secs] [--progress-frequency=secs]
        [-T|--tries=INT] [--align-payload=SECTORS]
        [--header-backup-file=STRING] [--use-random] [--use-urandom]
        [--shared] [--uuid=STRING] [--allow-discards] [--header=STRING]
        [--test-passphrase] [--tcrypt-hidden] [--tcrypt-system]
        [--tcrypt-backup] [--veracrypt] [--veracrypt-pim=INT]
        [--veracrypt-query-pim] [-M|--type=STRING] [--force-password]
        [--perf-same_cpu_crypt] [--perf-submit_from_crypt_cpus]
        [--perf-no_read_workqueue] [--perf-no_write_workqueue] [--deferred]
        [--serialize-memory-hard-pbkdf] [-i|--iter-time=msecs]
        [--pbkdf=STRING] [--pbkdf-memory=kilobytes]
        [--pbkdf-parallel=threads] [--pbkdf-force-iterations=LONG]
        [--priority=STRING] [--disable-locks] [--disable-keyring]
        [-I|--integrity=STRING] [--integrity-no-journal]
        [--integrity-no-wipe] [--integrity-legacy-padding] [--token-only]
        [--token-id=INT] [--key-description=STRING] [--sector-size=INT]
        [--iv-large-sectors] [--persistent] [--label=STRING]
        [--subsystem=STRING] [--unbound] [--json-file=STRING]
        [--luks2-metadata-size=bytes] [--luks2-keyslots-size=bytes]
        [--refresh] [--keyslot-key-size=BITS] [--keyslot-cipher=STRING]
        [--encrypt] [--decrypt] [--init-only] [--resume-only]
        [--reduce-device-size=bytes] [--hotzone-size=bytes]
        [--resilience=STRING] [--resilience-hash=STRING]
        [--active-name=STRING] [OPTION...] <action> <action-specific>

2023-06-30 15:07:49,148 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool 572e5d2c-32a0-4223-915c-502fe565eb8d (Filesystem) in libvirt
2023-06-30 15:07:49,149 DEBUG [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Looking for libvirtd connection at: qemu:///system
2023-06-30 15:07:49,151 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Found existing defined storage pool 572e5d2c-32a0-4223-915c-502fe565eb8d, using it.
2023-06-30 15:07:49,152 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Trying to fetch storage pool 572e5d2c-32a0-4223-915c-502fe565eb8d from libvirt
2023-06-30 15:07:49,152 DEBUG [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Looking for libvirtd connection at: qemu:///system
2023-06-30 15:07:49,171 DEBUG [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Successfully refreshed pool 572e5d2c-32a0-4223-915c-502fe565eb8d Capacity: (49.98 GB) 53660876800 Used: (4.33 GB) 4650647552 Available: (45.64 GB) 49010229248
2023-06-30 15:07:49,218 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) (logid:) Executing: hostname 
2023-06-30 15:07:49,221 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) (logid:) Executing while with timeout : 500
2023-06-30 15:07:49,221 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) (logid:) Execution is successful.
2023-06-30 15:07:49,223 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) (logid:) Executing: hostname 
2023-06-30 15:07:49,224 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) (logid:) Executing while with timeout : 500
2023-06-30 15:07:49,226 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) (logid:) Execution is successful.

    I don't know why this happen, It works fine when I upgrade cloudstack from 4.17.1 to 4.17.2.

    I checked the #3509 , the agent env is

    # rpm -qa | grep libvirt
libvirt-daemon-driver-storage-scsi-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-daemon-driver-nodedev-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-daemon-driver-storage-gluster-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-daemon-config-network-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-daemon-driver-storage-logical-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-docs-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-daemon-driver-storage-iscsi-direct-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-daemon-driver-nwfilter-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-gobject-3.0.0-1.el8.x86_64
libvirt-lock-sanlock-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
pcp-pmda-libvirt-5.3.7-16.0.2.el8.x86_64
libvirt-daemon-driver-storage-core-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-daemon-driver-storage-rbd-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-daemon-driver-qemu-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-dbus-1.3.0-2.module+el8.8.0+20990+60c1530a.x86_64
libvirt-daemon-config-nwfilter-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-daemon-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-daemon-driver-storage-mpath-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-gconfig-3.0.0-1.el8.x86_64
libvirt-nss-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
python3-libvirt-8.0.0-2.module+el8.8.0+20990+60c1530a.x86_64
libvirt-daemon-driver-storage-disk-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-daemon-driver-storage-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-libs-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-daemon-driver-network-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-daemon-kvm-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-devel-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-daemon-driver-storage-iscsi-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-glib-3.0.0-1.el8.x86_64
libvirt-wireshark-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-daemon-driver-interface-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-client-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
libvirt-daemon-driver-secret-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64


# rpm -qa | grep qemu
qemu-kvm-block-curl-6.2.0-32.module+el8.8.0+21044+01700444.x86_64
qemu-kvm-block-ssh-6.2.0-32.module+el8.8.0+21044+01700444.x86_64
ipxe-roms-qemu-20181214-11.git133f4c47.el8.noarch
qemu-kvm-ui-spice-6.2.0-32.module+el8.8.0+21044+01700444.x86_64
qemu-kvm-ui-opengl-6.2.0-32.module+el8.8.0+21044+01700444.x86_64
qemu-kvm-block-iscsi-6.2.0-32.module+el8.8.0+21044+01700444.x86_64
libvirt-daemon-driver-qemu-8.0.0-19.0.1.module+el8.8.0+21044+01700444.x86_64
qemu-kvm-block-rbd-6.2.0-32.module+el8.8.0+21044+01700444.x86_64
qemu-kvm-docs-6.2.0-32.module+el8.8.0+21044+01700444.x86_64
qemu-img-6.2.0-32.module+el8.8.0+21044+01700444.x86_64
qemu-guest-agent-6.2.0-32.module+el8.8.0+21044+01700444.x86_64
qemu-kvm-hw-usbredir-6.2.0-32.module+el8.8.0+21044+01700444.x86_64
qemu-kvm-block-gluster-6.2.0-32.module+el8.8.0+21044+01700444.x86_64
qemu-kvm-6.2.0-32.module+el8.8.0+21044+01700444.x86_64
qemu-kvm-common-6.2.0-32.module+el8.8.0+21044+01700444.x86_64
qemu-kvm-core-6.2.0-32.module+el8.8.0+21044+01700444.x86_64

# cat /etc/os-release 
NAME="Oracle Linux Server"
VERSION="8.8"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="8.8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Oracle Linux Server 8.8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:8:8:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://github.com/oracle/oracle-linux"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 8"
ORACLE_BUGZILLA_PRODUCT_VERSION=8.8
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=8.8

    I restart the agent node and system, It still can not work.

    The libvirtd status is ok when started but a few minitues it is

    # systemctl status libvirtd
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2023-06-30 12:42:47 CST; 2min 57s ago
     Docs: man:libvirtd(8)
           https://libvirt.org
 Main PID: 139796 (libvirtd)
    Tasks: 19 (limit: 32768)
   Memory: 20.7M
   CGroup: /system.slice/libvirtd.service
           └─139796 /usr/sbin/libvirtd --listen

Jun 30 12:42:59 ljzdckvm005.cn.prod libvirtd[139796]: End of file while reading data: Input/output error
Jun 30 12:43:12 ljzdckvm005.cn.prod libvirtd[139796]: End of file while reading data: Input/output error
Jun 30 12:43:12 ljzdckvm005.cn.prod libvirtd[139796]: End of file while reading data: Input/output error
Jun 30 12:43:51 ljzdckvm005.cn.prod libvirtd[139796]: End of file while reading data: Input/output error
Jun 30 12:44:04 ljzdckvm005.cn.prod libvirtd[139796]: End of file while reading data: Input/output error
Jun 30 12:44:17 ljzdckvm005.cn.prod libvirtd[139796]: End of file while reading data: Input/output error
Jun 30 12:44:17 ljzdckvm005.cn.prod libvirtd[139796]: End of file while reading data: Input/output error
Jun 30 12:44:30 ljzdckvm005.cn.prod libvirtd[139796]: End of file while reading data: Input/output error
Jun 30 12:44:56 ljzdckvm005.cn.prod libvirtd[139796]: End of file while reading data: Input/output error
Jun 30 12:45:22 ljzdckvm005.cn.prod libvirtd[139796]: End of file while reading data: Input/output error

    The cloudstack management error log is

    2023-06-30 11:02:10,603 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-18474:ctx-d508fe0e) (logid:fb1152f9) Dispatching resource state event CREATE_HO
ST_VO_FOR_CONNECTED to OvmDiscoverer
2023-06-30 11:02:10,603 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-18474:ctx-d508fe0e) (logid:fb1152f9) Dispatching resource state event CREATE_HO
ST_VO_FOR_CONNECTED to KvmServerDiscoverer
2023-06-30 11:02:10,606 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-18474:ctx-d508fe0e) (logid:fb1152f9) Failed to handle host connection: 
java.lang.IllegalArgumentException: Can't add host: 10.67.128.5 with hostOS: Oracle Linux Server into a cluster,in which there are Red hosts added
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.createHostVOForConnectedAgent(LibvirtServerDiscoverer.java:474)
        at com.cloud.resource.ResourceManagerImpl.dispatchToStateAdapters(ResourceManagerImpl.java:1997)
        at com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:2224)
        at com.cloud.resource.ResourceManagerImpl.createHostVOForConnectedAgent(ResourceManagerImpl.java:2556)
        at jdk.internal.reflect.GeneratedMethodAccessor96.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy200.createHostVOForConnectedAgent(Unknown Source)
        at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1121)
        at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1218)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-30 11:02:10,609 WARN  [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-18474:ctx-d508fe0e) (logid:fb1152f9) Unable to create attache for agent: Seq 0-0
:  { Cmd , MgmtId: -1, via: 0, Ver: v1, Flags: 1, [{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":"2","cpus":"32","speed":"3200","memory":"80898189
7216","dom0MinMemory":"1073741824","poolSync":"false","supportsClonedVolumes":"false","caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails"
:{"Host.OS.Kernel.Version":"5.4.17-2136.320.7.1.el8uek.x86_64","com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS.Version":"8.8","host.v
olume.encryption":"true","secured":"true","Host.OS":"Oracle Linux Server"},"hostTags":[],"groupDetails":{},"type":"Routing","dataCenter":"11","pod":"10","clust
er":"11","guid":"de321280-e263-39b9-927f-02627689485f-LibvirtComputingResource","name":"ljzdckvm005.cn.prod","id":"0","version":"4.18.0.0","iqn":"iqn.1988-12.c
om.oracle:ed86c2978caa","publicIpAddress":"10.67.128.5","publicNetmask":"255.255.255.0","publicMacAddress":"dc:f4:01:dd:a9:d0","privateIpAddress":"10.67.128.5"
,"privateMacAddress":"dc:f4:01:dd:a9:d0","privateNetmask":"255.255.255.0","storageIpAddress":"10.67.128.5","storageNetmask":"255.255.255.0","storageMacAddress"
:"dc:f4:01:dd:a9:d0","resourceName":"LibvirtComputingResource","gatewayIpAddress":"10.67.128.254","msHostList":"10.25.2.173@static","wait":"0","bypassHostMaint
enance":"false"}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":"(0 bytes) 0","poolInfo":{"uuid":"572e5d2c-32a0-4223-915c-502fe565eb8d","host":"10.
67.128.5","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":"(49.98 GB) 53660876800","availabl
eBytes":"(45.67 GB) 49040306176"},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"11","pod":"10","guid":"de321280-e263-39b9-927f-
02627689485f-LibvirtComputingResource","name":"ljzdckvm005.cn.prod","id":"0","version":"4.18.0.0","resourceName":"LibvirtComputingResource","msHostList":"10.25
.2.173@static","wait":"0","bypassHostMaintenance":"false"}}] }
2023-06-30 11:02:10,724 WARN  [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-10:null) (logid:) Throwing away a request because it came through as the first c
ommand on a connect: Seq 0--1:  { Cmd , MgmtId: -1, via: 0, Ver: v1, Flags: 111, [{"com.cloud.agent.api.ShutdownCommand":{"reason":"sig.kill","removeHost":"fal
se","wait":"0","bypassHostMaintenance":"false"}}] }

    I think it is a libvirtd and cloudstack connection problem,
    @weizhouapache @DaanHoogland @cc20110101 @mail2lawi @kiwiflyer cloud you help to handle this? I config the libvirtd using :https://www.sbarjatiya.com/notes_wiki/index.php/CentOS_8.x_Cloudstack_4.15_Setup_KVM_host

    The unupgrade agent node (4.17.2.0) is still work fine.

    The system message log is as below :
    messages.log

    The further test, I downgrade the cloudstack agent from 4.18.0.0 to 4.17.2.0 and the cloudstack-agent can connect the cloudstack-management.

  30. xuanyuanaosheng commented on Jul 6, 2023

    @xuanyuanaosheng
    xuanyuanaosheng
    on Jul 6, 2023

    Please take a look at when upgrade cloudstack: https://docs.cloudstack.apache.org/en/latest/upgrading/upgrade/mysql.html#

  31. DaanHoogland
    added this to the unplanned milestone on Sep 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions