Repository navigation
Missing logs when cloudstack-setup-agent is run with sudo #10703
Description
problem
Trying to setup a AlmaLinux 9.5 KVM host with Cloudstack 4.20.
So far, I followed the instructions in the wiki and created a cloudstack user with sudo permissions.
When running sudo -u cloudstack cloudstack-setup-agent, it did spit some strange error message
Please input the Hypervisor type kvm/lxc:[kvm]
DEBUG:root:execute:route -n|awk '/^0.0.0.0/ {print $2,$8}'
Failed to get default route. Please configure your network to have a default route
When running route -n|awk '/^0.0.0.0/ {print $2,$8}' alone it worked well.
So I started hacking into the python code, and added a raise statement.
This time, I got the following error message
PermissionError: [Errno 13] Permission denied: '/bin/sh'
This allowed me to debug and find out that I had Defaults noexec in my /etc/sudoers file set, even when running sudo I wasn't allowed to run /bin/sh.
The problem here is that running cloudstack-setup-agent with sudo doesn't create any error log in /var/log/cloudstack/agent/setup.log file, even if running sudo -u cloudstack echo "Test" >> /var/log/cloudstack/agent/setup.log works.
As a side note, the bash class should perhaps send those exception errors to stderr too
versions
The versions of ACS, hypervisors, storage, network etc..
The steps to reproduce the bug
...
What to do about it?
No response
Activity
boring-cyborg commentedon Apr 13, 2025
Thanks for opening your first issue here! Be sure to follow the issue template!
weizhouapache commentedon Apr 14, 2025
@deajan
I think the command should be like
sudo cloudstack-setup-agentdeajan commentedon Apr 14, 2025
For the sake of sanity, I re-ran the command as you suggested
This indeed creates the following log:
But there's still no trace of the reason why `route -n|awk '/^0.0.0.0/ {print $2,$8}' failed with a permission error from the cloudstack-setup-agent, whereas running the same from commandline as non root works, even without sudo.
So I still think that the raised exception is never written to the logfile.
weizhouapache commentedon Apr 14, 2025
the permission error does not raise, right ?
anyway, in my opinion, the general way of adding host to cloudstack is
deajan commentedon Apr 14, 2025
Yes, permission error does not raise, unless I "make it raise" by modding the source script.
Reading the sources, the bash Class should catch the traceback and print/log the corresponding error, but it doesn't.
I would love to add the host via the UI, but no:
My machine is a fresh AlmaLinux 9.5 with a SCAP profile enabled, so I can understand that the SCAP profile configs may break things, and I am willing to get my hands dirty configuring the right stuff.
But without even getting logs, it's hard to do so.
Other missing logs example:
For the above screenshot error, I tried to run manually
/usr/share/cloudstack-common/scripts/util/keystore-setup(obviously with the wrong arguments) which fails with:Looking into that script, I see that every ran command is redirected with
>/dev/null 2>&1instead of a log file, so things get hard to debug when trying to add the host from the UI.weizhouapache commentedon Apr 14, 2025
from my experience, these messages are misleading
have you already configured the bridges ?
deajan commentedon Apr 14, 2025
Yes, they are configured via NetworkManager, but they have other names (I read that cloudbr0 and cloudbr1 are not mandatory names, so I decided to go with names I can work with)
Btw, running
cloudstack-setup-agentmanually works (but the host doesn't appear on the management server):[EDIT]cloudstack-agent shows that server certificate isn't good, which isn't surprising since I used an internal dns name[/EDIT]
weizhouapache commentedon Apr 14, 2025
Go to zone->physical networks, update the kvm network traffic label of the physical networks to br_xxx0
deajan commentedon Apr 15, 2025
Hmmm... This was already set right a I used the wizard initially to configure Cloudstack and try to add the host.
weizhouapache commentedon Apr 15, 2025
can you
deajan commentedon Apr 15, 2025
I double checked that the labels are set correctly. All the bridges are up and connected to ethernet interfaces.
When I add the host on UI, I get the same error message as in the screenshot above (530: Failed to setup keystore on the KVM host).
On the KVM host, journalctl -r shows the following entries (no errors):
On the management server,
/var/log/cloudstack/management/management-server.logrelevant outputOf course, keytool is installed.
I decided to manually run the given keystore setup command on the KVM host:
So running the keystore script looks good.
Even after this step, trying to add the KVM host via the UI failed with the same 530 error.
weizhouapache commentedon Apr 15, 2025
@deajan
can you search the management-server.log by keyword
cloudstack-setup-agent?there should be some logs like
deajan commentedon Apr 15, 2025
grep -ri "cloudstack\-setup" /var/log/cloudstackdidn't produce any results on the management server, nor the KVM host.I did some more tests.
I found that my SCAP profile sets the following values, preventing sudo runs to succeed.
I commented all those out in order for the script run to succed. Perhaps this can be added to the KVM wiki. I didn't expect root to run the command with
sudo.With those variables setup, adding the host via UI still failed.
I modified the sources of
/usr/share/cloudstack-common/scripts/util/keystore-setup, changing all redirections to /dev/null to a log file.I found the following results in my log file:
Investigating further, I noticed that commented passwords are still used in the regex of the keystore-setup script.
I've improved the script to add logs and restricted the regex.
This time, I could add my host.
I've tested my script for another host.
How about I make a PR for that one ? and Perhaps the wiki entry for adding checks on sudoers file ?
weizhouapache commentedon Apr 15, 2025
@deajan
You can create a doc PR for sudoers file if you think it is needed.
current doc can be found at https://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/kvm.html#install-and-configure-the-agent
deajan commentedon Apr 15, 2025
Mind pointing me into the right direction to make the PR ? There are like 2.8k repositories on apache github :)
3 remaining items
flefsas commentedon Jul 25, 2025
I do not know if this is relevant in any sort to this issue - but just some information on my side.
I encountered the "could not setup keystore" error message while trying to setup my first zone in a fresh installation on Ubuntu Server 24.04 LTS with a sudo-privileged user and specifically following the latest KVM host installation guide (https://docs.cloudstack.apache.org/en/latest/installguide/configuration.html#overview-of-provisioning-steps). By the way with the same issue that protocols are very sparse when done this way.
Found out that for me this hasn't had anything to do with the keystore at all, but rather the abstinence of ifconfig (installable via the net-tools package) and the need for native root permissions for the cloudstack agent setup script. Could not get it to run anyhow else than to temporary unlock the root account, allowing root SSH login and installing the net-tools package.
This - in this form at least - is not documented in https://docs.cloudstack.apache.org/en/latest/installguide/configuration.html#overview-of-provisioning-steps - I only found out about this after tracing the cloudstack agent setup script myself.
Tracing:
DEBUG:root:execute:ip route show default | awk '{print $3,$5}'
DEBUG:root:execute:ifconfig cloudbr0
DEBUG:root:Failed to execute:/bin/sh: 1: ifconfig: not found
DEBUG:root:Failed to get address from ifconfig
Failed to get default route. Please configure your network to have a default route
DEBUG:root:execute:ip route show default | awk '{print $3,$5}'
DEBUG:root:execute:ifconfig cloudbr0
DEBUG:root:Found default network device:cloudbr0
Please choose which network used to create VM:[cloudbr0]
DEBUG:root:execute:uuidgen
Starting to configure your system:
DEBUG:root:execute:uname -r
DEBUG:root:execute:uname -m
Need to execute with root permission
Try to restore your system:
...and so on.
sureshanaparti commentedon Aug 29, 2025
Fixed in #10723