Repository navigation
Bug: agent, vm cleanup - iptables/ipset chain existence not checked before deletion, causing misleading error logs #12770
Description
Title
Cleanup: iptables/ipset chain existence not checked before deletion, causing misleading error logs
ISSUE TYPE
- Bug Report
COMPONENT NAME
scripts/vm/network/security_group.py
CLOUDSTACK VERSION
main branch (4.22)
SUMMARY
When VM cleanup runs, the destroy_network_rules_for_vm() function attempts to flush and delete iptables chains and ipsets without first checking if they exist. The execute() function logs CalledProcessError at ERROR level with full traceback before re-raising, causing confusing duplicate log entries when chains don't exist. Exit code 1 from iptables/ipset is normal when a chain/set doesn't exist (idempotent cleanup), not a real error.
STEPS TO REPRODUCE
- Create a VM with security groups in CloudStack
- Delete the VM
- Observe management server logs for the VM cleanup output
EXPECTED RESULTS
- Cleanup should be idempotent - safe to run multiple times
- "Chain doesn't exist" should be logged at DEBUG level
- Only real errors should be logged at ERROR level
- No duplicate logging of the same issue
ACTUAL RESULTS
2026-03-09 09:46:52,842 ERROR [c.c.v.s.security_group] (Thread-123) Command exited non-zero: iptables -X i-6-526-VM-eg
Traceback (most recent call last):
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", line 53, in execute
return check_output(cmd, shell=True).decode()
File "/usr/lib/python3.13/subprocess.py", line 472, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, ...)
File "/usr/lib/python3.13/subprocess.py", line 577, in run
raise CalledProcessError(retcode, process.args, output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'iptables -X i-6-526-VM-eg' returned non-zero exit status 1.
2026-03-09 09:46:52,842 DEBUG Ignoring failure to delete chain: i-6-526-VM-eg
The ERROR level log with full traceback is misleading - this is expected behavior when chains don't exist.
PROPOSED FIX
The fix should check if chains exist before attempting deletion. When iptables -S chain returns exit code 1, the chain doesn't exist - this is normal, not a real error. Only real errors (other exit codes) should be logged.
Add helper functions:
def iptables_chain_exists(chain):
"""Check if iptables chain exists"""
try:
execute("iptables -S %s 2>/dev/null" % chain)
return True
except CalledProcessError as e:
if e.returncode == 1:
return False # Chain not found - not an error
raise # Real error (other exit code)
def ipset_exists(setname):
"""Check if ipset exists"""
try:
execute("ipset list %s 2>/dev/null" % setname)
return True
except CalledProcessError as e:
if e.returncode == 1:
return False # Set not found - not an error
raise # Real error (other exit code)Update destroy_network_rules_for_vm() to check existence before deletion:
for chain in [_f for _f in chains if _f]:
if iptables_chain_exists(chain):
try:
execute("iptables -F " + chain)
execute("iptables -X " + chain)
except Exception as e:
logging.error("Failed to flush/delete chain %s: %s", chain, str(e))
else:
logging.debug("Chain %s does not exist, skipping", chain)
for ipset in [vm_ipsetname, vm_ipsetname + '-6']:
if ipset_exists(ipset):
try:
execute('ipset -F ' + ipset)
execute('ipset -X ' + ipset)
except Exception as e:
logging.error("Failed to flush/delete ipset %s: %s", ipset, str(e))
else:
logging.debug("Ipset %s does not exist, skipping", ipset)This approach:
- Makes cleanup idempotent (safe to run multiple times)
- Provides clear logging: "chain doesn't exist" at DEBUG, real errors at ERROR
- Distinguishes between "not found" (exit code 1, expected) and real errors (other exit codes)
- No duplicate logging
Activity
dheeraj12347 commentedon Mar 11, 2026
Hi @DaanHoogland @rybakovanton-metta,
I reviewed the issue and it seems the cleanup logic in
security_group.pyattempts to flush/delete iptables chains and ipsets without verifying whether they exist first. When they don't exist,iptables/ipsetreturn exit code 1, which is expected for idempotent cleanup but currently gets logged as an ERROR with a traceback.I’d like to work on this by adding checks to verify the chain/ipset existence before deletion and adjusting the logging so that non-existent chains are handled gracefully (e.g., logged at DEBUG instead of ERROR).
Please let me know if it’s okay for me to take this up.
github-actions commentedon Mar 11, 2026
🏷️ Issue Triaged
Hi
@rybakovanton-metta! I've categorized this issue as bug based on the following analysis:Reasoning: This reports a specific defect where VM cleanup causes misleading ERROR-level logs when iptables/ipset chains don't exist. The issue includes error reproduction, traceback, and a detailed proposed fix, making it clearly a bug report.
View Triage Details
Analysis
Recommended Next Steps
References: Triage run §22957405999
dheeraj12347 commentedon Mar 15, 2026
Hi @DaanHoogland, @rybakovanton-metta,
I’ve implemented the proposed fix in scripts/vm/network/security_group.py.
Changes made:
Added helper functions iptables_chain_exists, ip6tables_chain_exists, and ipset_exists that call the existing execute() wrapper and treat exit code 1 as “does not exist” (logged at DEBUG), while re‑raising other exit codes as real errors.
Updated destroy_network_rules_for_vm() so that for each VM‑related chain (-def, , and the egress chain) it checks iptables_chain_exists / ip6tables_chain_exists before calling iptables/ip6tables -F/-X, and logs “chain does not exist, skipping” at DEBUG when appropriate.
For the VM ipsets (IPv4 and IPv6) it checks ipset_exists before ipset -F/-X, and logs “ipset does not exist, skipping” at DEBUG when they’re already gone.
This makes the VM cleanup idempotent and avoids misleading ERROR‑level tracebacks during normal cleanup, while still surfacing real failures.
I’ve run python -m py_compile scripts/vm/network/security_group.py locally to verify syntax.
Please let me know if this approach looks good or if you’d like any adjustments.
weizhouapache commentedon Mar 16, 2026
@dheeraj12347
Thanks for the investigation and suggestion - this looks good to me. Could you open a PR for this?
Fix vm cleanup iptables/ipset misleading error logs (Fixes apache#12770)
dheeraj12347 commentedon Mar 18, 2026
done :)
github-actions commentedon Aug 18, 2026
🎯 Triage report
Agent-side VM cleanup calls iptables/ipset delete/flush without first checking chain/set existence, so idempotent no-op cleanups log full
CalledProcessErrortracebacks at ERROR level instead of DEBUG, creating misleading noise in logs. A fix has already been implemented and a PR opened by a contributor, confirmed by the maintainer.📊 Assessment
scripts/vm/network/security_group.pyruns on the KVM agent host🔗 Similar issues
No similar open issues found via search.
💡 Notes and suggestions
A PR from dheeraj12347 implementing
iptables_chain_exists/ip6tables_chain_exists/ipset_existshelpers has already been opened and approved in concept by weizhouapache. Maintainers should track/merge that PR rather than treat this as open work.Add this agentic workflows to your repo
To install this agentic workflow, run
Fix vm cleanup iptables/ipset misleading error logs (Fixes apache#12770)