Log Rotation Failures

Runbook: Log Rotation Failures

Symptoms

Prometheus alert: LogRotationFailures or FileServerLowDiskSpace
Metric fileserver_log_rotation_failed_total increasing
Metric fileserver_disk_space_free_bytes decreasing to critical levels
Old log files not being deleted
File server pod disk full

Diagnosis Steps

1. Check Cleanup Service Status

$ # Get file server pod
$ FILE_SERVER_POD=$(kubectl get pods -n nvsentinel -l app.kubernetes.io/name=incluster-file-server -o jsonpath='\{.items[0].metadata.name\}')
$ 
$ # Check if cleanup container is running
$ kubectl get pod -n nvsentinel $FILE_SERVER_POD -o jsonpath='\{.spec.containers[*].name\}'
$ # Should include: nginx, log-cleanup, (optionally) nginx-log-exporter
$ 
$ # Check cleanup container logs
$ kubectl logs -n nvsentinel $FILE_SERVER_POD -c log-cleanup --tail=100

2. Check Disk Usage

$ # Check disk space on file server
$ kubectl exec -n nvsentinel $FILE_SERVER_POD -- df -h /usr/share/nginx/html
$ 
$ # Check PVC status
$ kubectl get pvc -n nvsentinel
$ kubectl describe pvc -n nvsentinel nvsentinel-incluster-file-server-data
$ 
$ # Count and size of log files
$ kubectl exec -n nvsentinel $FILE_SERVER_POD -- sh -c "find /usr/share/nginx/html -type f | wc -l"
$ kubectl exec -n nvsentinel $FILE_SERVER_POD -- du -sh /usr/share/nginx/html/*

3. Check Rotation Configuration

$ # Get current configuration
$ kubectl get cm -n nvsentinel file-server-log-cleanup-config -o yaml
$ 
$ # Verify cleanup container environment variables
$ kubectl get pod -n nvsentinel $FILE_SERVER_POD -o jsonpath='\{.spec.containers[?(@.name=="log-cleanup")].env[*]\}' | jq

4. Check Metrics

$ # Port-forward to cleanup metrics endpoint
$ kubectl port-forward -n nvsentinel $FILE_SERVER_POD 9002:9002
$ 
$ # Query cleanup metrics
$ curl http://localhost:9002/metrics | grep fileserver_log_rotation
$ curl http://localhost:9002/metrics | grep fileserver_disk_space

Common Failure Causes and Solutions

Issue 1: Cleanup Container Not Running

Error: Cleanup container is missing or crashed

Solution:

$ # Check pod events
$ kubectl describe pod -n nvsentinel $FILE_SERVER_POD
$ 
$ # Check if cleanup is enabled in Helm values
$ helm get values -n nvsentinel nvsentinel -o yaml | grep -A 5 logCleanup
$ 
$ # If disabled, enable it:
$ # In values.yaml:
$ inclusterFileServer:
$   logCleanup:
$     enabled: true
$     retentionDays: 7
$ 
$ # Upgrade Helm release
$ helm upgrade nvsentinel -n nvsentinel ./distros/kubernetes/nvsentinel -f values.yaml

Issue 2: Permission Errors

Error in logs:

[ERROR] Cleanup operation failed: Permission denied
find: '/usr/share/nginx/html/upload': Permission denied

Cause: Incorrect file permissions or security context

Solution:

$ # Check file permissions
$ kubectl exec -n nvsentinel $FILE_SERVER_POD -- ls -la /usr/share/nginx/html
$ 
$ # Fix permissions if needed
$ kubectl exec -n nvsentinel $FILE_SERVER_POD -- chmod -R 775 /usr/share/nginx/html
$ 
$ # Verify init container ran successfully
$ kubectl logs -n nvsentinel $FILE_SERVER_POD -c init-perms

Issue 3: Timeout During Cleanup

Error in logs:

[ERROR] Cleanup operation timed out

Cause: Too many files to process within timeout period

Solution:

$ # Check number of files
$ kubectl exec -n nvsentinel $FILE_SERVER_POD -- find /usr/share/nginx/html -type f | wc -l
$ 
$ # Manually clean very old logs first
$ kubectl exec -n nvsentinel $FILE_SERVER_POD -- find /usr/share/nginx/html -type f -mtime +30 -delete
$ 
$ # Restart cleanup container
$ kubectl delete pod -n nvsentinel $FILE_SERVER_POD
$ # (Pod will be recreated by deployment)

Issue 4: Incorrect Retention Configuration

Error: Files not being deleted at expected age

Solution:

$ # Check current retention setting
$ kubectl get cm -n nvsentinel file-server-log-cleanup-config -o yaml | grep LOG_RETENTION_DAYS
$ 
$ # Update retention period
$ # In Helm values:
$ inclusterFileServer:
$   logCleanup:
$     retentionDays: 3  # Reduce from 7 to 3 days
$ 
$ # Apply changes
$ helm upgrade nvsentinel -n nvsentinel ./distros/kubernetes/nvsentinel -f values.yaml
$ 
$ # Force immediate cleanup cycle
$ kubectl exec -n nvsentinel $FILE_SERVER_POD -c log-cleanup -- pkill -HUP python3

Issue 5: Disk Full - Emergency Response

Critical: Disk is full and blocking new uploads

Immediate action:

$ # 1. Manually delete oldest logs
$ kubectl exec -n nvsentinel $FILE_SERVER_POD -- find /usr/share/nginx/html -type f -mtime +1 -delete
$ 
$ # 2. Delete logs for specific nodes that are no longer relevant
$ kubectl exec -n nvsentinel $FILE_SERVER_POD -- rm -rf /usr/share/nginx/html/&lt;old-node-name>
$ 
$ # 3. Check space recovered
$ kubectl exec -n nvsentinel $FILE_SERVER_POD -- df -h /usr/share/nginx/html
$ 
$ # 4. Increase PVC size (if storage class supports it)
$ kubectl patch pvc -n nvsentinel nvsentinel-incluster-file-server-data \
>   -p '\{"spec":\{"resources":\{"requests":\{"storage":"100Gi"\}\}\}\}'
$ 
$ # 5. Verify expansion
$ kubectl get pvc -n nvsentinel -w

Issue 6: Cleanup Running But Files Not Deleted

Error: Cleanup reports success but files remain

Solution:

$ # Test find command manually
$ kubectl exec -n nvsentinel $FILE_SERVER_POD -- find /usr/share/nginx/html -type f -mtime +7 -ls
$ 
$ # Check if files are actually older than retention period
$ kubectl exec -n nvsentinel $FILE_SERVER_POD -- find /usr/share/nginx/html -type f -printf '%T+ %p\n' | sort | head -20
$ 
$ # Force cleanup with debug
$ kubectl exec -n nvsentinel $FILE_SERVER_POD -c log-cleanup -- sh -c "
>   echo 'Running manual cleanup...'
>   find /usr/share/nginx/html -type f -mtime +7 -print -delete
> "

Resolution Steps

Immediate triage:

$ # Check if disk is critically low (&lt; 5GB)
$ DISK_FREE=$(kubectl exec -n nvsentinel $FILE_SERVER_POD -- df -BG /usr/share/nginx/html | tail -1 | awk '\{print $4\}' | sed 's/G//')
$ if [ "$DISK_FREE" -lt 5 ]; then
$   echo "CRITICAL: Immediate cleanup needed"
$   # Run emergency cleanup
$   kubectl exec -n nvsentinel $FILE_SERVER_POD -- find /usr/share/nginx/html -type f -mtime +1 -delete
$ fi

Fix root cause using solutions above

Verify cleanup is working:

$ # Monitor cleanup metrics
$ kubectl port-forward -n nvsentinel $FILE_SERVER_POD 9002:9002 &
$ watch -n 10 'curl -s http://localhost:9002/metrics | grep fileserver_log_rotation'
$ 
$ # Watch disk space
$ watch -n 30 'kubectl exec -n nvsentinel $FILE_SERVER_POD -- df -h /usr/share/nginx/html'

Test with a dry run:

$ # See what would be deleted
$ kubectl exec -n nvsentinel $FILE_SERVER_POD -- \
>   find /usr/share/nginx/html -type f -mtime +7 -ls

Force cleanup cycle:

$ # Trigger cleanup without waiting for next interval
$ kubectl exec -n nvsentinel $FILE_SERVER_POD -c log-cleanup -- pkill -HUP python3
$ 
$ # Watch logs
$ kubectl logs -n nvsentinel $FILE_SERVER_POD -c log-cleanup -f

$	# Get file server pod
$	FILE_SERVER_POD=$(kubectl get pods -n nvsentinel -l app.kubernetes.io/name=incluster-file-server -o jsonpath='\{.items[0].metadata.name\}')
$
$	# Check if cleanup container is running
$	kubectl get pod -n nvsentinel $FILE_SERVER_POD -o jsonpath='\{.spec.containers[*].name\}'
$	# Should include: nginx, log-cleanup, (optionally) nginx-log-exporter
$
$	# Check cleanup container logs
$	kubectl logs -n nvsentinel $FILE_SERVER_POD -c log-cleanup --tail=100

$	# Check disk space on file server
$	kubectl exec -n nvsentinel $FILE_SERVER_POD -- df -h /usr/share/nginx/html
$
$	# Check PVC status
$	kubectl get pvc -n nvsentinel
$	kubectl describe pvc -n nvsentinel nvsentinel-incluster-file-server-data
$
$	# Count and size of log files
$	kubectl exec -n nvsentinel $FILE_SERVER_POD -- sh -c "find /usr/share/nginx/html -type f \| wc -l"
$	kubectl exec -n nvsentinel $FILE_SERVER_POD -- du -sh /usr/share/nginx/html/*

$	# Get current configuration
$	kubectl get cm -n nvsentinel file-server-log-cleanup-config -o yaml
$
$	# Verify cleanup container environment variables
$	kubectl get pod -n nvsentinel $FILE_SERVER_POD -o jsonpath='\{.spec.containers[?(@.name=="log-cleanup")].env[*]\}' \| jq

$	# Port-forward to cleanup metrics endpoint
$	kubectl port-forward -n nvsentinel $FILE_SERVER_POD 9002:9002
$
$	# Query cleanup metrics
$	curl http://localhost:9002/metrics \| grep fileserver_log_rotation
$	curl http://localhost:9002/metrics \| grep fileserver_disk_space

$	# Check pod events
$	kubectl describe pod -n nvsentinel $FILE_SERVER_POD
$
$	# Check if cleanup is enabled in Helm values
$	helm get values -n nvsentinel nvsentinel -o yaml \| grep -A 5 logCleanup
$
$	# If disabled, enable it:
$	# In values.yaml:
$	inclusterFileServer:
$	logCleanup:
$	enabled: true
$	retentionDays: 7
$
$	# Upgrade Helm release
$	helm upgrade nvsentinel -n nvsentinel ./distros/kubernetes/nvsentinel -f values.yaml

$	# Check file permissions
$	kubectl exec -n nvsentinel $FILE_SERVER_POD -- ls -la /usr/share/nginx/html
$
$	# Fix permissions if needed
$	kubectl exec -n nvsentinel $FILE_SERVER_POD -- chmod -R 775 /usr/share/nginx/html
$
$	# Verify init container ran successfully
$	kubectl logs -n nvsentinel $FILE_SERVER_POD -c init-perms

$	# Check number of files
$	kubectl exec -n nvsentinel $FILE_SERVER_POD -- find /usr/share/nginx/html -type f \| wc -l
$
$	# Manually clean very old logs first
$	kubectl exec -n nvsentinel $FILE_SERVER_POD -- find /usr/share/nginx/html -type f -mtime +30 -delete
$
$	# Restart cleanup container
$	kubectl delete pod -n nvsentinel $FILE_SERVER_POD
$	# (Pod will be recreated by deployment)

$	# Check current retention setting
$	kubectl get cm -n nvsentinel file-server-log-cleanup-config -o yaml \| grep LOG_RETENTION_DAYS
$
$	# Update retention period
$	# In Helm values:
$	inclusterFileServer:
$	logCleanup:
$	retentionDays: 3 # Reduce from 7 to 3 days
$
$	# Apply changes
$	helm upgrade nvsentinel -n nvsentinel ./distros/kubernetes/nvsentinel -f values.yaml
$
$	# Force immediate cleanup cycle
$	kubectl exec -n nvsentinel $FILE_SERVER_POD -c log-cleanup -- pkill -HUP python3

$	# 1. Manually delete oldest logs
$	kubectl exec -n nvsentinel $FILE_SERVER_POD -- find /usr/share/nginx/html -type f -mtime +1 -delete
$
$	# 2. Delete logs for specific nodes that are no longer relevant
$	kubectl exec -n nvsentinel $FILE_SERVER_POD -- rm -rf /usr/share/nginx/html/<old-node-name>
$
$	# 3. Check space recovered
$	kubectl exec -n nvsentinel $FILE_SERVER_POD -- df -h /usr/share/nginx/html
$
$	# 4. Increase PVC size (if storage class supports it)
$	kubectl patch pvc -n nvsentinel nvsentinel-incluster-file-server-data \
>	-p '\{"spec":\{"resources":\{"requests":\{"storage":"100Gi"\}\}\}\}'
$
$	# 5. Verify expansion
$	kubectl get pvc -n nvsentinel -w

$	# Test find command manually
$	kubectl exec -n nvsentinel $FILE_SERVER_POD -- find /usr/share/nginx/html -type f -mtime +7 -ls
$
$	# Check if files are actually older than retention period
$	kubectl exec -n nvsentinel $FILE_SERVER_POD -- find /usr/share/nginx/html -type f -printf '%T+ %p\n' \| sort \| head -20
$
$	# Force cleanup with debug
$	kubectl exec -n nvsentinel $FILE_SERVER_POD -c log-cleanup -- sh -c "
>	echo 'Running manual cleanup...'
>	find /usr/share/nginx/html -type f -mtime +7 -print -delete
>	"

$	# Check if disk is critically low (< 5GB)
$	DISK_FREE=$(kubectl exec -n nvsentinel $FILE_SERVER_POD -- df -BG /usr/share/nginx/html \| tail -1 \| awk '\{print $4\}' \| sed 's/G//')
$	if [ "$DISK_FREE" -lt 5 ]; then
$	echo "CRITICAL: Immediate cleanup needed"
$	# Run emergency cleanup
$	kubectl exec -n nvsentinel $FILE_SERVER_POD -- find /usr/share/nginx/html -type f -mtime +1 -delete
$	fi

$	# Monitor cleanup metrics
$	kubectl port-forward -n nvsentinel $FILE_SERVER_POD 9002:9002 &
$	watch -n 10 'curl -s http://localhost:9002/metrics \| grep fileserver_log_rotation'
$
$	# Watch disk space
$	watch -n 30 'kubectl exec -n nvsentinel $FILE_SERVER_POD -- df -h /usr/share/nginx/html'

$	# See what would be deleted
$	kubectl exec -n nvsentinel $FILE_SERVER_POD -- \
>	find /usr/share/nginx/html -type f -mtime +7 -ls

$	# Trigger cleanup without waiting for next interval
$	kubectl exec -n nvsentinel $FILE_SERVER_POD -c log-cleanup -- pkill -HUP python3
$
$	# Watch logs
$	kubectl logs -n nvsentinel $FILE_SERVER_POD -c log-cleanup -f