Backup and Restore

NeMo Microservices use several storage systems that require backup consideration:

1. PostgreSQL database: stores entity metadata and relationships

2. Object storage: stores model files, datasets, and other artifacts

3. Persistent Volume Claims (PVC): stores Kubernetes persistent data

PostgreSQL Backup

For PostgreSQL backup and restore procedures, refer to the official PostgreSQL documentation:

• PostgreSQL Backup and Restore

• Continuous Archiving and Point-in-Time Recovery (PITR)

Object Storage Backup

Backup procedures for object storage depend on your chosen storage solution:

• Amazon S3: See AWS Backup documentation

• MinIO: See MinIO Data Recovery Guide

• Azure Blob Storage: See Azure Backup documentation

Kubernetes PVC Backup

For backing up Persistent Volume Claims in Kubernetes:

• Velero: See Velero documentation

• Native solutions: Refer to your cloud provider’s Kubernetes backup solutions:

  • GKE Backup

  • AKS Backup

  • EKS Backup

Best Practices

1. Schedule regular backups during low-usage periods

2. Test your restore procedures periodically

3. Store backups in a different physical location

4. Document your backup and restore procedures

5. Monitor backup job status and storage usage

6. Ensure consistent state across all storage systems when performing backups

For detailed implementation guidance, consult your storage provider’s documentation and your organization’s disaster recovery policies.