Database Migrations

• Migration files are stored in
  sophie_bot/db/migrations/
• Each migration has a timestamp prefix:
  YYYYMMDD_HHMMSS_description.py
• Migrations define
  Forward
  and
  Backward
  classes for apply/rollback
• Applied migrations are tracked in the
  migration_states
  collection
• Migrations run automatically on startup for beta instance

Quick Start

Manual Creation

• Create file:
  sophie_bot/db/migrations/20240125_120000_add_user_preferences.py
• Define old and new model structures if needed
• Implement Forward and Backward migration classes
• Test migration locally
• Add tests to
  tests/test_migrations.py

Example 1: Add New Field

Example 2: Rename Field

Example 3: Convert Data Type

Example 4: Bulk Migration with Free Fall

Iterative Migration

• Most document transformations
• When you need to transform data field-by-field
• Simple field additions, renames, or type conversions

• Beanie handles document loading and saving
• Memory efficient (processes one document at a time)
• Can be resumed if interrupted
• Good for large collections

• Limited control over database session
• Can be slower for bulk operations
• Hard to track progress

Free Fall Migration

• Bulk operations (rebuilding indexes, collections)
• When you need full database control
• Complex transformations requiring multiple collections
• Performance-critical operations

• Full control over database session
• Can use batch processing
• Better performance for bulk operations
• Can track progress manually

• More complex to implement
• Must handle document loading/saving manually
• Risk of memory issues with large collections

Automatic (Recommended for Beta)

Manual

Check Status

Rollback

Enabling Transactions

• Configure MongoDB replica set (required for transactions)
• Enable replica set in config:

When to Use Transactions

• Multi-collection migrations
• Critical data transformations
• Rollback safety is essential
• Collections with relational dependencies

• Very large collections (millions of documents)
• Simple field additions
• Non-critical migrations
• When replica set is not available

Transaction Limitations

• Document size: 16MB per document
• Transaction size: 16MB total
• Operation count: Varies by MongoDB version
• Execution time: 60 seconds default

Batching Strategy

Progress Monitoring

Performance Tips

• Use appropriate batch size: 1000-10000 typically works well
• Create indexes before migration: Improves query performance
• Drop indexes before bulk inserts: Faster for large insertions
• Monitor memory usage: Adjust batch size if memory issues occur
• Run during maintenance: Reduces impact on users

Running Tests

Writing Tests

• Create migration on development branch
• Test locally:
  make migrate_up
• Add tests to
  tests/test_migrations.py
• Test in staging (beta instance)
• Verify status:
  make migrate_status
• Open merge request with migration details
• CI runs tests automatically
• Merge to main
• Deploy to beta - migrations run automatically
• Verify in beta
• Deploy to stable if needed

Migration Failed Mid-Execution

• Migration error in logs
• Some documents migrated, some not
• Migration state may or may not be recorded

• Check logs for specific error details
• Manually fix affected documents if needed
• Re-run migration - it will skip already-applied steps
• Consider rolling back if state is inconsistent

Migration Too Slow

• Migration taking hours to complete
• High CPU/memory usage
• Database slow to respond

• Use
  @free_fall_migration()
  for better control
• Implement batching with
  migration_batch_size
• Create appropriate indexes before migration
• Consider running during maintenance window
• Monitor and optimize database performance

Need to Roll Back

• Stop application to prevent conflicts
• Identify migration to rollback:
  Loading...
• Run rollback:
  Loading...
• Verify rollback with status check
• Fix migration code if issue was in migration
• Re-apply migration if needed

• Rollbacks modify data - test in staging first
• Rollbacks may take time for large collections
• Monitor logs during rollback
• Have data backup if possible

Transaction Errors

• Transaction numbers do not match
  error
• Transaction is not in progress
  error
• Migration fails with transaction-related error

• Check MongoDB is running as replica set
• Verify
  mongo_use_replica_set = True
  in config
• Disable transactions for large collections:
  Loading...
• Use free fall migration instead

Before Creating Migration

• Document migration purpose and expected changes
• Identify affected models and collections
• Estimate document count for affected collections
• Plan both Forward and Backward logic
• Consider impact on running application
• Add tests for migration
• Update documentation
• Get code review

Before Deploying to Production

• Verify migration passed in CI
• Test in staging environment
• Confirm data integrity after migration
• Document rollback procedure
• Notify team of deployment window
• Monitor logs after deployment
• Verify application functionality

Common Pitfalls

• Create migrations without backward classes
• Skip testing migrations
• Run migrations on production without testing
• Use broad exception handlers that hide errors
• Forget to document migration impact
• Run migrations on stable instance (use beta only)

• Implement both Forward and Backward classes
• Test migrations thoroughly in staging
• Use descriptive migration names
• Handle large collections with batching
• Monitor migration progress with logs
• Document rollback procedures

• Beanie Migration Documentation
• Project AGENTS.md
• MongoDB Transactions
• Beanie ODM Documentation

• Migration file created with proper naming
• Forward and Backward classes implemented
• Docstring with description and impact
• Tests written in
  tests/test_migrations.py
• Code reviewed by team
• Tested locally with
  make migrate_up
• Tested in staging environment

• Migration applied successfully
• Data integrity verified
• Application functionality tested
• Performance acceptable
• Logs reviewed for errors
• Documentation updated
• Rollback procedure documented

• Staging migration successful
• Rollback procedure documented
• Team notified of deployment
• Maintenance window scheduled (if needed)
• Monitoring configured
• Backup taken (if possible)