rekordbox-viewer/S3_STORAGE_README.md

S3 Music Storage & Playback Feature

This document describes the implementation of S3-compatible storage for music files with browser playback capabilities in the Rekordbox Reader application.

🎵 Features

• S3-Compatible Storage: Store music files in MinIO (local) or any S3-compatible service
• Audio Metadata Extraction: Automatically extract artist, album, title, duration, etc.
• Browser Playback: Stream music files directly in the browser
• File Management: Upload, delete, and organize music files
• Rekordbox Integration: Link uploaded files to existing Rekordbox songs
• Search & Filter: Find music by artist, album, genre, or text search

🏗️ Architecture

Backend Components

1. S3Service (src/services/s3Service.ts)

   • Handles file upload/download/delete operations
   • Generates presigned URLs for secure access
   • Manages S3 bucket operations

2. AudioMetadataService (src/services/audioMetadataService.ts)

   • Extracts metadata from audio files
   • Validates audio file formats
   • Provides utility functions for formatting

3. MusicFile Model (src/models/MusicFile.ts)

   • MongoDB schema for music file metadata
   • Links to existing Song model
   • Includes search indexes

4. Music Routes (src/routes/music.ts)

   • REST API endpoints for music operations
   • File upload handling with multer
   • Streaming and metadata endpoints

Frontend Components (To be implemented)

1. Music Upload Component

   • Drag & drop file upload
   • Progress indicators
   • Batch upload support

2. Music Player Component

   • HTML5 audio player
   • Custom controls
   • Playlist integration

3. Music Library Component

   • Grid/list view of music files
   • Search and filter
   • Metadata display

🚀 Setup Instructions

1. Local Development Setup

Start MinIO (S3-compatible storage)

# Using Docker Compose
docker-compose -f docker-compose.dev.yml up -d

# Or manually
docker run -d \
  --name minio \
  -p 9000:9000 \
  -p 9001:9001 \
  -e MINIO_ROOT_USER=minioadmin \
  -e MINIO_ROOT_PASSWORD=minioadmin \
  -v minio_data:/data \
  minio/minio server /data --console-address ":9001"

Access MinIO Console

• URL: http://localhost:9001
• Username: minioadmin
• Password: minioadmin

Create Music Bucket

# Using MinIO client
mc alias set myminio http://localhost:9000 minioadmin minioadmin
mc mb myminio/music-files
mc policy set public myminio/music-files

2. Environment Variables

Create .env file in the backend directory:

# MongoDB
MONGODB_URI=mongodb://localhost:27017/rekordbox

# S3/MinIO Configuration
S3_ENDPOINT=http://localhost:9000
S3_ACCESS_KEY_ID=minioadmin
S3_SECRET_ACCESS_KEY=minioadmin
S3_BUCKET_NAME=music-files
S3_REGION=us-east-1

# Server
PORT=3000
NODE_ENV=development

3. Install Dependencies

# Install backend dependencies
cd packages/backend
npm install

# Install frontend dependencies (when implemented)
cd ../frontend
npm install

📡 API Endpoints

Music File Management

Upload Single File

POST /api/music/upload
Content-Type: multipart/form-data

file: [audio file]

Upload Multiple Files

POST /api/music/batch-upload
Content-Type: multipart/form-data

files: [audio files]

List Music Files

GET /api/music?page=1&limit=20&search=artist&genre=electronic

Get Streaming URL

GET /api/music/:id/stream

Get Presigned URL

GET /api/music/:id/presigned?expiresIn=3600

Get File Metadata

GET /api/music/:id/metadata

Delete File

DELETE /api/music/:id

Link to Song

POST /api/music/:id/link-song/:songId

Response Examples

Upload Response

{
  "message": "File uploaded successfully",
  "musicFile": {
    "_id": "64f8a1b2c3d4e5f6a7b8c9d0",
    "originalName": "track.mp3",
    "s3Key": "music/uuid.mp3",
    "s3Url": "music-files/music/uuid.mp3",
    "contentType": "audio/mpeg",
    "size": 5242880,
    "title": "Track Title",
    "artist": "Artist Name",
    "album": "Album Name",
    "duration": 180.5,
    "format": "mp3",
    "uploadedAt": "2024-01-15T10:30:00.000Z"
  }
}

Streaming Response

{
  "streamingUrl": "http://localhost:9000/music-files/music/uuid.mp3",
  "musicFile": {
    // ... music file metadata
  }
}

🎵 Supported Audio Formats

• MP3 (.mp3)
• WAV (.wav)
• FLAC (.flac)
• AAC (.aac, .m4a)
• OGG (.ogg)
• WMA (.wma)
• Opus (.opus)

🔧 Configuration Options

S3 Service Configuration

const s3Config = {
  endpoint: 'http://localhost:9000',        // MinIO endpoint
  accessKeyId: 'minioadmin',                // Access key
  secretAccessKey: 'minioadmin',            // Secret key
  bucketName: 'music-files',                // Bucket name
  region: 'us-east-1',                      // Region
};

File Upload Limits

• Single file: 100MB
• Batch upload: 10 files per request
• Supported formats: Audio files only

Streaming Configuration

• Direct URL: For public buckets (MinIO)
• Presigned URL: For private buckets (AWS S3)
• Expiration: Configurable (default: 1 hour)

🔒 Security Considerations

File Access

• Use presigned URLs for secure access
• Set appropriate expiration times
• Implement user authentication (future)

File Validation

• Validate file types on upload
• Check file size limits
• Sanitize file names

Storage Security

• Use environment variables for credentials
• Implement bucket policies
• Regular backup procedures

🚀 Production Deployment

AWS S3 Setup

1. Create S3 bucket
2. Configure CORS policy
3. Set up IAM user with appropriate permissions
4. Update environment variables

MinIO Production Setup

1. Deploy MinIO cluster
2. Configure SSL/TLS
3. Set up monitoring
4. Implement backup strategy

CDN Integration

• Use CloudFront with S3
• Configure caching policies
• Optimize for audio streaming

🧪 Testing

Manual Testing

1. Start MinIO and MongoDB
2. Upload test audio files
3. Verify metadata extraction
4. Test streaming functionality
5. Check file deletion

API Testing

# Test health endpoint
curl http://localhost:3000/api/health

# Test file upload
curl -X POST -F "file=@test.mp3" http://localhost:3000/api/music/upload

# Test streaming
curl http://localhost:3000/api/music/:id/stream

🔄 Integration with Existing Features

Rekordbox XML Import

• Match uploaded files with XML entries
• Link files to existing songs
• Maintain playlist relationships

Playlist Management

• Include music files in playlists
• Stream playlist sequences
• Export playlists with file references

📈 Performance Optimization

Streaming Optimization

• Implement range requests
• Use appropriate cache headers
• Optimize for mobile playback

Database Optimization

• Index frequently queried fields
• Implement pagination
• Use text search indexes

Storage Optimization

• Implement file compression
• Use appropriate storage classes
• Monitor storage costs

🐛 Troubleshooting

Common Issues

1. MinIO Connection Error

   • Check MinIO is running: docker ps
   • Verify endpoint URL
   • Check credentials

2. File Upload Fails

   • Check file size limits
   • Verify file format
   • Check bucket permissions

3. Streaming Issues

   • Verify bucket is public (MinIO)
   • Check CORS configuration
   • Test with different browsers

4. Metadata Extraction Fails

   • Check file format support
   • Verify file integrity
   • Check music-metadata library

Debug Commands

# Check MinIO status
docker logs minio

# Test S3 connection
mc ls myminio/music-files

# Check MongoDB connection
mongosh rekordbox --eval "db.musicfiles.find().limit(1)"

🔮 Future Enhancements

Planned Features

• Audio visualization (waveform)
• Playlist streaming
• Audio format conversion
• User authentication
• File sharing
• Mobile app support

Technical Improvements

• WebSocket streaming
• Progressive download
• Audio caching
• CDN integration
• Analytics tracking

📚 Resources

• MinIO Documentation
• AWS S3 Documentation
• music-metadata Library
• HTML5 Audio API