# S3 Music Storage & Playback Feature This document describes the implementation of S3-compatible storage for music files with browser playback capabilities in the Rekordbox Reader application. ## ๐ŸŽต Features - **S3-Compatible Storage**: Store music files in MinIO (local) or any S3-compatible service - **Audio Metadata Extraction**: Automatically extract artist, album, title, duration, etc. - **Browser Playback**: Stream music files directly in the browser - **File Management**: Upload, delete, and organize music files - **Rekordbox Integration**: Link uploaded files to existing Rekordbox songs - **Search & Filter**: Find music by artist, album, genre, or text search ## ๐Ÿ—๏ธ Architecture ### Backend Components 1. **S3Service** (`src/services/s3Service.ts`) - Handles file upload/download/delete operations - Generates presigned URLs for secure access - Manages S3 bucket operations 2. **AudioMetadataService** (`src/services/audioMetadataService.ts`) - Extracts metadata from audio files - Validates audio file formats - Provides utility functions for formatting 3. **MusicFile Model** (`src/models/MusicFile.ts`) - MongoDB schema for music file metadata - Links to existing Song model - Includes search indexes 4. **Music Routes** (`src/routes/music.ts`) - REST API endpoints for music operations - File upload handling with multer - Streaming and metadata endpoints ### Frontend Components (To be implemented) 1. **Music Upload Component** - Drag & drop file upload - Progress indicators - Batch upload support 2. **Music Player Component** - HTML5 audio player - Custom controls - Playlist integration 3. **Music Library Component** - Grid/list view of music files - Search and filter - Metadata display ## ๐Ÿš€ Setup Instructions ### 1. Local Development Setup #### Start MinIO (S3-compatible storage) ```bash # Using Docker Compose docker-compose -f docker-compose.dev.yml up -d # Or manually docker run -d \ --name minio \ -p 9000:9000 \ -p 9001:9001 \ -e MINIO_ROOT_USER=minioadmin \ -e MINIO_ROOT_PASSWORD=minioadmin \ -v minio_data:/data \ minio/minio server /data --console-address ":9001" ``` #### Access MinIO Console - URL: http://localhost:9001 - Username: `minioadmin` - Password: `minioadmin` #### Create Music Bucket ```bash # Using MinIO client mc alias set myminio http://localhost:9000 minioadmin minioadmin mc mb myminio/music-files mc policy set public myminio/music-files ``` ### 2. Environment Variables Create `.env` file in the backend directory: ```env # MongoDB MONGODB_URI=mongodb://localhost:27017/rekordbox # S3/MinIO Configuration S3_ENDPOINT=http://localhost:9000 S3_ACCESS_KEY_ID=minioadmin S3_SECRET_ACCESS_KEY=minioadmin S3_BUCKET_NAME=music-files S3_REGION=us-east-1 # Server PORT=3000 NODE_ENV=development ``` ### 3. Install Dependencies ```bash # Install backend dependencies cd packages/backend npm install # Install frontend dependencies (when implemented) cd ../frontend npm install ``` ## ๐Ÿ“ก API Endpoints ### Music File Management #### Upload Single File ```http POST /api/music/upload Content-Type: multipart/form-data file: [audio file] ``` #### Upload Multiple Files ```http POST /api/music/batch-upload Content-Type: multipart/form-data files: [audio files] ``` #### List Music Files ```http GET /api/music?page=1&limit=20&search=artist&genre=electronic ``` #### Get Streaming URL ```http GET /api/music/:id/stream ``` #### Get Presigned URL ```http GET /api/music/:id/presigned?expiresIn=3600 ``` #### Get File Metadata ```http GET /api/music/:id/metadata ``` #### Delete File ```http DELETE /api/music/:id ``` #### Link to Song ```http POST /api/music/:id/link-song/:songId ``` ### Response Examples #### Upload Response ```json { "message": "File uploaded successfully", "musicFile": { "_id": "64f8a1b2c3d4e5f6a7b8c9d0", "originalName": "track.mp3", "s3Key": "music/uuid.mp3", "s3Url": "music-files/music/uuid.mp3", "contentType": "audio/mpeg", "size": 5242880, "title": "Track Title", "artist": "Artist Name", "album": "Album Name", "duration": 180.5, "format": "mp3", "uploadedAt": "2024-01-15T10:30:00.000Z" } } ``` #### Streaming Response ```json { "streamingUrl": "http://localhost:9000/music-files/music/uuid.mp3", "musicFile": { // ... music file metadata } } ``` ## ๐ŸŽต Supported Audio Formats - MP3 (.mp3) - WAV (.wav) - FLAC (.flac) - AAC (.aac, .m4a) - OGG (.ogg) - WMA (.wma) - Opus (.opus) ## ๐Ÿ”ง Configuration Options ### S3 Service Configuration ```typescript const s3Config = { endpoint: 'http://localhost:9000', // MinIO endpoint accessKeyId: 'minioadmin', // Access key secretAccessKey: 'minioadmin', // Secret key bucketName: 'music-files', // Bucket name region: 'us-east-1', // Region }; ``` ### File Upload Limits - **Single file**: 100MB - **Batch upload**: 10 files per request - **Supported formats**: Audio files only ### Streaming Configuration - **Direct URL**: For public buckets (MinIO) - **Presigned URL**: For private buckets (AWS S3) - **Expiration**: Configurable (default: 1 hour) ## ๐Ÿ”’ Security Considerations ### File Access - Use presigned URLs for secure access - Set appropriate expiration times - Implement user authentication (future) ### File Validation - Validate file types on upload - Check file size limits - Sanitize file names ### Storage Security - Use environment variables for credentials - Implement bucket policies - Regular backup procedures ## ๐Ÿš€ Production Deployment ### AWS S3 Setup 1. Create S3 bucket 2. Configure CORS policy 3. Set up IAM user with appropriate permissions 4. Update environment variables ### MinIO Production Setup 1. Deploy MinIO cluster 2. Configure SSL/TLS 3. Set up monitoring 4. Implement backup strategy ### CDN Integration - Use CloudFront with S3 - Configure caching policies - Optimize for audio streaming ## ๐Ÿงช Testing ### Manual Testing 1. Start MinIO and MongoDB 2. Upload test audio files 3. Verify metadata extraction 4. Test streaming functionality 5. Check file deletion ### API Testing ```bash # Test health endpoint curl http://localhost:3000/api/health # Test file upload curl -X POST -F "file=@test.mp3" http://localhost:3000/api/music/upload # Test streaming curl http://localhost:3000/api/music/:id/stream ``` ## ๐Ÿ”„ Integration with Existing Features ### Rekordbox XML Import - Match uploaded files with XML entries - Link files to existing songs - Maintain playlist relationships ### Playlist Management - Include music files in playlists - Stream playlist sequences - Export playlists with file references ## ๐Ÿ“ˆ Performance Optimization ### Streaming Optimization - Implement range requests - Use appropriate cache headers - Optimize for mobile playback ### Database Optimization - Index frequently queried fields - Implement pagination - Use text search indexes ### Storage Optimization - Implement file compression - Use appropriate storage classes - Monitor storage costs ## ๐Ÿ› Troubleshooting ### Common Issues 1. **MinIO Connection Error** - Check MinIO is running: `docker ps` - Verify endpoint URL - Check credentials 2. **File Upload Fails** - Check file size limits - Verify file format - Check bucket permissions 3. **Streaming Issues** - Verify bucket is public (MinIO) - Check CORS configuration - Test with different browsers 4. **Metadata Extraction Fails** - Check file format support - Verify file integrity - Check music-metadata library ### Debug Commands ```bash # Check MinIO status docker logs minio # Test S3 connection mc ls myminio/music-files # Check MongoDB connection mongosh rekordbox --eval "db.musicfiles.find().limit(1)" ``` ## ๐Ÿ”ฎ Future Enhancements ### Planned Features - [ ] Audio visualization (waveform) - [ ] Playlist streaming - [ ] Audio format conversion - [ ] User authentication - [ ] File sharing - [ ] Mobile app support ### Technical Improvements - [ ] WebSocket streaming - [ ] Progressive download - [ ] Audio caching - [ ] CDN integration - [ ] Analytics tracking ## ๐Ÿ“š Resources - [MinIO Documentation](https://docs.min.io/) - [AWS S3 Documentation](https://docs.aws.amazon.com/s3/) - [music-metadata Library](https://github.com/Borewit/music-metadata) - [HTML5 Audio API](https://developer.mozilla.org/en-US/docs/Web/API/HTMLAudioElement)