master/rekordbox-viewer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: master:e58d42bea2c58a5b7b506ede1478928f22161b10
Branches Tags
master:main
master:frontend-ux-improvements
master:feature/desktop-sync
..
compare: master:dc8254772c8a0ddad1bd4488c49c6eb832acb80c
Branches Tags
master:frontend-ux-improvements
master:main
master:feature/desktop-sync

No commits in common. "e58d42bea2c58a5b7b506ede1478928f22161b10" and "dc8254772c8a0ddad1bd4488c49c6eb832acb80c" have entirely different histories.
e58d42bea2 ... dc8254772c

30 changed files with 261 additions and 3023 deletions
Show Stats Expand all files Collapse all files

30
README.md
View File

@ -9,8 +9,6 @@ A web application for reading, managing, and exporting Rekordbox XML files. This
- **Playlist Management**: Create, edit, and organize playlists and folders - **Playlist Management**: Create, edit, and organize playlists and folders
- **Song Details**: View detailed information about tracks including BPM, key, rating, etc. - **Song Details**: View detailed information about tracks including BPM, key, rating, etc.
- **Export Functionality**: Export modified libraries back to XML format - **Export Functionality**: Export modified libraries back to XML format
- **Music File Storage**: Upload and stream music files with multiple storage providers
- **Storage Providers**: Support for S3-compatible storage (AWS S3, MinIO) and WebDAV (Nextcloud, ownCloud)
- **Responsive Design**: Works on desktop and mobile devices - **Responsive Design**: Works on desktop and mobile devices
- **Database Storage**: Persistent storage using MongoDB - **Database Storage**: Persistent storage using MongoDB
@ -132,34 +130,6 @@ The frontend is configured to connect to the backend API. The API URL can be con
- Development: `packages/frontend/src/services/api.ts` - Development: `packages/frontend/src/services/api.ts`
- Production: Environment variable `VITE_API_URL` in Docker - Production: Environment variable `VITE_API_URL` in Docker
### Storage Configuration
The application supports multiple storage providers for music files:
#### S3-Compatible Storage (AWS S3, MinIO)
```env
STORAGE_PROVIDER=s3
S3_ENDPOINT=http://localhost:9000
S3_ACCESS_KEY_ID=minioadmin
S3_SECRET_ACCESS_KEY=minioadmin
S3_BUCKET_NAME=music-files
S3_REGION=us-east-1
S3_USE_SSL=false
```
#### WebDAV (Nextcloud, ownCloud)
```env
STORAGE_PROVIDER=webdav
WEBDAV_URL=https://your-nextcloud.com/remote.php/dav/files/username/
WEBDAV_USERNAME=your-username
WEBDAV_PASSWORD=your-password-or-app-password
WEBDAV_BASE_PATH=/music-files
```
You can also configure storage through the web interface at **Configuration → Storage Configuration**.
For detailed setup instructions, see [WEBDAV_INTEGRATION.md](./WEBDAV_INTEGRATION.md).
## 📊 API Endpoints ## 📊 API Endpoints
- `GET /api/health` - Health check - `GET /api/health` - Health check

258
WEBDAV_INTEGRATION.md
View File

@ -1,258 +0,0 @@
# WebDAV Integration for Rekordbox Reader
This document describes the WebDAV integration that allows Rekordbox Reader to work with self-hosted storage solutions like Nextcloud and ownCloud.
## 🎯 Overview
The application now supports two storage providers:
- **S3-Compatible Storage** (AWS S3, MinIO, etc.)
- **WebDAV** (Nextcloud, ownCloud, etc.)
Users can choose between these providers in the Storage Configuration page.
## 🏗️ Architecture
### Storage Provider Abstraction
The system uses a provider abstraction pattern:
```
StorageProvider Interface
├── S3Service (implements StorageProvider)
└── WebDAVService (implements StorageProvider)
```
### Key Components
1. **StorageProvider Interface** (`src/services/storageProvider.ts`)
- Defines common operations for all storage providers
- Factory pattern for creating providers
- Configuration loading and validation
2. **WebDAVService** (`src/services/webdavService.ts`)
- Implements WebDAV operations using the `webdav` npm package
- Supports Nextcloud, ownCloud, and other WebDAV-compatible servers
- Handles file upload, download, listing, and deletion
3. **Updated Configuration System**
- New `/api/config/storage` endpoints
- Support for both S3 and WebDAV configuration
- Backward compatibility with existing S3 configuration
4. **Frontend Storage Configuration**
- Provider selection (S3 vs WebDAV)
- Dynamic form fields based on selected provider
- Connection testing for both providers
## 🚀 Setup Instructions
### 1. Backend Configuration
The backend automatically detects the storage provider from the configuration file:
**File: `storage-config.json`**
```json
{
"provider": "webdav",
"url": "https://your-nextcloud.com/remote.php/dav/files/username/",
"username": "your-username",
"password": "your-password-or-app-password",
"basePath": "/music-files"
}
```
### 2. Environment Variables
You can also configure WebDAV using environment variables:
```bash
STORAGE_PROVIDER=webdav
WEBDAV_URL=https://your-nextcloud.com/remote.php/dav/files/username/
WEBDAV_USERNAME=your-username
WEBDAV_PASSWORD=your-password-or-app-password
WEBDAV_BASE_PATH=/music-files
```
### 3. Frontend Configuration
1. Navigate to **Configuration → Storage Configuration**
2. Select **WebDAV** as the storage provider
3. Fill in your WebDAV server details:
- **URL**: Your Nextcloud/ownCloud WebDAV URL
- **Username**: Your account username
- **Password**: Your password or app password
- **Base Path**: Optional subfolder for music files
4. Click **Test Connection** to verify
5. Click **Save Configuration** to apply
## 🔧 Nextcloud Setup
### 1. Enable WebDAV
WebDAV is enabled by default in Nextcloud. You can verify this in:
- **Settings → Administration → Basic settings**
### 2. Create App Password (Recommended)
For better security, create an app password:
1. Go to **Settings → Personal → Security**
2. Scroll down to **App passwords**
3. Create a new app password for "Rekordbox Reader"
4. Use this password instead of your main password
### 3. Get WebDAV URL
Your WebDAV URL follows this pattern:
```
https://your-nextcloud.com/remote.php/dav/files/username/
```
Replace:
- `your-nextcloud.com` with your Nextcloud domain
- `username` with your Nextcloud username
### 4. Create Music Folder (Optional)
You can create a dedicated folder for music files:
1. In Nextcloud, create a folder called `music-files`
2. Set this as the `basePath` in the configuration
## 🔧 ownCloud Setup
The setup is similar to Nextcloud:
1. Enable WebDAV in ownCloud settings
2. Create an app password for security
3. Use the WebDAV URL: `https://your-owncloud.com/remote.php/dav/files/username/`
4. Configure the same way as Nextcloud
## 🧪 Testing
### Backend Test
Run the WebDAV test script:
```bash
cd packages/backend
node test-webdav.js
```
Make sure to update the configuration in the test file first.
### Frontend Test
1. Go to **Configuration → Storage Configuration**
2. Select **WebDAV** provider
3. Enter your WebDAV details
4. Click **Test Connection**
5. Verify the connection is successful
## 📁 File Operations
The WebDAV service supports all standard file operations:
- **Upload**: Upload music files to WebDAV storage
- **Download**: Download files for playback
- **List**: List all files and folders
- **Delete**: Remove files from storage
- **Metadata**: Get file information
- **Streaming**: Generate streaming URLs
## 🔒 Security Considerations
1. **Use App Passwords**: Don't use your main Nextcloud/ownCloud password
2. **HTTPS Only**: Always use HTTPS URLs for WebDAV connections
3. **Base Path**: Use a dedicated folder for music files
4. **Permissions**: Ensure the WebDAV user has appropriate permissions
## 🐛 Troubleshooting
### Common Issues
1. **Connection Failed**
- Verify the WebDAV URL is correct
- Check username and password
- Ensure WebDAV is enabled on your server
2. **Permission Denied**
- Check if the user has write permissions
- Verify the base path exists and is accessible
3. **SSL/TLS Errors**
- Ensure you're using HTTPS
- Check if the SSL certificate is valid
4. **File Upload Fails**
- Check available storage space
- Verify file permissions
- Ensure the file format is supported
### Debug Mode
Enable debug logging by setting:
```bash
DEBUG=webdav:*
```
## 🔄 Migration from S3
If you're migrating from S3 to WebDAV:
1. Export your current configuration
2. Set up WebDAV storage
3. Update the configuration to use WebDAV
4. The application will automatically use the new storage provider
## 📊 Performance
WebDAV performance depends on:
- Network latency to your server
- Server performance and storage type
- File sizes and concurrent operations
For best performance:
- Use a local or fast Nextcloud/ownCloud instance
- Consider using SSD storage
- Optimize your network connection
## 🎵 Supported File Formats
The WebDAV integration supports all audio formats supported by the application:
- MP3
- WAV
- FLAC
- M4A
- AAC
- OGG
- OPUS
- WMA
## 📝 API Endpoints
### Storage Configuration
- `GET /api/config/storage` - Get current storage configuration
- `POST /api/config/storage` - Save storage configuration
- `POST /api/config/storage/test` - Test storage connection
### Legacy S3 Endpoints
The following endpoints are still available for backward compatibility:
- `GET /api/config/s3` - Get S3 configuration
- `POST /api/config/s3` - Save S3 configuration
- `POST /api/config/s3/test` - Test S3 connection
## 🤝 Contributing
When adding new storage providers:
1. Implement the `StorageProvider` interface
2. Add the provider to `StorageProviderFactory`
3. Update the frontend configuration UI
4. Add appropriate tests
5. Update this documentation
## 📄 License
This WebDAV integration follows the same license as the main Rekordbox Reader project.

270
package-lock.json generated
View File

@ -1180,14 +1180,6 @@
"node": ">=6.9.0" "node": ">=6.9.0"
} }
}, },
"node_modules/@buttercup/fetch": {
"version": "0.2.1",
"resolved": "https://registry.npmjs.org/@buttercup/fetch/-/fetch-0.2.1.tgz",
"integrity": "sha512-sCgECOx8wiqY8NN1xN22BqqKzXYIG2AicNLlakOAI4f0WgyLVUbAigMf8CZhBtJxdudTcB1gD5lciqi44jwJvg==",
"optionalDependencies": {
"node-fetch": "^3.3.0"
}
},
"node_modules/@chakra-ui/accordion": { "node_modules/@chakra-ui/accordion": {
"version": "2.3.1", "version": "2.3.1",
"resolved": "https://registry.npmjs.org/@chakra-ui/accordion/-/accordion-2.3.1.tgz", "resolved": "https://registry.npmjs.org/@chakra-ui/accordion/-/accordion-2.3.1.tgz",
@ -4269,13 +4261,9 @@
"version": "1.0.2", "version": "1.0.2",
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz", "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz",
"integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==", "integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==",
"dev": true,
"license": "MIT" "license": "MIT"
}, },
"node_modules/base-64": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/base-64/-/base-64-1.0.0.tgz",
"integrity": "sha512-kwDPIFCGx0NZHog36dj+tHiwP4QMzsZ3AgMViUBKI0+V5n4U0ufTCUMhnQ04diaRI8EX/QcPfql7zlhZ7j4zgg=="
},
"node_modules/base64-js": { "node_modules/base64-js": {
"version": "1.5.1", "version": "1.5.1",
"resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz", "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz",
@ -4440,11 +4428,6 @@
"node": ">=10.16.0" "node": ">=10.16.0"
} }
}, },
"node_modules/byte-length": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/byte-length/-/byte-length-1.0.2.tgz",
"integrity": "sha512-ovBpjmsgd/teRmgcPh23d4gJvxDoXtAzEL9xTfMU8Yc2kqCDb7L9jAG0XHl1nzuGl+h3ebCIF1i62UFyA9V/2Q=="
},
"node_modules/bytes": { "node_modules/bytes": {
"version": "3.1.2", "version": "3.1.2",
"resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz", "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz",
@ -4527,14 +4510,6 @@
"url": "https://github.com/chalk/chalk?sponsor=1" "url": "https://github.com/chalk/chalk?sponsor=1"
} }
}, },
"node_modules/charenc": {
"version": "0.0.2",
"resolved": "https://registry.npmjs.org/charenc/-/charenc-0.0.2.tgz",
"integrity": "sha512-yrLQ/yVUFXkzg7EDQsPieE/53+0RlaWTs+wBrvW36cyilJ2SaDWfl4Yj7MtLTXleV9uEKefbAGUPv2/iWSooRA==",
"engines": {
"node": "*"
}
},
"node_modules/cliui": { "node_modules/cliui": {
"version": "8.0.1", "version": "8.0.1",
"resolved": "https://registry.npmjs.org/cliui/-/cliui-8.0.1.tgz", "resolved": "https://registry.npmjs.org/cliui/-/cliui-8.0.1.tgz",
@ -4741,14 +4716,6 @@
"node": ">= 8" "node": ">= 8"
} }
}, },
"node_modules/crypt": {
"version": "0.0.2",
"resolved": "https://registry.npmjs.org/crypt/-/crypt-0.0.2.tgz",
"integrity": "sha512-mCxBlsHFYh9C+HVpiEacem8FEBnMXgU9gy4zmNC+SXAZNB/1idgp/aulFJ4FgCi7GPEVbfyng092GqL2k2rmow==",
"engines": {
"node": "*"
}
},
"node_modules/css-box-model": { "node_modules/css-box-model": {
"version": "1.2.1", "version": "1.2.1",
"resolved": "https://registry.npmjs.org/css-box-model/-/css-box-model-1.2.1.tgz", "resolved": "https://registry.npmjs.org/css-box-model/-/css-box-model-1.2.1.tgz",
@ -4763,14 +4730,6 @@
"integrity": "sha512-M1uQkMl8rQK/szD0LNhtqxIPLpimGm8sOBwU7lLnCpSbTyY3yeU1Vc7l4KT5zT4s/yOxHH5O7tIuuLOCnLADRw==", "integrity": "sha512-M1uQkMl8rQK/szD0LNhtqxIPLpimGm8sOBwU7lLnCpSbTyY3yeU1Vc7l4KT5zT4s/yOxHH5O7tIuuLOCnLADRw==",
"license": "MIT" "license": "MIT"
}, },
"node_modules/data-uri-to-buffer": {
"version": "4.0.1",
"resolved": "https://registry.npmjs.org/data-uri-to-buffer/-/data-uri-to-buffer-4.0.1.tgz",
"integrity": "sha512-0R9ikRb668HB7QDxT1vkpuUBtqc53YyAwMwGeUFKRojY/NWKvdZ+9UYtRfGmhqNbRkTSVpMbmyhXipFFv2cb/A==",
"engines": {
"node": ">= 12"
}
},
"node_modules/date-fns": { "node_modules/date-fns": {
"version": "2.30.0", "version": "2.30.0",
"resolved": "https://registry.npmjs.org/date-fns/-/date-fns-2.30.0.tgz", "resolved": "https://registry.npmjs.org/date-fns/-/date-fns-2.30.0.tgz",
@ -4883,17 +4842,6 @@
"node": ">= 0.8" "node": ">= 0.8"
} }
}, },
"node_modules/entities": {
"version": "6.0.1",
"resolved": "https://registry.npmjs.org/entities/-/entities-6.0.1.tgz",
"integrity": "sha512-aN97NXWF6AWBTahfVOIrB/NShkzi5H7F9r1s9mD3cDj4Ko5f2qhhVoYMibXF7GlLveb/D2ioWay8lxI97Ven3g==",
"engines": {
"node": ">=0.12"
},
"funding": {
"url": "https://github.com/fb55/entities?sponsor=1"
}
},
"node_modules/error-ex": { "node_modules/error-ex": {
"version": "1.3.2", "version": "1.3.2",
"resolved": "https://registry.npmjs.org/error-ex/-/error-ex-1.3.2.tgz", "resolved": "https://registry.npmjs.org/error-ex/-/error-ex-1.3.2.tgz",
@ -5336,28 +5284,6 @@
"reusify": "^1.0.4" "reusify": "^1.0.4"
} }
}, },
"node_modules/fetch-blob": {
"version": "3.2.0",
"resolved": "https://registry.npmjs.org/fetch-blob/-/fetch-blob-3.2.0.tgz",
"integrity": "sha512-7yAQpD2UMJzLi1Dqv7qFYnPbaPx7ZfFK6PiIxQ4PfkGPyNyl2Ugx+a/umUonmKqjhM4DnfbMvdX6otXq83soQQ==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/jimmywarting"
},
{
"type": "paypal",
"url": "https://paypal.me/jimmywarting"
}
],
"dependencies": {
"node-domexception": "^1.0.0",
"web-streams-polyfill": "^3.0.3"
},
"engines": {
"node": "^12.20 || >= 14.13"
}
},
"node_modules/file-entry-cache": { "node_modules/file-entry-cache": {
"version": "8.0.0", "version": "8.0.0",
"resolved": "https://registry.npmjs.org/file-entry-cache/-/file-entry-cache-8.0.0.tgz", "resolved": "https://registry.npmjs.org/file-entry-cache/-/file-entry-cache-8.0.0.tgz",
@ -5496,17 +5422,6 @@
"node": ">=10" "node": ">=10"
} }
}, },
"node_modules/formdata-polyfill": {
"version": "4.0.10",
"resolved": "https://registry.npmjs.org/formdata-polyfill/-/formdata-polyfill-4.0.10.tgz",
"integrity": "sha512-buewHzMvYL29jdeQTVILecSaZKnt/RJWjoZCF5OW60Z67/GmSLBkOFM7qh1PI3zFNtJbaZL5eQu1vLfazOwj4g==",
"dependencies": {
"fetch-blob": "^3.1.2"
},
"engines": {
"node": ">=12.20.0"
}
},
"node_modules/forwarded": { "node_modules/forwarded": {
"version": "0.2.0", "version": "0.2.0",
"resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz", "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz",
@ -5760,11 +5675,6 @@
"react-is": "^16.7.0" "react-is": "^16.7.0"
} }
}, },
"node_modules/hot-patcher": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/hot-patcher/-/hot-patcher-2.0.1.tgz",
"integrity": "sha512-ECg1JFG0YzehicQaogenlcs2qg6WsXQsxtnbr1i696u5tLUjtJdQAh0u2g0Q5YV45f263Ta1GnUJsc8WIfJf4Q=="
},
"node_modules/http-errors": { "node_modules/http-errors": {
"version": "2.0.0", "version": "2.0.0",
"resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.0.tgz", "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.0.tgz",
@ -5865,11 +5775,6 @@
"integrity": "sha512-zz06S8t0ozoDXMG+ube26zeCTNXcKIPJZJi8hBrF4idCLms4CG9QtK7qBl1boi5ODzFpjswb5JPmHCbMpjaYzg==", "integrity": "sha512-zz06S8t0ozoDXMG+ube26zeCTNXcKIPJZJi8hBrF4idCLms4CG9QtK7qBl1boi5ODzFpjswb5JPmHCbMpjaYzg==",
"license": "MIT" "license": "MIT"
}, },
"node_modules/is-buffer": {
"version": "1.1.6",
"resolved": "https://registry.npmjs.org/is-buffer/-/is-buffer-1.1.6.tgz",
"integrity": "sha512-NcdALwpXkTm5Zvvbk7owOUSvVvBKDgKP5/ewfXEznmQFfs4ZRmanOeKBTjRVjka3QFoN6XJ+9F3USqfHqTaU5w=="
},
"node_modules/is-core-module": { "node_modules/is-core-module": {
"version": "2.16.1", "version": "2.16.1",
"resolved": "https://registry.npmjs.org/is-core-module/-/is-core-module-2.16.1.tgz", "resolved": "https://registry.npmjs.org/is-core-module/-/is-core-module-2.16.1.tgz",
@ -6023,11 +5928,6 @@
"json-buffer": "3.0.1" "json-buffer": "3.0.1"
} }
}, },
"node_modules/layerr": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/layerr/-/layerr-3.0.0.tgz",
"integrity": "sha512-tv754Ki2dXpPVApOrjTyRo4/QegVb9eVFq4mjqp4+NM5NaX7syQvN5BBNfV/ZpAHCEHV24XdUVrBAoka4jt3pA=="
},
"node_modules/levn": { "node_modules/levn": {
"version": "0.4.1", "version": "0.4.1",
"resolved": "https://registry.npmjs.org/levn/-/levn-0.4.1.tgz", "resolved": "https://registry.npmjs.org/levn/-/levn-0.4.1.tgz",
@ -6111,16 +6011,6 @@
"node": ">= 0.4" "node": ">= 0.4"
} }
}, },
"node_modules/md5": {
"version": "2.3.0",
"resolved": "https://registry.npmjs.org/md5/-/md5-2.3.0.tgz",
"integrity": "sha512-T1GITYmFaKuO91vxyoQMFETst+O71VUPEU3ze5GNzDm0OWdP8v1ziTaAEPUr/3kLsY3Sftgz242A1SetQiDL7g==",
"dependencies": {
"charenc": "0.0.2",
"crypt": "0.0.2",
"is-buffer": "~1.1.6"
}
},
"node_modules/media-typer": { "node_modules/media-typer": {
"version": "0.3.0", "version": "0.3.0",
"resolved": "https://registry.npmjs.org/media-typer/-/media-typer-0.3.0.tgz", "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-0.3.0.tgz",
@ -6429,47 +6319,6 @@
"node": ">= 0.6" "node": ">= 0.6"
} }
}, },
"node_modules/nested-property": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/nested-property/-/nested-property-4.0.0.tgz",
"integrity": "sha512-yFehXNWRs4cM0+dz7QxCd06hTbWbSkV0ISsqBfkntU6TOY4Qm3Q88fRRLOddkGh2Qq6dZvnKVAahfhjcUvLnyA=="
},
"node_modules/node-domexception": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/node-domexception/-/node-domexception-1.0.0.tgz",
"integrity": "sha512-/jKZoMpw0F8GRwl4/eLROPA3cfcXtLApP0QzLmUT/HuPCZWyB7IY9ZrMeKw2O/nFIqPQB3PVM9aYm0F312AXDQ==",
"deprecated": "Use your platform's native DOMException instead",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/jimmywarting"
},
{
"type": "github",
"url": "https://paypal.me/jimmywarting"
}
],
"engines": {
"node": ">=10.5.0"
}
},
"node_modules/node-fetch": {
"version": "3.3.2",
"resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-3.3.2.tgz",
"integrity": "sha512-dRB78srN/l6gqWulah9SrxeYnxeddIG30+GOqK/9OlLVyLg3HPnr6SqOWTWOXKRwC2eGYCkZ59NNuSgvSrpgOA==",
"dependencies": {
"data-uri-to-buffer": "^4.0.0",
"fetch-blob": "^3.1.4",
"formdata-polyfill": "^4.0.10"
},
"engines": {
"node": "^12.20.0 || ^14.13.1 || >=16.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/node-fetch"
}
},
"node_modules/node-releases": { "node_modules/node-releases": {
"version": "2.0.19", "version": "2.0.19",
"resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.19.tgz", "resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.19.tgz",
@ -6631,11 +6480,6 @@
"integrity": "sha512-LDJzPVEEEPR+y48z93A0Ed0yXb8pAByGWo/k5YYdYgpY2/2EsOsksJrq7lOHxryrVOn1ejG6oAp8ahvOIQD8sw==", "integrity": "sha512-LDJzPVEEEPR+y48z93A0Ed0yXb8pAByGWo/k5YYdYgpY2/2EsOsksJrq7lOHxryrVOn1ejG6oAp8ahvOIQD8sw==",
"license": "MIT" "license": "MIT"
}, },
"node_modules/path-posix": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/path-posix/-/path-posix-1.0.0.tgz",
"integrity": "sha512-1gJ0WpNIiYcQydgg3Ed8KzvIqTsDpNwq+cjBCssvBtuTWjEqY1AW+i+OepiEMqDCzyro9B2sLAe4RBPajMYFiA=="
},
"node_modules/path-to-regexp": { "node_modules/path-to-regexp": {
"version": "0.1.12", "version": "0.1.12",
"resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.12.tgz", "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.12.tgz",
@ -6780,11 +6624,6 @@
"url": "https://github.com/sponsors/ljharb" "url": "https://github.com/sponsors/ljharb"
} }
}, },
"node_modules/querystringify": {
"version": "2.2.0",
"resolved": "https://registry.npmjs.org/querystringify/-/querystringify-2.2.0.tgz",
"integrity": "sha512-FIqgj2EUvTa7R50u0rGsyTftzjYmv/a3hO345bZNrqabNqjtgiDMgmo4mkUjd+nzU5oF3dClKqFIPUKybUyqoQ=="
},
"node_modules/queue-microtask": { "node_modules/queue-microtask": {
"version": "1.2.3", "version": "1.2.3",
"resolved": "https://registry.npmjs.org/queue-microtask/-/queue-microtask-1.2.3.tgz", "resolved": "https://registry.npmjs.org/queue-microtask/-/queue-microtask-1.2.3.tgz",
@ -7100,11 +6939,6 @@
"node": ">=0.10.0" "node": ">=0.10.0"
} }
}, },
"node_modules/requires-port": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/requires-port/-/requires-port-1.0.0.tgz",
"integrity": "sha512-KigOCHcocU3XODJxsu8i/j8T9tzT4adHiecwORRQ0ZZFcp7ahwXuRU1m+yuO90C5ZUyGeGfocHDI14M3L3yDAQ=="
},
"node_modules/resolve": { "node_modules/resolve": {
"version": "1.22.10", "version": "1.22.10",
"resolved": "https://registry.npmjs.org/resolve/-/resolve-1.22.10.tgz", "resolved": "https://registry.npmjs.org/resolve/-/resolve-1.22.10.tgz",
@ -7841,23 +7675,6 @@
"punycode": "^2.1.0" "punycode": "^2.1.0"
} }
}, },
"node_modules/url-join": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/url-join/-/url-join-5.0.0.tgz",
"integrity": "sha512-n2huDr9h9yzd6exQVnH/jU5mr+Pfx08LRXXZhkLLetAMESRj+anQsTAh940iMrIetKAmry9coFuZQ2jY8/p3WA==",
"engines": {
"node": "^12.20.0 || ^14.13.1 || >=16.0.0"
}
},
"node_modules/url-parse": {
"version": "1.5.10",
"resolved": "https://registry.npmjs.org/url-parse/-/url-parse-1.5.10.tgz",
"integrity": "sha512-WypcfiRhfeUP9vvF0j6rw0J3hrWrw6iZv3+22h6iRMJ/8z1Tj6XfLP4DsUix5MhMPnXpiHDoKyoZ/bdCkwBCiQ==",
"dependencies": {
"querystringify": "^2.1.1",
"requires-port": "^1.0.0"
}
},
"node_modules/use-callback-ref": { "node_modules/use-callback-ref": {
"version": "1.3.3", "version": "1.3.3",
"resolved": "https://registry.npmjs.org/use-callback-ref/-/use-callback-ref-1.3.3.tgz", "resolved": "https://registry.npmjs.org/use-callback-ref/-/use-callback-ref-1.3.3.tgz",
@ -8004,88 +7821,6 @@
} }
} }
}, },
"node_modules/web-streams-polyfill": {
"version": "3.3.3",
"resolved": "https://registry.npmjs.org/web-streams-polyfill/-/web-streams-polyfill-3.3.3.tgz",
"integrity": "sha512-d2JWLCivmZYTSIoge9MsgFCZrt571BikcWGYkjC1khllbTeDlGqZ2D8vD8E/lJa8WGWbb7Plm8/XJYV7IJHZZw==",
"engines": {
"node": ">= 8"
}
},
"node_modules/webdav": {
"version": "5.8.0",
"resolved": "https://registry.npmjs.org/webdav/-/webdav-5.8.0.tgz",
"integrity": "sha512-iuFG7NamJ41Oshg4930iQgfIpRrUiatPWIekeznYgEf2EOraTRcDPTjy7gIOMtkdpKTaqPk1E68NO5PAGtJahA==",
"dependencies": {
"@buttercup/fetch": "^0.2.1",
"base-64": "^1.0.0",
"byte-length": "^1.0.2",
"entities": "^6.0.0",
"fast-xml-parser": "^4.5.1",
"hot-patcher": "^2.0.1",
"layerr": "^3.0.0",
"md5": "^2.3.0",
"minimatch": "^9.0.5",
"nested-property": "^4.0.0",
"node-fetch": "^3.3.2",
"path-posix": "^1.0.0",
"url-join": "^5.0.0",
"url-parse": "^1.5.10"
},
"engines": {
"node": ">=14"
}
},
"node_modules/webdav/node_modules/brace-expansion": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.0.2.tgz",
"integrity": "sha512-Jt0vHyM+jmUBqojB7E1NIYadt0vI0Qxjxd2TErW94wDz+E2LAm5vKMXXwg6ZZBTHPuUlDgQHKXvjGBdfcF1ZDQ==",
"dependencies": {
"balanced-match": "^1.0.0"
}
},
"node_modules/webdav/node_modules/fast-xml-parser": {
"version": "4.5.3",
"resolved": "https://registry.npmjs.org/fast-xml-parser/-/fast-xml-parser-4.5.3.tgz",
"integrity": "sha512-RKihhV+SHsIUGXObeVy9AXiBbFwkVk7Syp8XgwN5U3JV416+Gwp/GO9i0JYKmikykgz/UHRrrV4ROuZEo/T0ig==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/NaturalIntelligence"
}
],
"dependencies": {
"strnum": "^1.1.1"
},
"bin": {
"fxparser": "src/cli/cli.js"
}
},
"node_modules/webdav/node_modules/minimatch": {
"version": "9.0.5",
"resolved": "https://registry.npmjs.org/minimatch/-/minimatch-9.0.5.tgz",
"integrity": "sha512-G6T0ZX48xgozx7587koeX9Ys2NYy6Gmv//P89sEte9V9whIapMNF4idKxnW2QtCcLiTWlb/wfCabAtAFWhhBow==",
"dependencies": {
"brace-expansion": "^2.0.1"
},
"engines": {
"node": ">=16 || 14 >=14.17"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/webdav/node_modules/strnum": {
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/strnum/-/strnum-1.1.2.tgz",
"integrity": "sha512-vrN+B7DBIoTTZjnPNewwhx6cBA/H+IS7rfW68n7XxC1y7uoiGQBxaKzqucGUgavX15dJgiGztLJ8vxuEzwqBdA==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/NaturalIntelligence"
}
]
},
"node_modules/webidl-conversions": { "node_modules/webidl-conversions": {
"version": "7.0.0", "version": "7.0.0",
"resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-7.0.0.tgz", "resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-7.0.0.tgz",
@ -8268,8 +8003,7 @@
"mongoose": "^8.2.1", "mongoose": "^8.2.1",
"multer": "^2.0.0-rc.3", "multer": "^2.0.0-rc.3",
"music-metadata": "^8.1.0", "music-metadata": "^8.1.0",
"uuid": "^11.1.0", "uuid": "^11.1.0"
"webdav": "^5.3.0"
}, },
"devDependencies": { "devDependencies": {
"@types/cors": "^2.8.17", "@types/cors": "^2.8.17",

116
packages/backend/debug-mp3-files.js
View File

@ -1,116 +0,0 @@
import { createClient } from 'webdav';
import fs from 'fs';
async function debugMP3Files() {
try {
// Load configuration
const configData = fs.readFileSync('storage-config.json', 'utf-8');
const config = JSON.parse(configData);
console.log('🔍 WebDAV Configuration:');
console.log('URL:', config.url);
console.log('Username:', config.username);
console.log('Base Path:', config.basePath);
console.log('');
// Create WebDAV client
const client = createClient(config.url, {
username: config.username,
password: config.password,
});
console.log('🔗 Testing connection...');
const basePath = config.basePath || '/Music';
try {
// Test deep listing
console.log('📁 Testing deep directory listing for MP3 files...');
const deepContents = await client.getDirectoryContents(basePath, { deep: true });
console.log('Deep listing response type:', typeof deepContents);
console.log('Is array:', Array.isArray(deepContents));
if (Array.isArray(deepContents)) {
console.log('Total items found:', deepContents.length);
// Filter for MP3 files specifically
const mp3Files = deepContents.filter(item => {
if (item && typeof item === 'object' && 'type' in item && item.type === 'file') {
const filename = item.basename || item.filename.split('/').pop() || '';
return filename.toLowerCase().endsWith('.mp3');
}
return false;
});
console.log('🎵 MP3 Files found:', mp3Files.length);
console.log('');
// Show first 20 MP3 files
console.log('🎵 First 20 MP3 files:');
mp3Files.slice(0, 20).forEach((file, index) => {
const relativePath = file.filename.replace(basePath + '/', '');
console.log(` ${index + 1}. ${relativePath} (${file.size} bytes)`);
});
if (mp3Files.length > 20) {
console.log(` ... and ${mp3Files.length - 20} more MP3 files`);
}
// Check if there are any patterns in the file paths
console.log('');
console.log('📁 Directory distribution of MP3 files:');
const dirCounts = new Map();
mp3Files.forEach(file => {
const relativePath = file.filename.replace(basePath + '/', '');
const dir = relativePath.split('/')[0];
dirCounts.set(dir, (dirCounts.get(dir) || 0) + 1);
});
const sortedDirs = Array.from(dirCounts.entries()).sort((a, b) => b[1] - a[1]);
sortedDirs.forEach(([dir, count]) => {
console.log(` 📁 ${dir}: ${count} MP3 files`);
});
// Test if there's a limit by checking specific directories
console.log('');
console.log('🔍 Testing specific directories for MP3 files...');
const testDirs = ['Gekocht', 'Merijn Music', 'Musica'];
for (const testDir of testDirs) {
try {
const dirPath = `${basePath}/${testDir}`;
const dirContents = await client.getDirectoryContents(dirPath, { deep: true });
const dirItems = Array.isArray(dirContents) ? dirContents : [dirContents];
const dirMp3Files = dirItems.filter(item => {
if (item && typeof item === 'object' && 'type' in item && item.type === 'file') {
const filename = item.basename || item.filename.split('/').pop() || '';
return filename.toLowerCase().endsWith('.mp3');
}
return false;
});
console.log(` 📁 ${testDir}: ${dirMp3Files.length} MP3 files`);
} catch (error) {
console.log(` ❌ ${testDir}: Error - ${error.message}`);
}
}
} else {
console.log('❌ Deep listing returned non-array response:', deepContents);
}
} catch (error) {
console.error('❌ Error during WebDAV operations:', error);
console.error('Error details:', error.message);
if (error.response) {
console.error('Response status:', error.response.status);
console.error('Response data:', error.response.data);
}
}
} catch (error) {
console.error('❌ Failed to load configuration or create client:', error);
}
}
// Run the debug function
debugMP3Files().catch(console.error);

122
packages/backend/debug-webdav-files.js
View File

@ -1,122 +0,0 @@
import { createClient } from 'webdav';
import fs from 'fs';
async function debugWebDAVFiles() {
try {
// Load configuration
const configData = fs.readFileSync('storage-config.json', 'utf-8');
const config = JSON.parse(configData);
console.log('🔍 WebDAV Configuration:');
console.log('URL:', config.url);
console.log('Username:', config.username);
console.log('Base Path:', config.basePath);
console.log('');
// Create WebDAV client
const client = createClient(config.url, {
username: config.username,
password: config.password,
});
console.log('🔗 Testing connection...');
const basePath = config.basePath || '/Music';
try {
const baseContents = await client.getDirectoryContents(basePath);
console.log('✅ Connection successful');
console.log('Base directory contents count:', Array.isArray(baseContents) ? baseContents.length : 1);
console.log('');
// Test deep listing
console.log('📁 Testing deep directory listing...');
const deepContents = await client.getDirectoryContents(basePath, { deep: true });
console.log('Deep listing response type:', typeof deepContents);
console.log('Is array:', Array.isArray(deepContents));
if (Array.isArray(deepContents)) {
console.log('Total items found:', deepContents.length);
// Count files vs directories
let fileCount = 0;
let dirCount = 0;
const fileExtensions = new Map();
const directories = new Set();
for (const item of deepContents) {
if (item && typeof item === 'object' && 'type' in item) {
if (item.type === 'file') {
fileCount++;
// Track file extensions
const ext = item.basename?.split('.').pop()?.toLowerCase() || 'unknown';
fileExtensions.set(ext, (fileExtensions.get(ext) || 0) + 1);
} else if (item.type === 'directory') {
dirCount++;
const relativePath = item.filename.replace(basePath + '/', '');
if (relativePath) {
directories.add(relativePath);
}
}
}
}
console.log('📊 File Statistics:');
console.log('Files:', fileCount);
console.log('Directories:', dirCount);
console.log('Total items:', fileCount + dirCount);
console.log('');
console.log('📁 Directory structure (first 20):');
const sortedDirs = Array.from(directories).sort();
sortedDirs.slice(0, 20).forEach(dir => {
console.log(' 📁', dir);
});
if (sortedDirs.length > 20) {
console.log(` ... and ${sortedDirs.length - 20} more directories`);
}
console.log('');
console.log('🎵 File extensions:');
const sortedExts = Array.from(fileExtensions.entries()).sort((a, b) => b[1] - a[1]);
sortedExts.forEach(([ext, count]) => {
console.log(` .${ext}: ${count} files`);
});
console.log('');
// Show some sample files
console.log('🎵 Sample files (first 10):');
const sampleFiles = deepContents
.filter(item => item && typeof item === 'object' && 'type' in item && item.type === 'file')
.slice(0, 10);
sampleFiles.forEach((file, index) => {
const relativePath = file.filename.replace(basePath + '/', '');
console.log(` ${index + 1}. ${relativePath} (${file.size} bytes)`);
});
if (fileCount > 10) {
console.log(` ... and ${fileCount - 10} more files`);
}
} else {
console.log('❌ Deep listing returned non-array response:', deepContents);
}
} catch (error) {
console.error('❌ Error during WebDAV operations:', error);
console.error('Error details:', error.message);
if (error.response) {
console.error('Response status:', error.response.status);
console.error('Response data:', error.response.data);
}
}
} catch (error) {
console.error('❌ Failed to load configuration or create client:', error);
}
}
// Run the debug function
debugWebDAVFiles().catch(console.error);

3
packages/backend/package.json
View File

@ -17,8 +17,7 @@
"mongoose": "^8.2.1", "mongoose": "^8.2.1",
"multer": "^2.0.0-rc.3", "multer": "^2.0.0-rc.3",
"music-metadata": "^8.1.0", "music-metadata": "^8.1.0",
"uuid": "^11.1.0", "uuid": "^11.1.0"
"webdav": "^5.3.0"
}, },
"devDependencies": { "devDependencies": {
"@types/cors": "^2.8.17", "@types/cors": "^2.8.17",

4
packages/backend/src/routes/backgroundJobs.ts
View File

@ -10,8 +10,8 @@ router.post('/start', async (req, res) => {
try { try {
const { type, options } = req.body; const { type, options } = req.body;
if (!type || !['storage-sync', 'song-matching'].includes(type)) { if (!type || !['s3-sync', 'song-matching'].includes(type)) {
return res.status(400).json({ error: 'Invalid job type. Must be "storage-sync" or "song-matching"' }); return res.status(400).json({ error: 'Invalid job type. Must be "s3-sync" or "song-matching"' });
} }
console.log(`🚀 Starting background job: ${type}`); console.log(`🚀 Starting background job: ${type}`);

227
packages/backend/src/routes/config.ts
View File

@ -2,16 +2,14 @@ import express from 'express';
import { S3Client, ListBucketsCommand, HeadBucketCommand } from '@aws-sdk/client-s3'; import { S3Client, ListBucketsCommand, HeadBucketCommand } from '@aws-sdk/client-s3';
import fs from 'fs/promises'; import fs from 'fs/promises';
import path from 'path'; import path from 'path';
import { reloadStorageService } from './music.js'; import { reloadS3Service } from './music.js';
import { StorageProviderFactory, StorageConfig } from '../services/storageProvider.js';
const router = express.Router(); const router = express.Router();
// Path to the storage configuration file // Path to the S3 configuration file
const CONFIG_FILE_PATH = path.join(process.cwd(), 'storage-config.json'); const CONFIG_FILE_PATH = path.join(process.cwd(), 's3-config.json');
interface S3Config extends StorageConfig { interface S3Config {
provider: 's3';
endpoint: string; endpoint: string;
region: string; region: string;
accessKeyId: string; accessKeyId: string;
@ -20,56 +18,8 @@ interface S3Config extends StorageConfig {
useSSL: boolean; useSSL: boolean;
} }
interface WebDAVConfig extends StorageConfig {
provider: 'webdav';
url: string;
username: string;
password: string;
basePath?: string;
}
/** /**
* Get current storage configuration * Get current S3 configuration
*/
router.get('/storage', async (req, res) => {
try {
// Check if config file exists
try {
const configData = await fs.readFile(CONFIG_FILE_PATH, 'utf-8');
const config = JSON.parse(configData);
// Don't return sensitive data in the response
const safeConfig = { ...config };
if (config.provider === 's3') {
safeConfig.accessKeyId = config.accessKeyId ? '***' : '';
safeConfig.secretAccessKey = config.secretAccessKey ? '***' : '';
} else if (config.provider === 'webdav') {
safeConfig.password = config.password ? '***' : '';
}
res.json({ config: safeConfig });
} catch (error) {
// Config file doesn't exist, return default config
res.json({
config: {
provider: 's3',
endpoint: process.env.S3_ENDPOINT || '',
region: process.env.S3_REGION || 'us-east-1',
accessKeyId: process.env.S3_ACCESS_KEY_ID ? '***' : '',
secretAccessKey: process.env.S3_SECRET_ACCESS_KEY ? '***' : '',
bucketName: process.env.S3_BUCKET_NAME || '',
useSSL: process.env.S3_USE_SSL !== 'false',
}
});
}
} catch (error) {
console.error('Error loading storage config:', error);
res.status(500).json({ error: 'Failed to load storage configuration' });
}
});
/**
* Get current S3 configuration (legacy endpoint)
*/ */
router.get('/s3', async (req, res) => { router.get('/s3', async (req, res) => {
try { try {
@ -106,88 +56,7 @@ router.get('/s3', async (req, res) => {
}); });
/** /**
* Save storage configuration * Save S3 configuration
*/
router.post('/storage', async (req, res) => {
try {
const newConfig: StorageConfig = req.body;
// Load existing configuration to merge with
let existingConfig: StorageConfig | null = null;
try {
const existingData = await fs.readFile(CONFIG_FILE_PATH, 'utf-8');
existingConfig = JSON.parse(existingData);
} catch (error) {
// No existing config file, that's okay
}
// Merge with existing configuration to preserve passwords/credentials that weren't provided
const mergedConfig = { ...existingConfig };
// Only override fields that are actually provided (not empty strings)
Object.keys(newConfig).forEach(key => {
const value = newConfig[key];
if (value !== '' && value !== null && value !== undefined) {
mergedConfig[key] = value;
}
});
// Validate required fields based on provider
if (mergedConfig.provider === 's3') {
const s3Config = mergedConfig as S3Config;
if (!s3Config.endpoint || !s3Config.accessKeyId || !s3Config.secretAccessKey || !s3Config.bucketName) {
return res.status(400).json({
error: 'Missing required S3 fields: endpoint, accessKeyId, secretAccessKey, bucketName'
});
}
} else if (mergedConfig.provider === 'webdav') {
const webdavConfig = mergedConfig as WebDAVConfig;
if (!webdavConfig.url || !webdavConfig.username || !webdavConfig.password) {
return res.status(400).json({
error: 'Missing required WebDAV fields: url, username, password'
});
}
} else {
return res.status(400).json({
error: 'Invalid provider. Must be "s3" or "webdav"'
});
}
// Save configuration to file
await fs.writeFile(CONFIG_FILE_PATH, JSON.stringify(mergedConfig, null, 2));
// Update environment variables for current session
if (mergedConfig.provider === 's3') {
const s3Config = mergedConfig as S3Config;
process.env.S3_ENDPOINT = s3Config.endpoint;
process.env.S3_REGION = s3Config.region;
process.env.S3_ACCESS_KEY_ID = s3Config.accessKeyId;
process.env.S3_SECRET_ACCESS_KEY = s3Config.secretAccessKey;
process.env.S3_BUCKET_NAME = s3Config.bucketName;
process.env.S3_USE_SSL = s3Config.useSSL.toString();
} else if (mergedConfig.provider === 'webdav') {
const webdavConfig = mergedConfig as WebDAVConfig;
process.env.WEBDAV_URL = webdavConfig.url;
process.env.WEBDAV_USERNAME = webdavConfig.username;
process.env.WEBDAV_PASSWORD = webdavConfig.password;
process.env.WEBDAV_BASE_PATH = webdavConfig.basePath || '/music-files';
}
// Reload storage service with new configuration
const reloadSuccess = await reloadStorageService();
res.json({
message: `${mergedConfig.provider.toUpperCase()} configuration saved successfully`,
storageServiceReloaded: reloadSuccess
});
} catch (error) {
console.error('Error saving storage config:', error);
res.status(500).json({ error: 'Failed to save storage configuration' });
}
});
/**
* Save S3 configuration (legacy endpoint)
*/ */
router.post('/s3', async (req, res) => { router.post('/s3', async (req, res) => {
try { try {
@ -211,8 +80,8 @@ router.post('/s3', async (req, res) => {
process.env.S3_BUCKET_NAME = config.bucketName; process.env.S3_BUCKET_NAME = config.bucketName;
process.env.S3_USE_SSL = config.useSSL.toString(); process.env.S3_USE_SSL = config.useSSL.toString();
// Reload storage service with new configuration // Reload S3 service with new configuration
const reloadSuccess = await reloadStorageService(); const reloadSuccess = await reloadS3Service();
res.json({ res.json({
message: 'S3 configuration saved successfully', message: 'S3 configuration saved successfully',
@ -225,83 +94,7 @@ router.post('/s3', async (req, res) => {
}); });
/** /**
* Test storage connection * Test S3 connection
*/
router.post('/storage/test', async (req, res) => {
try {
const newConfig: StorageConfig = req.body;
// Load existing configuration to merge with
let existingConfig: StorageConfig | null = null;
try {
const existingData = await fs.readFile(CONFIG_FILE_PATH, 'utf-8');
existingConfig = JSON.parse(existingData);
} catch (error) {
// No existing config file, that's okay
}
// Merge with existing configuration to preserve passwords/credentials that weren't provided
const config = { ...existingConfig };
// Only override fields that are actually provided (not empty strings)
Object.keys(newConfig).forEach(key => {
const value = newConfig[key];
if (value !== '' && value !== null && value !== undefined) {
config[key] = value;
}
});
// Validate required fields based on provider
if (config.provider === 's3') {
const s3Config = config as S3Config;
if (!s3Config.endpoint || !s3Config.accessKeyId || !s3Config.secretAccessKey || !s3Config.bucketName) {
return res.status(400).json({
error: 'Missing required S3 fields: endpoint, accessKeyId, secretAccessKey, bucketName'
});
}
} else if (config.provider === 'webdav') {
const webdavConfig = config as WebDAVConfig;
if (!webdavConfig.url || !webdavConfig.username || !webdavConfig.password) {
return res.status(400).json({
error: 'Missing required WebDAV fields: url, username, password'
});
}
} else {
return res.status(400).json({
error: 'Invalid provider. Must be "s3" or "webdav"'
});
}
// Create storage provider and test connection
const provider = await StorageProviderFactory.createProvider(config);
const connectionTest = await provider.testConnection();
if (connectionTest) {
res.json({
success: true,
message: `${config.provider.toUpperCase()} connection test successful`,
provider: config.provider
});
} else {
res.status(400).json({
success: false,
error: `${config.provider.toUpperCase()} connection test failed`,
provider: config.provider
});
}
} catch (error) {
console.error('Error testing storage connection:', error);
res.status(500).json({
error: 'Failed to test storage connection',
details: error instanceof Error ? error.message : 'Unknown error'
});
}
});
/**
* Test S3 connection (legacy endpoint)
*/ */
router.post('/s3/test', async (req, res) => { router.post('/s3/test', async (req, res) => {
try { try {
@ -377,6 +170,4 @@ router.post('/s3/test', async (req, res) => {
} }
}); });
export { router as configRouter }; export { router as configRouter };

141
packages/backend/src/routes/music.ts
View File

@ -1,6 +1,6 @@
import express from 'express'; import express from 'express';
import multer from 'multer'; import multer from 'multer';
import { StorageProviderFactory, StorageProvider } from '../services/storageProvider.js'; import { S3Service } from '../services/s3Service.js';
import { AudioMetadataService } from '../services/audioMetadataService.js'; import { AudioMetadataService } from '../services/audioMetadataService.js';
import { MusicFile } from '../models/MusicFile.js'; import { MusicFile } from '../models/MusicFile.js';
import { Song } from '../models/Song.js'; import { Song } from '../models/Song.js';
@ -24,43 +24,40 @@ const upload = multer({
}); });
// Initialize services // Initialize services
let storageService: StorageProvider; let s3Service: S3Service;
// Initialize storage service with configuration from file // Initialize S3 service with configuration from file
async function initializeStorageService() { async function initializeS3Service() {
try { try {
const config = await StorageProviderFactory.loadConfig(); s3Service = await S3Service.createFromConfig();
storageService = await StorageProviderFactory.createProvider(config); console.log('✅ S3 service initialized with configuration from s3-config.json');
console.log(`✅ Storage service initialized (${config.provider}) with configuration from storage-config.json`);
} catch (error) { } catch (error) {
console.error('❌ Failed to initialize storage service:', error); console.error('❌ Failed to initialize S3 service:', error);
// Fallback to S3 with environment variables // Fallback to environment variables
storageService = await StorageProviderFactory.createProvider({ s3Service = new S3Service({
provider: 's3',
endpoint: process.env.S3_ENDPOINT || 'http://localhost:9000', endpoint: process.env.S3_ENDPOINT || 'http://localhost:9000',
accessKeyId: process.env.S3_ACCESS_KEY_ID || 'minioadmin', accessKeyId: process.env.S3_ACCESS_KEY_ID || 'minioadmin',
secretAccessKey: process.env.S3_SECRET_ACCESS_KEY || 'minioadmin', secretAccessKey: process.env.S3_SECRET_ACCESS_KEY || 'minioadmin',
bucketName: process.env.S3_BUCKET_NAME || 'music-files', bucketName: process.env.S3_BUCKET_NAME || 'music-files',
region: process.env.S3_REGION || 'us-east-1', region: process.env.S3_REGION || 'us-east-1',
}); });
console.log('⚠️ Storage service initialized with S3 environment variables as fallback'); console.log('⚠️ S3 service initialized with environment variables as fallback');
} }
} }
// Initialize storage service on startup // Initialize S3 service on startup
initializeStorageService(); initializeS3Service();
/** /**
* Reload storage service with updated configuration * Reload S3 service with updated configuration
*/ */
export async function reloadStorageService() { export async function reloadS3Service() {
try { try {
const config = await StorageProviderFactory.loadConfig(); s3Service = await S3Service.createFromConfig();
storageService = await StorageProviderFactory.createProvider(config); console.log('✅ S3 service reloaded with updated configuration');
console.log(`✅ Storage service reloaded (${config.provider}) with updated configuration`);
return true; return true;
} catch (error) { } catch (error) {
console.error('❌ Failed to reload storage service:', error); console.error('❌ Failed to reload S3 service:', error);
return false; return false;
} }
} }
@ -80,8 +77,8 @@ router.post('/upload', upload.single('file'), async (req, res) => {
const targetFolder = typeof req.body?.targetFolder === 'string' ? req.body.targetFolder : undefined; const targetFolder = typeof req.body?.targetFolder === 'string' ? req.body.targetFolder : undefined;
const markForScan = String(req.body?.markForScan || '').toLowerCase() === 'true'; const markForScan = String(req.body?.markForScan || '').toLowerCase() === 'true';
// Upload to storage // Upload to S3
const uploadResult = await storageService.uploadFile(buffer, originalname, mimetype, targetFolder); const uploadResult = await s3Service.uploadFile(buffer, originalname, mimetype, targetFolder);
// Extract audio metadata // Extract audio metadata
const metadata = await audioMetadataService.extractMetadata(buffer, originalname); const metadata = await audioMetadataService.extractMetadata(buffer, originalname);
@ -180,8 +177,8 @@ router.post('/batch-upload', upload.array('files', 10), async (req, res) => {
try { try {
const { buffer, originalname, mimetype } = file; const { buffer, originalname, mimetype } = file;
// Upload to storage // Upload to S3
const uploadResult = await storageService.uploadFile(buffer, originalname, mimetype); const uploadResult = await s3Service.uploadFile(buffer, originalname, mimetype);
// Extract audio metadata // Extract audio metadata
const metadata = await audioMetadataService.extractMetadata(buffer, originalname); const metadata = await audioMetadataService.extractMetadata(buffer, originalname);
@ -301,7 +298,7 @@ router.get('/folders', async (req, res) => {
return res.json({ folders: folderCache.folders }); return res.json({ folders: folderCache.folders });
} }
const folders = await storageService.listAllFolders(''); const folders = await s3Service.listAllFolders('');
const result = ['', ...folders]; const result = ['', ...folders];
// Cache the result // Cache the result
@ -342,12 +339,12 @@ router.post('/sync-s3', async (req, res) => {
// Start the background job // Start the background job
const jobId = await backgroundJobService.startJob({ const jobId = await backgroundJobService.startJob({
type: 'storage-sync', type: 's3-sync',
options: req.body options: req.body
}); });
res.json({ res.json({
message: 'Storage sync started as background job', message: 'S3 sync started as background job',
jobId, jobId,
status: 'started' status: 'started'
}); });
@ -371,75 +368,20 @@ router.get('/:id/stream', async (req, res) => {
return res.status(404).json({ error: 'Music file not found' }); return res.status(404).json({ error: 'Music file not found' });
} }
// For WebDAV, use a proxy endpoint to handle authentication // Use presigned URL for secure access instead of direct URL
// For S3, use presigned URL for direct access const presignedUrl = await s3Service.getPresignedUrl(musicFile.s3Key, 3600); // 1 hour expiry
const config = await StorageProviderFactory.loadConfig();
if (config.provider === 'webdav') {
// Use proxy endpoint for WebDAV to handle authentication
const proxyUrl = `${req.protocol}://${req.get('host')}/api/music/${musicFile._id}/proxy`;
res.json({
streamingUrl: proxyUrl,
musicFile,
contentType: musicFile.contentType || undefined,
});
} else {
// Use presigned URL for S3
const presignedUrl = await storageService.getPresignedUrl(musicFile.s3Key, 3600); // 1 hour expiry
res.json({ res.json({
streamingUrl: presignedUrl, streamingUrl: presignedUrl,
musicFile, musicFile,
contentType: musicFile.contentType || undefined, contentType: musicFile.contentType || undefined,
}); });
}
} catch (error) { } catch (error) {
console.error('Streaming error:', error); console.error('Streaming error:', error);
res.status(500).json({ error: 'Failed to get streaming URL' }); res.status(500).json({ error: 'Failed to get streaming URL' });
} }
}); });
/**
* Proxy endpoint for WebDAV streaming with authentication
*/
router.get('/:id/proxy', async (req, res) => {
try {
const musicFile = await MusicFile.findById(req.params.id);
if (!musicFile) {
return res.status(404).json({ error: 'Music file not found' });
}
// Set appropriate headers for audio streaming
res.setHeader('Content-Type', musicFile.contentType || 'audio/mpeg');
res.setHeader('Accept-Ranges', 'bytes');
res.setHeader('Cache-Control', 'public, max-age=3600');
// Get the file content from WebDAV
const fileBuffer = await storageService.getFileContent(musicFile.s3Key);
// Handle range requests for seeking
const range = req.headers.range;
if (range) {
const fileSize = fileBuffer.length;
const parts = range.replace(/bytes=/, "").split("-");
const start = parseInt(parts[0], 10);
const end = parts[1] ? parseInt(parts[1], 10) : fileSize - 1;
const chunksize = (end - start) + 1;
const chunk = fileBuffer.slice(start, end + 1);
res.status(206);
res.setHeader('Content-Range', `bytes ${start}-${end}/${fileSize}`);
res.setHeader('Content-Length', chunksize.toString());
res.end(chunk);
} else {
// No range request, send entire file
res.setHeader('Content-Length', fileBuffer.length.toString());
res.end(fileBuffer);
}
} catch (error) {
console.error('Proxy streaming error:', error);
res.status(500).json({ error: 'Failed to stream music file' });
}
});
/** /**
* Get presigned URL for secure access * Get presigned URL for secure access
*/ */
@ -451,7 +393,7 @@ router.get('/:id/presigned', async (req, res) => {
} }
const expiresIn = parseInt(req.query.expiresIn as string) || 3600; const expiresIn = parseInt(req.query.expiresIn as string) || 3600;
const presignedUrl = await storageService.getPresignedUrl(musicFile.s3Key, expiresIn); const presignedUrl = await s3Service.getPresignedUrl(musicFile.s3Key, expiresIn);
res.json({ res.json({
presignedUrl, presignedUrl,
@ -534,8 +476,31 @@ router.get('/', async (req, res) => {
} }
}); });
// DELETE endpoint removed to keep WebDAV integration read-only /**
// Music files cannot be deleted to prevent accidental data loss from WebDAV * Delete a music file
*/
router.delete('/:id', async (req, res) => {
try {
const musicFile = await MusicFile.findById(req.params.id);
if (!musicFile) {
return res.status(404).json({ error: 'Music file not found' });
}
// Delete from S3
await s3Service.deleteFile(musicFile.s3Key);
// Delete from database
await MusicFile.findByIdAndDelete(req.params.id);
// Invalidate folder cache since we removed a file
invalidateFolderCache();
res.json({ message: 'Music file deleted successfully' });
} catch (error) {
console.error('Delete error:', error);
res.status(500).json({ error: 'Failed to delete music file' });
}
});
/** /**
* Link music file to existing song * Link music file to existing song

5
packages/backend/src/services/audioMetadataService.ts
View File

@ -37,9 +37,6 @@ export class AudioMetadataService {
'wma': 'WMA', 'wma': 'WMA',
'OPUS': 'OPUS', 'OPUS': 'OPUS',
'opus': 'OPUS', 'opus': 'OPUS',
'AIFF': 'AIFF',
'aiff': 'AIFF',
'aif': 'AIFF',
}; };
// Try to map the container format // Try to map the container format
@ -207,7 +204,7 @@ export class AudioMetadataService {
*/ */
isAudioFile(fileName: string): boolean { isAudioFile(fileName: string): boolean {
const supportedFormats = [ const supportedFormats = [
'mp3', 'wav', 'flac', 'aac', 'ogg', 'm4a', 'wma', 'opus', 'aif', 'aiff' 'mp3', 'wav', 'flac', 'aac', 'ogg', 'm4a', 'wma', 'opus'
]; ];
const extension = fileName.split('.').pop()?.toLowerCase(); const extension = fileName.split('.').pop()?.toLowerCase();

92
packages/backend/src/services/backgroundJobService.ts
View File

@ -1,6 +1,6 @@
export interface JobProgress { export interface JobProgress {
jobId: string; jobId: string;
type: 'storage-sync' | 'song-matching'; type: 's3-sync' | 'song-matching';
status: 'running' | 'completed' | 'failed'; status: 'running' | 'completed' | 'failed';
progress: number; // 0-100 progress: number; // 0-100
current: number; current: number;
@ -13,7 +13,7 @@ export interface JobProgress {
} }
export interface JobOptions { export interface JobOptions {
type: 'storage-sync' | 'song-matching'; type: 's3-sync' | 'song-matching';
options?: any; options?: any;
} }
@ -140,8 +140,8 @@ class BackgroundJobService {
private async runJob(jobId: string, jobOptions: JobOptions): Promise<void> { private async runJob(jobId: string, jobOptions: JobOptions): Promise<void> {
try { try {
switch (jobOptions.type) { switch (jobOptions.type) {
case 'storage-sync': case 's3-sync':
await this.runStorageSyncJob(jobId, jobOptions.options); await this.runS3SyncJob(jobId, jobOptions.options);
break; break;
case 'song-matching': case 'song-matching':
await this.runSongMatchingJob(jobId, jobOptions.options); await this.runSongMatchingJob(jobId, jobOptions.options);
@ -156,20 +156,18 @@ class BackgroundJobService {
} }
/** /**
* Run storage sync job (works with any storage provider) * Run S3 sync job
*/ */
private async runStorageSyncJob(jobId: string, options?: any): Promise<void> { private async runS3SyncJob(jobId: string, options?: any): Promise<void> {
try { try {
// Import here to avoid circular dependencies // Import here to avoid circular dependencies
const { StorageProviderFactory } = await import('./storageProvider.js'); const { S3Service } = await import('./s3Service.js');
const { AudioMetadataService } = await import('./audioMetadataService.js'); const { AudioMetadataService } = await import('./audioMetadataService.js');
const { SongMatchingService } = await import('./songMatchingService.js'); const { SongMatchingService } = await import('./songMatchingService.js');
const { MusicFile } = await import('../models/MusicFile.js'); const { MusicFile } = await import('../models/MusicFile.js');
const { Song } = await import('../models/Song.js'); const { Song } = await import('../models/Song.js');
// Get the configured storage provider const s3Service = await S3Service.createFromConfig();
const config = await StorageProviderFactory.loadConfig();
const storageService = await StorageProviderFactory.createProvider(config);
const audioMetadataService = new AudioMetadataService(); const audioMetadataService = new AudioMetadataService();
const songMatchingService = new SongMatchingService(); const songMatchingService = new SongMatchingService();
@ -185,8 +183,6 @@ class BackgroundJobService {
case 'ogg': return 'audio/ogg'; case 'ogg': return 'audio/ogg';
case 'opus': return 'audio/opus'; case 'opus': return 'audio/opus';
case 'wma': return 'audio/x-ms-wma'; case 'wma': return 'audio/x-ms-wma';
case 'aif': return 'audio/aiff';
case 'aiff': return 'audio/aiff';
default: return 'application/octet-stream'; default: return 'application/octet-stream';
} }
}; };
@ -217,14 +213,14 @@ class BackgroundJobService {
// Phase 1: Quick filename matching // Phase 1: Quick filename matching
this.updateProgress(jobId, { this.updateProgress(jobId, {
message: `Phase 1: Fetching files from ${config.provider.toUpperCase()}...`, message: 'Phase 1: Fetching files from S3...',
current: 0, current: 0,
total: 0 total: 0
}); });
const storageFiles = await storageService.listAllFiles(); const s3Files = await s3Service.listAllFiles();
const audioFiles = storageFiles.filter(storageFile => { const audioFiles = s3Files.filter(s3File => {
const filename = storageFile.key.split('/').pop() || storageFile.key; const filename = s3File.key.split('/').pop() || s3File.key;
return audioMetadataService.isAudioFile(filename); return audioMetadataService.isAudioFile(filename);
}); });
@ -236,8 +232,8 @@ class BackgroundJobService {
// Get existing files // Get existing files
const existingFiles = await MusicFile.find({}, { s3Key: 1 }); const existingFiles = await MusicFile.find({}, { s3Key: 1 });
const existingStorageKeys = new Set(existingFiles.map(f => f.s3Key)); const existingS3Keys = new Set(existingFiles.map(f => f.s3Key));
const newAudioFiles = options?.force ? audioFiles : audioFiles.filter(storageFile => !existingStorageKeys.has(storageFile.key)); const newAudioFiles = options?.force ? audioFiles : audioFiles.filter(s3File => !existingS3Keys.has(s3File.key));
this.updateProgress(jobId, { this.updateProgress(jobId, {
message: `Phase 1: Processing ${newAudioFiles.length} new audio files...`, message: `Phase 1: Processing ${newAudioFiles.length} new audio files...`,
@ -253,11 +249,11 @@ class BackgroundJobService {
let processedCount = 0; let processedCount = 0;
let phase1Errors = 0; let phase1Errors = 0;
for (const storageFile of newAudioFiles) { for (const s3File of newAudioFiles) {
processedCount++; processedCount++;
try { try {
const filename = storageFile.key.split('/').pop() || storageFile.key; const filename = s3File.key.split('/').pop() || s3File.key;
this.updateProgress(jobId, { this.updateProgress(jobId, {
message: `Phase 1: Quick filename matching`, message: `Phase 1: Quick filename matching`,
@ -269,7 +265,7 @@ class BackgroundJobService {
// Decode URL-encoded sequences so %20, %27 etc. are compared correctly // Decode URL-encoded sequences so %20, %27 etc. are compared correctly
const safeDecode = (s: string): string => { try { return decodeURIComponent(s); } catch { return s; } }; const safeDecode = (s: string): string => { try { return decodeURIComponent(s); } catch { return s; } };
const stripDiacritics = (s: string) => s.normalize('NFKD').replace(/[\u0300-\u036f]/g, ''); const stripDiacritics = (s: string) => s.normalize('NFKD').replace(/[\u0300-\u036f]/g, '');
const normalizedStorageFilename = stripDiacritics(safeDecode(filename)).replace(/\.[^/.]+$/, '').toLowerCase(); const normalizedS3Filename = stripDiacritics(safeDecode(filename)).replace(/\.[^/.]+$/, '').toLowerCase();
let matchedSong = null; let matchedSong = null;
for (const song of allSongs) { for (const song of allSongs) {
@ -277,7 +273,7 @@ class BackgroundJobService {
const rekordboxFilename = song.location.split(/[/\\]/).pop() || song.location; const rekordboxFilename = song.location.split(/[/\\]/).pop() || song.location;
const normalizedRekordboxFilename = stripDiacritics(safeDecode(rekordboxFilename)).replace(/\.[^/.]+$/, '').toLowerCase(); const normalizedRekordboxFilename = stripDiacritics(safeDecode(rekordboxFilename)).replace(/\.[^/.]+$/, '').toLowerCase();
if (normalizedStorageFilename === normalizedRekordboxFilename) { if (normalizedS3Filename === normalizedRekordboxFilename) {
matchedSong = song; matchedSong = song;
break; break;
} }
@ -287,31 +283,30 @@ class BackgroundJobService {
if (matchedSong) { if (matchedSong) {
const basicMetadata = audioMetadataService.extractBasicMetadataFromFilename(filename); const basicMetadata = audioMetadataService.extractBasicMetadataFromFilename(filename);
// Reuse existing MusicFile if present (force mode), otherwise create new // Reuse existing MusicFile if present (force mode), otherwise create new
let musicFile = await MusicFile.findOne({ s3Key: storageFile.key }); let musicFile = await MusicFile.findOne({ s3Key: s3File.key });
if (!musicFile) { if (!musicFile) {
musicFile = new MusicFile({ s3Key: storageFile.key }); musicFile = new MusicFile({ s3Key: s3File.key });
} }
musicFile.originalName = filename; musicFile.originalName = filename;
musicFile.s3Key = storageFile.key; musicFile.s3Key = s3File.key;
musicFile.s3Url = await storageService.getPresignedUrl(storageFile.key); musicFile.s3Url = `${process.env.S3_ENDPOINT}/${process.env.S3_BUCKET_NAME}/${s3File.key}`;
musicFile.contentType = guessContentType(filename); musicFile.contentType = guessContentType(filename);
musicFile.size = storageFile.size; musicFile.size = s3File.size;
Object.assign(musicFile, basicMetadata); Object.assign(musicFile, basicMetadata);
musicFile.songId = matchedSong._id; musicFile.songId = matchedSong._id;
await musicFile.save(); await musicFile.save();
quickMatches.push(musicFile); quickMatches.push(musicFile);
// Update the Song document to indicate it has a storage file // Update the Song document to indicate it has an S3 file
const storageUrl = await storageService.getPresignedUrl(storageFile.key);
await Song.updateOne( await Song.updateOne(
{ _id: matchedSong._id }, { _id: matchedSong._id },
{ {
$set: { $set: {
's3File.musicFileId': musicFile._id, 's3File.musicFileId': musicFile._id,
's3File.s3Key': storageFile.key, 's3File.s3Key': s3File.key,
's3File.s3Url': storageUrl, 's3File.s3Url': `${process.env.S3_ENDPOINT}/${process.env.S3_BUCKET_NAME}/${s3File.key}`,
's3File.streamingUrl': storageUrl, 's3File.streamingUrl': `${process.env.S3_ENDPOINT}/${process.env.S3_BUCKET_NAME}/${s3File.key}`,
's3File.hasS3File': true 's3File.hasS3File': true
} }
} }
@ -319,14 +314,14 @@ class BackgroundJobService {
console.log(`✅ Quick match saved immediately: ${filename}`); console.log(`✅ Quick match saved immediately: ${filename}`);
} else { } else {
unmatchedFiles.push(storageFile); unmatchedFiles.push(s3File);
} }
} catch (error) { } catch (error) {
console.error(`Error in quick matching ${storageFile.key}:`, error); console.error(`Error in quick matching ${s3File.key}:`, error);
unmatchedFiles.push(storageFile); unmatchedFiles.push(s3File);
phase1Errors++; phase1Errors++;
} }
} }
@ -351,10 +346,10 @@ class BackgroundJobService {
const processedFiles: any[] = []; const processedFiles: any[] = [];
for (let i = 0; i < unmatchedFiles.length; i++) { for (let i = 0; i < unmatchedFiles.length; i++) {
const storageFile = unmatchedFiles[i]; const s3File = unmatchedFiles[i];
try { try {
const filename = storageFile.key.split('/').pop() || storageFile.key; const filename = s3File.key.split('/').pop() || s3File.key;
this.updateProgress(jobId, { this.updateProgress(jobId, {
message: `Phase 2: Complex matching`, message: `Phase 2: Complex matching`,
@ -363,19 +358,19 @@ class BackgroundJobService {
}); });
// Download file and extract metadata // Download file and extract metadata
const fileBuffer = await storageService.getFileContent(storageFile.key); const fileBuffer = await s3Service.getFileContent(s3File.key);
const metadata = await audioMetadataService.extractMetadata(fileBuffer, filename); const metadata = await audioMetadataService.extractMetadata(fileBuffer, filename);
// Reuse existing MusicFile document if present to avoid duplicate key errors // Reuse existing MusicFile document if present to avoid duplicate key errors
let musicFile = await MusicFile.findOne({ s3Key: storageFile.key }); let musicFile = await MusicFile.findOne({ s3Key: s3File.key });
if (!musicFile) { if (!musicFile) {
musicFile = new MusicFile({ s3Key: storageFile.key }); musicFile = new MusicFile({ s3Key: s3File.key });
} }
musicFile.originalName = filename; musicFile.originalName = filename;
musicFile.s3Key = storageFile.key; musicFile.s3Key = s3File.key;
musicFile.s3Url = await storageService.getPresignedUrl(storageFile.key); musicFile.s3Url = `${process.env.S3_ENDPOINT}/${process.env.S3_BUCKET_NAME}/${s3File.key}`;
musicFile.contentType = guessContentType(filename); musicFile.contentType = guessContentType(filename);
musicFile.size = storageFile.size; musicFile.size = s3File.size;
Object.assign(musicFile, metadata); Object.assign(musicFile, metadata);
// Try complex matching // Try complex matching
@ -391,16 +386,15 @@ class BackgroundJobService {
musicFile.songId = bestMatch.song._id; musicFile.songId = bestMatch.song._id;
complexMatches++; complexMatches++;
// Update the Song document to indicate it has a storage file // Update the Song document to indicate it has an S3 file
const storageUrl = await storageService.getPresignedUrl(storageFile.key);
await Song.updateOne( await Song.updateOne(
{ _id: bestMatch.song._id }, { _id: bestMatch.song._id },
{ {
$set: { $set: {
's3File.musicFileId': musicFile._id, 's3File.musicFileId': musicFile._id,
's3File.s3Key': storageFile.key, 's3File.s3Key': s3File.key,
's3File.s3Url': storageUrl, 's3File.s3Url': `${process.env.S3_ENDPOINT}/${process.env.S3_BUCKET_NAME}/${s3File.key}`,
's3File.streamingUrl': storageUrl, 's3File.streamingUrl': `${process.env.S3_ENDPOINT}/${process.env.S3_BUCKET_NAME}/${s3File.key}`,
's3File.hasS3File': true 's3File.hasS3File': true
} }
} }
@ -418,7 +412,7 @@ class BackgroundJobService {
} catch (error) { } catch (error) {
console.error(`Error processing ${storageFile.key}:`, error); console.error(`Error processing ${s3File.key}:`, error);
stillUnmatched++; stillUnmatched++;
phase2Errors++; phase2Errors++;
} }

120
packages/backend/src/services/s3Service.ts
View File

@ -3,10 +3,8 @@ import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { v4 as uuidv4 } from 'uuid'; import { v4 as uuidv4 } from 'uuid';
import fs from 'fs/promises'; import fs from 'fs/promises';
import path from 'path'; import path from 'path';
import { StorageProvider, StorageConfig, UploadResult, FileInfo } from './storageProvider.js';
export interface S3Config extends StorageConfig { export interface S3Config {
provider: 's3';
endpoint: string; endpoint: string;
accessKeyId: string; accessKeyId: string;
secretAccessKey: string; secretAccessKey: string;
@ -15,7 +13,21 @@ export interface S3Config extends StorageConfig {
useSSL?: boolean; useSSL?: boolean;
} }
export class S3Service implements StorageProvider { export interface UploadResult {
key: string;
url: string;
size: number;
contentType: string;
}
export interface S3FileInfo {
key: string;
size: number;
lastModified: Date;
contentType?: string;
}
export class S3Service {
private client: S3Client; private client: S3Client;
private bucketName: string; private bucketName: string;
@ -43,7 +55,6 @@ export class S3Service implements StorageProvider {
} catch (error) { } catch (error) {
console.warn('Failed to load s3-config.json, using environment variables as fallback'); console.warn('Failed to load s3-config.json, using environment variables as fallback');
return { return {
provider: 's3' as const,
endpoint: process.env.S3_ENDPOINT || 'http://localhost:9000', endpoint: process.env.S3_ENDPOINT || 'http://localhost:9000',
accessKeyId: process.env.S3_ACCESS_KEY_ID || 'minioadmin', accessKeyId: process.env.S3_ACCESS_KEY_ID || 'minioadmin',
secretAccessKey: process.env.S3_SECRET_ACCESS_KEY || 'minioadmin', secretAccessKey: process.env.S3_SECRET_ACCESS_KEY || 'minioadmin',
@ -71,22 +82,16 @@ export class S3Service implements StorageProvider {
contentType: string, contentType: string,
targetFolder?: string targetFolder?: string
): Promise<UploadResult> { ): Promise<UploadResult> {
// Sanitize filename to be safe for S3 const fileExtension = originalName.split('.').pop();
const sanitizedFilename = this.sanitizeFilename(originalName);
const cleaned = typeof targetFolder === 'string' ? targetFolder.replace(/^\/+|\/+$/g, '') : ''; const cleaned = typeof targetFolder === 'string' ? targetFolder.replace(/^\/+|\/+$/g, '') : '';
const safeFolder = cleaned; const safeFolder = cleaned;
// Use original filename instead of UUID
const key = safeFolder const key = safeFolder
? `${safeFolder}/${sanitizedFilename}` ? `${safeFolder}/${uuidv4()}.${fileExtension}`
: sanitizedFilename; : `${uuidv4()}.${fileExtension}`;
// Check if file already exists and handle conflicts
const finalKey = await this.handleFilenameConflict(key);
const command = new PutObjectCommand({ const command = new PutObjectCommand({
Bucket: this.bucketName, Bucket: this.bucketName,
Key: finalKey, Key: key,
Body: file, Body: file,
ContentType: contentType, ContentType: contentType,
Metadata: { Metadata: {
@ -98,8 +103,8 @@ export class S3Service implements StorageProvider {
await this.client.send(command); await this.client.send(command);
return { return {
key: finalKey, key,
url: `${this.bucketName}/${finalKey}`, url: `${this.bucketName}/${key}`,
size: file.length, size: file.length,
contentType, contentType,
}; };
@ -108,8 +113,8 @@ export class S3Service implements StorageProvider {
/** /**
* Recursively list all files in the S3 bucket * Recursively list all files in the S3 bucket
*/ */
async listAllFiles(prefix: string = ''): Promise<FileInfo[]> { async listAllFiles(prefix: string = ''): Promise<S3FileInfo[]> {
const files: FileInfo[] = []; const files: S3FileInfo[] = [];
let continuationToken: string | undefined; let continuationToken: string | undefined;
do { do {
@ -191,7 +196,12 @@ export class S3Service implements StorageProvider {
* Delete a file from S3 * Delete a file from S3
*/ */
async deleteFile(key: string): Promise<void> { async deleteFile(key: string): Promise<void> {
throw new Error('File deletion is disabled to prevent accidental data loss'); const command = new DeleteObjectCommand({
Bucket: this.bucketName,
Key: key,
});
await this.client.send(command);
} }
/** /**
@ -253,74 +263,4 @@ export class S3Service implements StorageProvider {
async getStreamingUrl(key: string): Promise<string> { async getStreamingUrl(key: string): Promise<string> {
return `${process.env.S3_ENDPOINT}/${this.bucketName}/${key}`; return `${process.env.S3_ENDPOINT}/${this.bucketName}/${key}`;
} }
/**
* Test the connection to S3
*/
async testConnection(): Promise<boolean> {
try {
// Try to list objects with a limit of 1 to test the connection
const command = new ListObjectsV2Command({
Bucket: this.bucketName,
MaxKeys: 1,
});
await this.client.send(command);
return true;
} catch (error) {
console.error('S3 connection test failed:', error);
return false;
}
}
/**
* Sanitize filename to be safe for S3
*/
private sanitizeFilename(filename: string): string {
// Remove or replace characters that might cause issues in S3
return filename
.replace(/[<>:"|?*]/g, '_') // Replace problematic characters
.replace(/[\x00-\x1f\x80-\x9f]/g, '') // Remove control characters
.replace(/\s+/g, ' ') // Normalize whitespace
.trim();
}
/**
* Handle filename conflicts by adding a number suffix
*/
private async handleFilenameConflict(key: string): Promise<string> {
try {
// Check if file exists
const command = new HeadObjectCommand({
Bucket: this.bucketName,
Key: key,
});
await this.client.send(command);
// File exists, generate a new name with number suffix
const pathParts = key.split('/');
const filename = pathParts.pop() || '';
const dir = pathParts.join('/');
const nameWithoutExt = filename.substring(0, filename.lastIndexOf('.'));
const extension = filename.substring(filename.lastIndexOf('.'));
let counter = 1;
let newKey: string;
do {
const newFilename = `${nameWithoutExt} (${counter})${extension}`;
newKey = dir ? `${dir}/${newFilename}` : newFilename;
counter++;
// Prevent infinite loop
if (counter > 1000) {
throw new Error('Too many filename conflicts');
}
} while (await this.fileExists(newKey));
return newKey;
} catch (error) {
// File doesn't exist, use original key
return key;
}
}
} }

143
packages/backend/src/services/storageProvider.ts
View File

@ -1,143 +0,0 @@
/**
* Storage Provider Interface
*
* This interface defines the contract that all storage providers (S3, WebDAV, etc.)
* must implement to ensure consistent behavior across different storage backends.
*/
export interface StorageConfig {
provider: 's3' | 'webdav';
[key: string]: any; // Allow additional provider-specific config
}
export interface UploadResult {
key: string;
url: string;
size: number;
contentType: string;
}
export interface FileInfo {
key: string;
size: number;
lastModified: Date;
contentType?: string;
}
export interface StorageProvider {
/**
* Upload a file to storage
*/
uploadFile(
file: Buffer,
originalName: string,
contentType: string,
targetFolder?: string
): Promise<UploadResult>;
/**
* List all files in storage
*/
listAllFiles(prefix?: string): Promise<FileInfo[]>;
/**
* List all folders in storage
*/
listAllFolders(prefix?: string): Promise<string[]>;
/**
* Generate a presigned/secure URL for file access
*/
getPresignedUrl(key: string, expiresIn?: number): Promise<string>;
/**
* Delete a file from storage
*/
deleteFile(key: string): Promise<void>;
/**
* Check if a file exists
*/
fileExists(key: string): Promise<boolean>;
/**
* Get file metadata
*/
getFileMetadata(key: string): Promise<any>;
/**
* Get file content as buffer
*/
getFileContent(key: string): Promise<Buffer>;
/**
* Get streaming URL for a file
*/
getStreamingUrl(key: string): Promise<string>;
/**
* Test the connection to the storage provider
*/
testConnection(): Promise<boolean>;
}
/**
* Storage Provider Factory
* Creates the appropriate storage provider based on configuration
*/
export class StorageProviderFactory {
static async createProvider(config: StorageConfig): Promise<StorageProvider> {
switch (config.provider) {
case 's3':
const { S3Service } = await import('./s3Service.js');
return new S3Service(config as any);
case 'webdav':
const { WebDAVService } = await import('./webdavService.js');
return new WebDAVService(config as any);
default:
throw new Error(`Unsupported storage provider: ${config.provider}`);
}
}
/**
* Load storage configuration from file or environment
*/
static async loadConfig(): Promise<StorageConfig> {
try {
const configPath = path.join(process.cwd(), 'storage-config.json');
const configData = await fs.readFile(configPath, 'utf-8');
return JSON.parse(configData);
} catch (error) {
console.warn('Failed to load storage-config.json, using environment variables as fallback');
// Determine provider from environment
const provider = process.env.STORAGE_PROVIDER || 's3';
if (provider === 'webdav') {
return {
provider: 'webdav',
url: process.env.WEBDAV_URL || '',
username: process.env.WEBDAV_USERNAME || '',
password: process.env.WEBDAV_PASSWORD || '',
basePath: process.env.WEBDAV_BASE_PATH || '/music-files',
};
} else {
return {
provider: 's3',
endpoint: process.env.S3_ENDPOINT || 'http://localhost:9000',
accessKeyId: process.env.S3_ACCESS_KEY_ID || 'minioadmin',
secretAccessKey: process.env.S3_SECRET_ACCESS_KEY || 'minioadmin',
bucketName: process.env.S3_BUCKET_NAME || 'music-files',
region: process.env.S3_REGION || 'us-east-1',
useSSL: process.env.S3_USE_SSL !== 'false',
};
}
}
}
}
// Import required modules
import fs from 'fs/promises';
import path from 'path';

337
packages/backend/src/services/webdavService.ts
View File

@ -1,337 +0,0 @@
import { createClient, WebDAVClient, FileStat } from 'webdav';
import { v4 as uuidv4 } from 'uuid';
import { StorageProvider, StorageConfig, UploadResult, FileInfo } from './storageProvider.js';
export interface WebDAVConfig extends StorageConfig {
provider: 'webdav';
url: string;
username: string;
password: string;
basePath?: string;
}
export class WebDAVService implements StorageProvider {
private client: WebDAVClient;
private basePath: string;
private config: WebDAVConfig;
constructor(config: WebDAVConfig) {
this.config = config;
this.client = createClient(config.url, {
username: config.username,
password: config.password,
});
this.basePath = config.basePath || '/music-files';
}
/**
* Load WebDAV configuration from file or environment
*/
static async loadConfig(): Promise<WebDAVConfig> {
try {
const configPath = path.join(process.cwd(), 'storage-config.json');
const configData = await fs.readFile(configPath, 'utf-8');
const config = JSON.parse(configData);
if (config.provider !== 'webdav') {
throw new Error('Configuration is not for WebDAV provider');
}
return config;
} catch (error) {
console.warn('Failed to load storage-config.json, using environment variables as fallback');
return {
provider: 'webdav',
url: process.env.WEBDAV_URL || '',
username: process.env.WEBDAV_USERNAME || '',
password: process.env.WEBDAV_PASSWORD || '',
basePath: process.env.WEBDAV_BASE_PATH || '/music-files',
};
}
}
/**
* Create WebDAVService instance with configuration from file
*/
static async createFromConfig(): Promise<WebDAVService> {
const config = await this.loadConfig();
return new WebDAVService(config);
}
/**
* Upload a file to WebDAV
*/
async uploadFile(
file: Buffer,
originalName: string,
contentType: string,
targetFolder?: string
): Promise<UploadResult> {
// Sanitize filename to be safe for WebDAV
const sanitizedFilename = this.sanitizeFilename(originalName);
const cleaned = typeof targetFolder === 'string' ? targetFolder.replace(/^\/+|\/+$/g, '') : '';
const safeFolder = cleaned;
// Use original filename instead of UUID
const key = safeFolder
? `${safeFolder}/${sanitizedFilename}`
: sanitizedFilename;
const remotePath = `${this.basePath}/${key}`;
// Ensure the directory exists
const dirPath = remotePath.substring(0, remotePath.lastIndexOf('/'));
await this.ensureDirectoryExists(dirPath);
// Check if file already exists and handle conflicts
const finalKey = await this.handleFilenameConflict(key, remotePath);
// Upload the file
await this.client.putFileContents(`${this.basePath}/${finalKey}`, file, {
overwrite: true,
headers: {
'Content-Type': contentType,
},
});
return {
key: finalKey,
url: `${this.config.url}${this.basePath}/${finalKey}`,
size: file.length,
contentType,
};
}
/**
* Recursively list all files in the WebDAV directory
*/
async listAllFiles(prefix: string = ''): Promise<FileInfo[]> {
const files: FileInfo[] = [];
const searchPath = `${this.basePath}/${prefix}`.replace(/\/+/g, '/');
try {
const response = await this.client.getDirectoryContents(searchPath, {
deep: true,
maxDepth: -1, // No depth limit
});
// Handle both single item and array responses
const items = Array.isArray(response) ? response : [response];
for (const item of items) {
if (item && typeof item === 'object' && 'type' in item && item.type === 'file') {
const fileItem = item as any;
const relativePath = fileItem.filename.replace(this.basePath + '/', '');
files.push({
key: relativePath,
size: fileItem.size || 0,
lastModified: new Date(fileItem.lastmod || Date.now()),
contentType: this.getContentTypeFromFilename(fileItem.basename),
});
}
}
} catch (error) {
console.error('Error listing WebDAV files:', error);
}
return files;
}
/**
* List all folders in the WebDAV directory
*/
async listAllFolders(prefix: string = ''): Promise<string[]> {
const folders = new Set<string>();
const searchPath = `${this.basePath}/${prefix}`.replace(/\/+/g, '/');
try {
const response = await this.client.getDirectoryContents(searchPath, {
deep: true,
maxDepth: -1, // No depth limit
});
// Handle both single item and array responses
const items = Array.isArray(response) ? response : [response];
for (const item of items) {
if (item && typeof item === 'object' && 'type' in item && item.type === 'directory') {
const dirItem = item as any;
const relativePath = dirItem.filename.replace(this.basePath + '/', '');
if (relativePath) {
folders.add(relativePath);
}
}
}
} catch (error) {
console.error('Error listing WebDAV folders:', error);
}
return Array.from(folders).sort();
}
/**
* Generate a direct URL for file access (WebDAV doesn't support presigned URLs)
*/
async getPresignedUrl(key: string, expiresIn: number = 3600): Promise<string> {
// WebDAV doesn't support presigned URLs, so we return a direct URL
// Use the config URL directly since WebDAV client doesn't expose getURL()
const baseUrl = this.config.url;
return `${baseUrl}${this.basePath}/${key}`;
}
/**
* Delete a file from WebDAV - DISABLED for read-only safety
*/
async deleteFile(key: string): Promise<void> {
throw new Error('File deletion is disabled for WebDAV integration to prevent accidental data loss');
}
/**
* Check if a file exists
*/
async fileExists(key: string): Promise<boolean> {
try {
const remotePath = `${this.basePath}/${key}`;
const response = await this.client.stat(remotePath);
const stat = Array.isArray(response) ? response[0] : response;
return stat && typeof stat === 'object' && 'type' in stat && stat.type === 'file';
} catch (error) {
return false;
}
}
/**
* Get file metadata
*/
async getFileMetadata(key: string): Promise<any> {
const remotePath = `${this.basePath}/${key}`;
const response = await this.client.stat(remotePath);
return Array.isArray(response) ? response[0] : response;
}
/**
* Get file content as buffer
*/
async getFileContent(key: string): Promise<Buffer> {
const remotePath = `${this.basePath}/${key}`;
const arrayBuffer = await this.client.getFileContents(remotePath, {
format: 'binary',
});
return Buffer.from(arrayBuffer as ArrayBuffer);
}
/**
* Get streaming URL for a file
*/
async getStreamingUrl(key: string): Promise<string> {
// Use the config URL directly since WebDAV client doesn't expose getURL()
const baseUrl = this.config.url;
return `${baseUrl}${this.basePath}/${key}`;
}
/**
* Test the connection to WebDAV
*/
async testConnection(): Promise<boolean> {
try {
// Try to list the base directory to test the connection
await this.client.getDirectoryContents(this.basePath);
return true;
} catch (error) {
console.error('WebDAV connection test failed:', error);
return false;
}
}
/**
* Ensure a directory exists, creating it if necessary
*/
private async ensureDirectoryExists(dirPath: string): Promise<void> {
try {
// Check if directory exists
await this.client.stat(dirPath);
} catch (error) {
// Directory doesn't exist, create it
try {
await this.client.createDirectory(dirPath, { recursive: true });
} catch (createError) {
console.error('Failed to create directory:', dirPath, createError);
throw createError;
}
}
}
/**
* Get content type from filename
*/
private getContentTypeFromFilename(filename: string): string {
const ext = filename.split('.').pop()?.toLowerCase();
switch (ext) {
case 'mp3': return 'audio/mpeg';
case 'wav': return 'audio/wav';
case 'flac': return 'audio/flac';
case 'm4a': return 'audio/mp4';
case 'aac': return 'audio/aac';
case 'ogg': return 'audio/ogg';
case 'opus': return 'audio/opus';
case 'wma': return 'audio/x-ms-wma';
case 'aif': return 'audio/aiff';
case 'aiff': return 'audio/aiff';
default: return 'application/octet-stream';
}
}
/**
* Sanitize filename to be safe for WebDAV
*/
private sanitizeFilename(filename: string): string {
// Remove or replace characters that might cause issues in WebDAV
return filename
.replace(/[<>:"|?*]/g, '_') // Replace problematic characters
.replace(/[\x00-\x1f\x80-\x9f]/g, '') // Remove control characters
.replace(/\s+/g, ' ') // Normalize whitespace
.trim();
}
/**
* Handle filename conflicts by adding a number suffix
*/
private async handleFilenameConflict(key: string, remotePath: string): Promise<string> {
try {
// Check if file exists
await this.client.stat(remotePath);
// File exists, generate a new name with number suffix
const pathParts = key.split('/');
const filename = pathParts.pop() || '';
const dir = pathParts.join('/');
const nameWithoutExt = filename.substring(0, filename.lastIndexOf('.'));
const extension = filename.substring(filename.lastIndexOf('.'));
let counter = 1;
let newKey: string;
let newRemotePath: string;
do {
const newFilename = `${nameWithoutExt} (${counter})${extension}`;
newKey = dir ? `${dir}/${newFilename}` : newFilename;
newRemotePath = `${this.basePath}/${newKey}`;
counter++;
// Prevent infinite loop
if (counter > 1000) {
throw new Error('Too many filename conflicts');
}
} while (await this.fileExists(newKey));
return newKey;
} catch (error) {
// File doesn't exist, use original key
return key;
}
}
}
// Import required modules
import fs from 'fs/promises';
import path from 'path';

7
packages/backend/storage-config.json
View File

@ -1,7 +0,0 @@
{
"provider": "webdav",
"url": "https://cloud.geertrademakers.nl/remote.php/dav/files/admin",
"username": "admin",
"password": "XPZK2-MGQ5W-7Yetf-nr8gf-s5g5Z",
"basePath": "/Music"
}

45
packages/backend/test-audio-detection.js
View File

@ -1,45 +0,0 @@
import { AudioMetadataService } from './dist/services/audioMetadataService.js';
const audioService = new AudioMetadataService();
// Test files from the debug output
const testFiles = [
'01 Gas Op Die Lollie.mp3',
'ACRAZE - Do It To It (Extended Mix).mp3',
'test.flac',
'sample.wav',
'music.m4a',
'song.aac',
'track.ogg',
'audio.opus',
'file.wma',
'sound.aif',
'music.aiff',
'image.jpg',
'archive.zip',
'script.py',
'info.nfo',
'video.mp4',
'installer.dmg',
'playlist.m3u',
'readme.md',
'script.sh'
];
console.log('🎵 Testing audio file detection:');
console.log('');
testFiles.forEach(filename => {
const isAudio = audioService.isAudioFile(filename);
const status = isAudio ? '✅' : '❌';
console.log(`${status} ${filename} -> ${isAudio ? 'AUDIO' : 'NOT AUDIO'}`);
});
console.log('');
console.log('📊 Summary:');
const audioFiles = testFiles.filter(f => audioService.isAudioFile(f));
const nonAudioFiles = testFiles.filter(f => !audioService.isAudioFile(f));
console.log(`Audio files: ${audioFiles.length}`);
console.log(`Non-audio files: ${nonAudioFiles.length}`);
console.log(`Total files: ${testFiles.length}`);

113
packages/backend/test-background-job-flow.js
View File

@ -1,113 +0,0 @@
import { StorageProviderFactory } from './dist/services/storageProvider.js';
import { AudioMetadataService } from './dist/services/audioMetadataService.js';
async function testBackgroundJobFlow() {
try {
console.log('🔍 Testing Background Job Flow:');
console.log('');
// Step 1: Load config and create provider (same as background job)
console.log('1️⃣ Loading storage configuration...');
const config = await StorageProviderFactory.loadConfig();
console.log('Config provider:', config.provider);
console.log('Config basePath:', config.basePath);
console.log('');
// Step 2: Create storage service (same as background job)
console.log('2️⃣ Creating storage service...');
const storageService = await StorageProviderFactory.createProvider(config);
console.log('Storage service created:', storageService.constructor.name);
console.log('');
// Step 3: List all files (same as background job)
console.log('3️⃣ Listing all files from storage...');
const startTime = Date.now();
const storageFiles = await storageService.listAllFiles();
const endTime = Date.now();
console.log(`✅ listAllFiles completed in ${endTime - startTime}ms`);
console.log('Total storage files found:', storageFiles.length);
console.log('');
// Step 4: Filter for audio files (same as background job)
console.log('4️⃣ Filtering for audio files...');
const audioMetadataService = new AudioMetadataService();
const audioFiles = storageFiles.filter(storageFile => {
const filename = storageFile.key.split('/').pop() || storageFile.key;
const isAudio = audioMetadataService.isAudioFile(filename);
if (!isAudio) {
console.log(` ❌ Not audio: ${filename}`);
}
return isAudio;
});
console.log('Audio files found:', audioFiles.length);
console.log('');
// Step 5: Show breakdown by file type
console.log('5️⃣ File type breakdown:');
const fileTypes = new Map();
storageFiles.forEach(file => {
const filename = file.key.split('/').pop() || file.key;
const ext = filename.split('.').pop()?.toLowerCase() || 'no-extension';
fileTypes.set(ext, (fileTypes.get(ext) || 0) + 1);
});
const sortedTypes = Array.from(fileTypes.entries()).sort((a, b) => b[1] - a[1]);
sortedTypes.forEach(([ext, count]) => {
const isAudio = audioMetadataService.isAudioFile(`test.${ext}`);
const status = isAudio ? '🎵' : '📄';
console.log(` ${status} .${ext}: ${count} files`);
});
console.log('');
// Step 6: Show MP3 breakdown
console.log('6️⃣ MP3 files breakdown:');
const mp3Files = storageFiles.filter(file => {
const filename = file.key.split('/').pop() || file.key;
return filename.toLowerCase().endsWith('.mp3');
});
console.log('MP3 files found:', mp3Files.length);
// Show directory distribution of MP3 files
const mp3DirCounts = new Map();
mp3Files.forEach(file => {
const dir = file.key.split('/')[0];
mp3DirCounts.set(dir, (mp3DirCounts.get(dir) || 0) + 1);
});
const sortedMp3Dirs = Array.from(mp3DirCounts.entries()).sort((a, b) => b[1] - a[1]);
console.log('MP3 files by directory:');
sortedMp3Dirs.forEach(([dir, count]) => {
console.log(` 📁 ${dir}: ${count} MP3 files`);
});
console.log('');
// Step 7: Test with different prefixes (if WebDAV)
if (config.provider === 'webdav') {
console.log('7️⃣ Testing with different prefixes...');
const testPrefixes = ['', 'Gekocht', 'Merijn Music', 'Musica'];
for (const prefix of testPrefixes) {
try {
const prefixFiles = await storageService.listAllFiles(prefix);
const prefixMp3Files = prefixFiles.filter(file => {
const filename = file.key.split('/').pop() || file.key;
return filename.toLowerCase().endsWith('.mp3');
});
console.log(` 📁 Prefix "${prefix}": ${prefixFiles.length} total, ${prefixMp3Files.length} MP3`);
} catch (error) {
console.log(` ❌ Prefix "${prefix}": Error - ${error.message}`);
}
}
}
} catch (error) {
console.error('❌ Failed to test background job flow:', error);
console.error('Error details:', error.message);
console.error('Stack trace:', error.stack);
}
}
// Run the test
testBackgroundJobFlow().catch(console.error);

94
packages/backend/test-background-job-simulation.js
View File

@ -1,94 +0,0 @@
import { StorageProviderFactory } from './dist/services/storageProvider.js';
import { AudioMetadataService } from './dist/services/audioMetadataService.js';
async function testBackgroundJobSimulation() {
try {
console.log('🔍 Testing Background Job Simulation:');
console.log('');
// Step 1: Load config and create provider (exactly like background job)
console.log('1️⃣ Loading storage configuration...');
const config = await StorageProviderFactory.loadConfig();
console.log('Config provider:', config.provider);
console.log('Config basePath:', config.basePath);
console.log('');
// Step 2: Create storage service (exactly like background job)
console.log('2️⃣ Creating storage service...');
const storageService = await StorageProviderFactory.createProvider(config);
console.log('Storage service created:', storageService.constructor.name);
console.log('');
// Step 3: Create audio metadata service (exactly like background job)
console.log('3️⃣ Creating audio metadata service...');
const audioMetadataService = new AudioMetadataService();
console.log('Audio metadata service created');
console.log('');
// Step 4: List all files (exactly like background job)
console.log('4️⃣ Listing all files from storage...');
const startTime = Date.now();
const storageFiles = await storageService.listAllFiles();
const endTime = Date.now();
console.log(`✅ listAllFiles completed in ${endTime - startTime}ms`);
console.log('Total storage files found:', storageFiles.length);
console.log('');
// Step 5: Filter for audio files (exactly like background job)
console.log('5️⃣ Filtering for audio files...');
const audioFiles = storageFiles.filter(storageFile => {
const filename = storageFile.key.split('/').pop() || storageFile.key;
const isAudio = audioMetadataService.isAudioFile(filename);
return isAudio;
});
console.log('Audio files found:', audioFiles.length);
console.log('');
// Step 6: Show MP3 breakdown
console.log('6️⃣ MP3 files breakdown:');
const mp3Files = audioFiles.filter(file => {
const filename = file.key.split('/').pop() || file.key;
return filename.toLowerCase().endsWith('.mp3');
});
console.log('MP3 files found:', mp3Files.length);
console.log('');
// Step 7: Show file type breakdown
console.log('7️⃣ File type breakdown:');
const fileTypes = new Map();
audioFiles.forEach(file => {
const filename = file.key.split('/').pop() || file.key;
const ext = filename.split('.').pop()?.toLowerCase() || 'no-extension';
fileTypes.set(ext, (fileTypes.get(ext) || 0) + 1);
});
const sortedTypes = Array.from(fileTypes.entries()).sort((a, b) => b[1] - a[1]);
sortedTypes.forEach(([ext, count]) => {
console.log(` .${ext}: ${count} files`);
});
console.log('');
// Step 8: Show directory breakdown
console.log('8️⃣ Directory breakdown:');
const dirCounts = new Map();
audioFiles.forEach(file => {
const dir = file.key.split('/')[0];
dirCounts.set(dir, (dirCounts.get(dir) || 0) + 1);
});
const sortedDirs = Array.from(dirCounts.entries()).sort((a, b) => b[1] - a[1]);
sortedDirs.forEach(([dir, count]) => {
console.log(` 📁 ${dir}: ${count} audio files`);
});
} catch (error) {
console.error('❌ Failed to test background job simulation:', error);
console.error('Error details:', error.message);
console.error('Stack trace:', error.stack);
}
}
// Run the test
testBackgroundJobSimulation().catch(console.error);

51
packages/backend/test-current-webdav.js
View File

@ -1,51 +0,0 @@
import { StorageProviderFactory } from './dist/services/storageProvider.js';
async function testCurrentWebDAV() {
try {
console.log('🔍 Testing Current WebDAV Service:');
console.log('');
// Load config and create provider (same as background job)
const config = await StorageProviderFactory.loadConfig();
console.log('Config provider:', config.provider);
console.log('Config basePath:', config.basePath);
console.log('');
// Create storage service
const storageService = await StorageProviderFactory.createProvider(config);
console.log('Storage service created:', storageService.constructor.name);
console.log('');
// List all files
console.log('📁 Listing all files...');
const startTime = Date.now();
const storageFiles = await storageService.listAllFiles();
const endTime = Date.now();
console.log(`✅ listAllFiles completed in ${endTime - startTime}ms`);
console.log('Total storage files found:', storageFiles.length);
console.log('');
// Filter for MP3 files
const mp3Files = storageFiles.filter(file => {
const filename = file.key.split('/').pop() || file.key;
return filename.toLowerCase().endsWith('.mp3');
});
console.log('🎵 MP3 files found:', mp3Files.length);
console.log('');
// Show first 10 MP3 files
console.log('🎵 First 10 MP3 files:');
mp3Files.slice(0, 10).forEach((file, index) => {
console.log(` ${index + 1}. ${file.key} (${file.size} bytes)`);
});
} catch (error) {
console.error('❌ Failed to test current WebDAV service:', error);
console.error('Error details:', error.message);
console.error('Stack trace:', error.stack);
}
}
// Run the test
testCurrentWebDAV().catch(console.error);

1
packages/backend/test-playback.mp3
View File

@ -1 +0,0 @@
test content

54
packages/backend/test-server-webdav.js
View File

@ -1,54 +0,0 @@
import fetch from 'node-fetch';
async function testServerWebDAV() {
try {
console.log('🔍 Testing Server WebDAV via API:');
console.log('');
// Test the storage sync endpoint
console.log('1️⃣ Testing storage sync endpoint...');
const response = await fetch('http://localhost:3000/api/music/sync-s3', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({ force: true })
});
if (response.ok) {
const result = await response.json();
console.log('✅ Storage sync started:', result);
console.log('');
// Wait a bit and check job progress
console.log('2️⃣ Waiting for job to start...');
await new Promise(resolve => setTimeout(resolve, 5000));
const jobsResponse = await fetch('http://localhost:3000/api/background-jobs/jobs');
if (jobsResponse.ok) {
const jobs = await jobsResponse.json();
const latestJob = jobs.jobs[jobs.jobs.length - 1];
console.log('📊 Latest job status:', {
jobId: latestJob.jobId,
status: latestJob.status,
progress: latestJob.progress,
message: latestJob.message,
current: latestJob.current,
total: latestJob.total
});
if (latestJob.result) {
console.log('📊 Job result:', latestJob.result);
}
}
} else {
console.error('❌ Failed to start storage sync:', response.status, response.statusText);
}
} catch (error) {
console.error('❌ Failed to test server WebDAV:', error);
}
}
// Run the test
testServerWebDAV().catch(console.error);

92
packages/backend/test-webdav-service.js
View File

@ -1,92 +0,0 @@
import { WebDAVService } from './src/services/webdavService.js';
import fs from 'fs';
async function testWebDAVService() {
try {
// Load configuration
const configData = fs.readFileSync('storage-config.json', 'utf-8');
const config = JSON.parse(configData);
console.log('🔍 Testing WebDAV Service:');
console.log('URL:', config.url);
console.log('Username:', config.username);
console.log('Base Path:', config.basePath);
console.log('');
// Create WebDAV service
const webdavService = new WebDAVService(config);
console.log('🔗 Testing connection...');
const connectionTest = await webdavService.testConnection();
console.log('Connection test:', connectionTest ? '✅ Success' : '❌ Failed');
console.log('');
if (connectionTest) {
console.log('📁 Testing listAllFiles...');
const startTime = Date.now();
const allFiles = await webdavService.listAllFiles();
const endTime = Date.now();
console.log(`✅ listAllFiles completed in ${endTime - startTime}ms`);
console.log('Total files found:', allFiles.length);
console.log('');
// Filter for MP3 files
const mp3Files = allFiles.filter(file => {
const filename = file.key.split('/').pop() || file.key;
return filename.toLowerCase().endsWith('.mp3');
});
console.log('🎵 MP3 Files found by service:', mp3Files.length);
console.log('');
// Show first 20 MP3 files
console.log('🎵 First 20 MP3 files from service:');
mp3Files.slice(0, 20).forEach((file, index) => {
console.log(` ${index + 1}. ${file.key} (${file.size} bytes)`);
});
if (mp3Files.length > 20) {
console.log(` ... and ${mp3Files.length - 20} more MP3 files`);
}
// Check directory distribution
console.log('');
console.log('📁 Directory distribution of MP3 files:');
const dirCounts = new Map();
mp3Files.forEach(file => {
const dir = file.key.split('/')[0];
dirCounts.set(dir, (dirCounts.get(dir) || 0) + 1);
});
const sortedDirs = Array.from(dirCounts.entries()).sort((a, b) => b[1] - a[1]);
sortedDirs.forEach(([dir, count]) => {
console.log(` 📁 ${dir}: ${count} MP3 files`);
});
// Test with different prefixes
console.log('');
console.log('🔍 Testing with different prefixes...');
const testPrefixes = ['', 'Gekocht', 'Merijn Music', 'Musica'];
for (const prefix of testPrefixes) {
try {
const prefixFiles = await webdavService.listAllFiles(prefix);
const prefixMp3Files = prefixFiles.filter(file => {
const filename = file.key.split('/').pop() || file.key;
return filename.toLowerCase().endsWith('.mp3');
});
console.log(` 📁 Prefix "${prefix}": ${prefixMp3Files.length} MP3 files`);
} catch (error) {
console.log(` ❌ Prefix "${prefix}": Error - ${error.message}`);
}
}
}
} catch (error) {
console.error('❌ Failed to test WebDAV service:', error);
}
}
// Run the test
testWebDAVService().catch(console.error);

61
packages/backend/test-webdav.js
View File

@ -1,61 +0,0 @@
#!/usr/bin/env node
/**
* WebDAV Integration Test
*
* This script tests the WebDAV service integration with a sample Nextcloud instance.
* Update the configuration below to match your Nextcloud setup.
*/
import { WebDAVService } from './dist/services/webdavService.js';
// Test configuration - update these values for your Nextcloud instance
const testConfig = {
provider: 'webdav',
url: 'https://your-nextcloud.com/remote.php/dav/files/username/',
username: 'your-username',
password: 'your-password-or-app-password',
basePath: '/music-files'
};
async function testWebDAVConnection() {
console.log('🧪 Testing WebDAV connection...');
console.log('📋 Configuration:', {
...testConfig,
password: '***' // Hide password in logs
});
try {
// Create WebDAV service
const webdavService = new WebDAVService(testConfig);
// Test connection
console.log('🔗 Testing connection...');
const isConnected = await webdavService.testConnection();
if (isConnected) {
console.log('✅ WebDAV connection successful!');
// Test listing files
console.log('📁 Testing file listing...');
const files = await webdavService.listAllFiles();
console.log(`📊 Found ${files.length} files`);
// Test listing folders
console.log('📂 Testing folder listing...');
const folders = await webdavService.listAllFolders();
console.log(`📊 Found ${folders.length} folders:`, folders);
console.log('🎉 All WebDAV tests passed!');
} else {
console.log('❌ WebDAV connection failed');
}
} catch (error) {
console.error('❌ WebDAV test failed:', error.message);
console.error('💡 Make sure to update the configuration in this file with your Nextcloud details');
}
}
// Run the test
testWebDAVConnection().catch(console.error);

104
packages/frontend/src/components/BackgroundJobProgress.tsx
View File

@ -27,7 +27,7 @@ import { api } from '../services/api';
interface JobProgress { interface JobProgress {
jobId: string; jobId: string;
type: 'storage-sync' | 'song-matching'; type: 's3-sync' | 'song-matching';
status: 'running' | 'completed' | 'failed'; status: 'running' | 'completed' | 'failed';
progress: number; progress: number;
current: number; current: number;
@ -56,42 +56,98 @@ export const BackgroundJobProgress: React.FC<BackgroundJobProgressProps> = ({
const { isOpen, onClose } = useDisclosure(); const { isOpen, onClose } = useDisclosure();
const intervalRef = useRef<NodeJS.Timeout | null>(null); const intervalRef = useRef<NodeJS.Timeout | null>(null);
// Simple polling function // Load all jobs
const pollJobs = async () => { const loadJobs = async () => {
try { try {
setLoading(true);
const jobsData = await api.getAllJobs();
setJobs(jobsData);
setError(null);
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to load jobs');
} finally {
setLoading(false);
}
};
// Update specific job progress
const updateJobProgress = async (jobId: string) => {
try {
const progress = await api.getJobProgress(jobId);
setJobs(prev => prev.map(job =>
job.jobId === jobId ? progress : job
));
// Handle job completion
if (progress.status === 'completed' && onJobComplete) {
onJobComplete(progress.result);
} else if (progress.status === 'failed' && onJobError) {
onJobError(progress.error || 'Job failed');
}
} catch (err) {
console.error('Error updating job progress:', err);
}
};
// Start polling for jobs and update progress
const startPolling = () => {
if (intervalRef.current) {
clearInterval(intervalRef.current);
}
const tick = async () => {
try {
// Always reload job list to detect newly started jobs
const jobsData = await api.getAllJobs(); const jobsData = await api.getAllJobs();
setJobs(jobsData); setJobs(jobsData);
// Handle job completion for the specific job if provided // Update progress for active jobs
const activeJobIds = jobsData.filter((j: JobProgress) => j.status === 'running').map((j: JobProgress) => j.jobId);
for (const id of activeJobIds) {
await updateJobProgress(id);
}
if (jobId) { if (jobId) {
const specificJob = jobsData.find((j: JobProgress) => j.jobId === jobId); await updateJobProgress(jobId);
if (specificJob) {
if (specificJob.status === 'completed' && onJobComplete) {
onJobComplete(specificJob.result);
} else if (specificJob.status === 'failed' && onJobError) {
onJobError(specificJob.error || 'Job failed');
}
}
} }
} catch (err) { } catch (err) {
// ignore transient polling errors // ignore transient polling errors
} }
}; };
// Adaptive interval: 2s if active jobs, else 10s
const schedule = async () => {
await tick();
const hasActive = (jobs || []).some(j => j.status === 'running');
const delay = hasActive ? 2000 : 10000;
intervalRef.current = setTimeout(schedule, delay) as any;
};
schedule();
};
// Stop polling
const stopPolling = () => {
if (intervalRef.current) {
clearTimeout(intervalRef.current as any);
intervalRef.current = null;
}
};
// Start polling on mount and stop on unmount // Start polling on mount and stop on unmount
useEffect(() => { useEffect(() => {
// Initial poll loadJobs();
pollJobs(); startPolling();
return () => stopPolling();
// Set up interval polling }, []);
const interval = setInterval(() => {
pollJobs();
}, 10000); // Simple 10-second interval
// Cleanup on unmount
useEffect(() => {
return () => { return () => {
clearInterval(interval); stopPolling();
}; };
}, [jobId, onJobComplete, onJobError]); }, []);
const getStatusColor = (status: string) => { const getStatusColor = (status: string) => {
switch (status) { switch (status) {
@ -152,7 +208,7 @@ export const BackgroundJobProgress: React.FC<BackgroundJobProgressProps> = ({
<Box key={job.jobId} p={3} bg="gray.700" borderRadius="md"> <Box key={job.jobId} p={3} bg="gray.700" borderRadius="md">
<Flex justify="space-between" align="center" mb={2}> <Flex justify="space-between" align="center" mb={2}>
<Text fontSize="sm" fontWeight="medium" color="gray.100"> <Text fontSize="sm" fontWeight="medium" color="gray.100">
{job.type === 'storage-sync' ? 'Storage Sync' : 'Song Matching'} {job.type === 's3-sync' ? 'S3 Sync' : 'Song Matching'}
</Text> </Text>
<Badge colorScheme={getStatusColor(job.status)} size="sm"> <Badge colorScheme={getStatusColor(job.status)} size="sm">
{job.status} {job.status}
@ -190,7 +246,7 @@ export const BackgroundJobProgress: React.FC<BackgroundJobProgressProps> = ({
<VStack spacing={4} align="stretch"> <VStack spacing={4} align="stretch">
<HStack justify="space-between"> <HStack justify="space-between">
<Text fontWeight="bold" color="gray.100">All Jobs</Text> <Text fontWeight="bold" color="gray.100">All Jobs</Text>
<Button size="sm" onClick={() => pollJobs()} isLoading={loading}> <Button size="sm" onClick={loadJobs} isLoading={loading}>
Refresh Refresh
</Button> </Button>
</HStack> </HStack>
@ -216,7 +272,7 @@ export const BackgroundJobProgress: React.FC<BackgroundJobProgressProps> = ({
<Tr key={job.jobId}> <Tr key={job.jobId}>
<Td> <Td>
<Text fontSize="sm" color="gray.100"> <Text fontSize="sm" color="gray.100">
{job.type === 'storage-sync' ? 'Storage Sync' : 'Song Matching'} {job.type === 's3-sync' ? 'S3 Sync' : 'Song Matching'}
</Text> </Text>
</Td> </Td>
<Td> <Td>

12
packages/frontend/src/components/DuplicatesViewer.tsx
View File

@ -162,12 +162,12 @@ export const DuplicatesViewer: React.FC = () => {
}} }}
/> />
</Tooltip> </Tooltip>
<Tooltip label="Merge duplicates (keeps target, removes others from playlists only)"> <Tooltip label="Delete other duplicates (optionally remove music files)">
<IconButton <IconButton
aria-label="Merge duplicates" aria-label="Delete duplicates"
icon={<FiCheck />} icon={<FiTrash2 />}
size="sm" size="sm"
colorScheme="blue" colorScheme="red"
variant="outline" variant="outline"
isLoading={processingGroupKey === group.key} isLoading={processingGroupKey === group.key}
onClick={async () => { onClick={async () => {
@ -175,7 +175,7 @@ export const DuplicatesViewer: React.FC = () => {
try { try {
const targetId = it.songId; const targetId = it.songId;
const others = group.items.map(x => x.songId).filter(id => id !== targetId); const others = group.items.map(x => x.songId).filter(id => id !== targetId);
// Merge playlists (safe), but don't delete songs or music files // First merge playlists (safe), then delete redundant songs and optionally their music files
const allPlaylists = await api.getPlaylists(); const allPlaylists = await api.getPlaylists();
const updated = allPlaylists.map(p => { const updated = allPlaylists.map(p => {
if (p.type === 'playlist') { if (p.type === 'playlist') {
@ -196,7 +196,7 @@ export const DuplicatesViewer: React.FC = () => {
return p; return p;
}); });
await api.savePlaylists(updated as any); await api.savePlaylists(updated as any);
// Note: We don't call deleteDuplicateSongs anymore to keep it read-only await api.deleteDuplicateSongs(targetId, others, true);
await loadDuplicates(minGroupSize); await loadDuplicates(minGroupSize);
} finally { } finally {
setProcessingGroupKey(null); setProcessingGroupKey(null);

3
packages/frontend/src/components/MusicUpload.tsx
View File

@ -249,9 +249,6 @@ export const MusicUpload: React.FC<MusicUploadProps> = ({ onUploadComplete }) =>
<Text fontSize="sm" color="gray.400"> <Text fontSize="sm" color="gray.400">
Supports MP3, WAV, FLAC, AAC, OGG, WMA, Opus (max 100MB per file) Supports MP3, WAV, FLAC, AAC, OGG, WMA, Opus (max 100MB per file)
</Text> </Text>
<Text fontSize="xs" color="gray.500">
Original filenames and metadata will be preserved
</Text>
</VStack> </VStack>
</Box> </Box>

2
packages/frontend/src/components/SongMatching.tsx
View File

@ -261,7 +261,7 @@ export const SongMatching: React.FC = () => {
<CardBody> <CardBody>
<VStack spacing={4}> <VStack spacing={4}>
<Text color="gray.300" textAlign="center"> <Text color="gray.300" textAlign="center">
Try linking any remaining unmatched files. The main storage sync already performs matching; use this for leftovers. Try linking any remaining unmatched files. The main S3 sync already performs matching; use this for leftovers.
</Text> </Text>
<Button <Button
leftIcon={<FiZap />} leftIcon={<FiZap />}

45
packages/frontend/src/pages/Configuration.tsx
View File

@ -29,12 +29,12 @@ import { FiDatabase, FiSettings, FiUpload, FiLink, FiRefreshCw, FiTrash2 } from
import { useNavigate } from "react-router-dom"; import { useNavigate } from "react-router-dom";
import { useXmlParser } from "../hooks/useXmlParser"; import { useXmlParser } from "../hooks/useXmlParser";
import { StyledFileInput } from "../components/StyledFileInput"; import { StyledFileInput } from "../components/StyledFileInput";
import { StorageConfiguration } from "./StorageConfiguration"; import { S3Configuration } from "./S3Configuration";
import { MusicUpload } from "../components/MusicUpload"; import { MusicUpload } from "../components/MusicUpload";
import { SongMatching } from "../components/SongMatching"; import { SongMatching } from "../components/SongMatching";
import { api } from "../services/api"; import { api } from "../services/api";
import { DuplicatesViewer } from "../components/DuplicatesViewer.tsx"; import { DuplicatesViewer } from "../components/DuplicatesViewer.tsx";
import { useState, useMemo, useEffect } from "react"; import { useState, useMemo } from "react";
interface MusicFile { interface MusicFile {
_id: string; _id: string;
@ -63,27 +63,10 @@ export function Configuration() {
return stored ? parseInt(stored, 10) : 0; return stored ? parseInt(stored, 10) : 0;
}, []); }, []);
const [tabIndex, setTabIndex] = useState<number>(initialTabIndex); const [tabIndex, setTabIndex] = useState<number>(initialTabIndex);
const [storageProvider, setStorageProvider] = useState<string>('Storage');
// No explicit tab index enum needed // No explicit tab index enum needed
// Load current storage provider for dynamic button labels // S3 config fetch removed; Sync buttons remain available in the panel
useEffect(() => {
const loadStorageProvider = async () => {
try {
const response = await fetch('/api/config/storage');
if (response.ok) {
const data = await response.json();
setStorageProvider(data.config.provider.toUpperCase());
}
} catch (error) {
console.error('Failed to load storage provider:', error);
}
};
loadStorageProvider();
}, []);
// Storage config fetch removed; Sync buttons remain available in the panel
// Removed Music Library sync handlers from Config (moved to Sync & Matching panel) // Removed Music Library sync handlers from Config (moved to Sync & Matching panel)
@ -196,7 +179,7 @@ export function Configuration() {
<Tab color="gray.300" _selected={{ bg: "gray.700", color: "white", borderColor: "gray.600" }}> <Tab color="gray.300" _selected={{ bg: "gray.700", color: "white", borderColor: "gray.600" }}>
<HStack spacing={2}> <HStack spacing={2}>
<Icon as={FiSettings} /> <Icon as={FiSettings} />
<Text>Storage Configuration</Text> <Text>S3 Configuration</Text>
</HStack> </HStack>
</Tab> </Tab>
</TabList> </TabList>
@ -253,8 +236,8 @@ export function Configuration() {
Upload Music Files Upload Music Files
</Heading> </Heading>
<Text color="gray.400" mb={4}> <Text color="gray.400" mb={4}>
Drag and drop your music files here or click to select. Files will be uploaded to your configured storage Drag and drop your music files here or click to select. Files will be uploaded to S3 storage
(S3 or WebDAV) and metadata will be automatically extracted. and metadata will be automatically extracted.
</Text> </Text>
</Box> </Box>
<MusicUpload onUploadComplete={handleUploadComplete} /> <MusicUpload onUploadComplete={handleUploadComplete} />
@ -272,25 +255,25 @@ export function Configuration() {
leftIcon={<FiRefreshCw />} leftIcon={<FiRefreshCw />}
colorScheme="blue" colorScheme="blue"
variant="solid" variant="solid"
onClick={() => api.startStorageSync()} onClick={() => api.startS3Sync()}
> >
Sync {storageProvider} (incremental) Sync S3 (incremental)
</Button> </Button>
<Button <Button
leftIcon={<FiRefreshCw />} leftIcon={<FiRefreshCw />}
colorScheme="orange" colorScheme="orange"
variant="outline" variant="outline"
onClick={() => api.startStorageSync({ force: true })} onClick={() => api.startS3Sync({ force: true })}
> >
Force {storageProvider} Sync (rescan all) Force Sync (rescan all)
</Button> </Button>
<Button <Button
leftIcon={<FiTrash2 />} leftIcon={<FiTrash2 />}
colorScheme="red" colorScheme="red"
variant="outline" variant="outline"
onClick={() => api.startStorageSync({ clearLinks: true, force: true })} onClick={() => api.startS3Sync({ clearLinks: true, force: true })}
> >
Clear Links + Force {storageProvider} Sync Clear Links + Force Sync
</Button> </Button>
</HStack> </HStack>
<SongMatching /> <SongMatching />
@ -302,10 +285,10 @@ export function Configuration() {
<DuplicatesViewer /> <DuplicatesViewer />
</TabPanel> </TabPanel>
{/* Storage Configuration Tab */} {/* S3 Configuration Tab */}
<TabPanel bg="gray.800" p={0}> <TabPanel bg="gray.800" p={0}>
<Box p={6}> <Box p={6}>
<StorageConfiguration /> <S3Configuration />
</Box> </Box>
</TabPanel> </TabPanel>
</TabPanels> </TabPanels>

702
packages/frontend/src/pages/StorageConfiguration.tsx
View File

@ -1,702 +0,0 @@
import React, { useState, useEffect } from 'react';
import {
Box,
VStack,
HStack,
Text,
Heading,
FormControl,
FormLabel,
Input,
Button,
useToast,
Alert,
AlertIcon,
AlertTitle,
AlertDescription,
Card,
CardBody,
CardHeader,
Spinner,
Badge,
Icon,
Switch,
FormHelperText,
Select,
Divider,
} from '@chakra-ui/react';
import { FiSettings, FiZap, FiSave, FiCloud, FiServer } from 'react-icons/fi';
interface S3Config {
provider: 's3';
endpoint: string;
region: string;
accessKeyId: string;
secretAccessKey: string;
bucketName: string;
useSSL: boolean;
}
interface WebDAVConfig {
provider: 'webdav';
url: string;
username: string;
password: string;
basePath?: string;
}
type StorageConfig = S3Config | WebDAVConfig;
interface TestResult {
success: boolean;
message: string;
details?: any;
provider?: string;
}
export const StorageConfiguration: React.FC = () => {
const [config, setConfig] = useState<StorageConfig>({
provider: 's3',
endpoint: '',
region: 'us-east-1',
accessKeyId: '',
secretAccessKey: '',
bucketName: '',
useSSL: true,
});
const [isLoading, setIsLoading] = useState(false);
const [isTesting, setIsTesting] = useState(false);
const [isSaving, setIsSaving] = useState(false);
const [testResult, setTestResult] = useState<TestResult | null>(null);
const [currentConfig, setCurrentConfig] = useState<StorageConfig | null>(null);
const toast = useToast();
// Load current configuration on component mount
useEffect(() => {
loadCurrentConfig();
}, []);
const loadCurrentConfig = async () => {
setIsLoading(true);
try {
const response = await fetch('/api/config/storage');
if (response.ok) {
const data = await response.json();
setCurrentConfig(data.config);
// Handle masked passwords - don't set masked values as initial state
const configWithEmptyPasswords = { ...data.config };
if (configWithEmptyPasswords.provider === 'webdav' && configWithEmptyPasswords.password === '***') {
configWithEmptyPasswords.password = '';
}
if (configWithEmptyPasswords.provider === 's3') {
if (configWithEmptyPasswords.accessKeyId === '***') {
configWithEmptyPasswords.accessKeyId = '';
}
if (configWithEmptyPasswords.secretAccessKey === '***') {
configWithEmptyPasswords.secretAccessKey = '';
}
}
setConfig(configWithEmptyPasswords);
}
} catch (error) {
console.error('Error loading storage config:', error);
} finally {
setIsLoading(false);
}
};
const handleProviderChange = (provider: 's3' | 'webdav') => {
if (provider === 's3') {
setConfig({
provider: 's3',
endpoint: '',
region: 'us-east-1',
accessKeyId: '',
secretAccessKey: '',
bucketName: '',
useSSL: true,
});
} else {
setConfig({
provider: 'webdav',
url: '',
username: '',
password: '',
basePath: '/music-files',
});
}
};
const handleInputChange = (field: string, value: string | boolean) => {
setConfig(prev => ({
...prev,
[field]: value,
}));
};
const testConnection = async () => {
setIsTesting(true);
setTestResult(null);
try {
// For testing, merge current form state with existing config to preserve passwords
const testConfig = { ...currentConfig, ...config };
const response = await fetch('/api/config/storage/test', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify(testConfig),
});
const result = await response.json();
if (response.ok) {
setTestResult({
success: true,
message: 'Connection successful!',
details: result,
provider: testConfig.provider,
});
toast({
title: 'Connection Test Successful',
description: `${testConfig.provider.toUpperCase()} connection is working properly`,
status: 'success',
duration: 5000,
isClosable: true,
});
} else {
setTestResult({
success: false,
message: result.error || 'Connection failed',
details: result,
provider: testConfig.provider,
});
toast({
title: 'Connection Test Failed',
description: result.error || `Failed to connect to ${testConfig.provider.toUpperCase()}`,
status: 'error',
duration: 5000,
isClosable: true,
});
}
} catch (error) {
console.error('Error testing storage connection:', error);
setTestResult({
success: false,
message: 'Network error or server unavailable',
provider: testConfig.provider,
});
toast({
title: 'Connection Test Failed',
description: 'Network error or server unavailable',
status: 'error',
duration: 5000,
isClosable: true,
});
} finally {
setIsTesting(false);
}
};
const saveConfiguration = async () => {
setIsSaving(true);
try {
// Always send the complete configuration
const configToSave = { ...config };
const response = await fetch('/api/config/storage', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify(configToSave),
});
if (response.ok) {
setCurrentConfig(config);
toast({
title: 'Configuration Saved',
description: `${config.provider.toUpperCase()} configuration has been saved successfully`,
status: 'success',
duration: 3000,
isClosable: true,
});
} else {
const error = await response.json();
throw new Error(error.error || 'Failed to save configuration');
}
} catch (error) {
console.error('Error saving storage config:', error);
toast({
title: 'Save Failed',
description: error instanceof Error ? error.message : 'Failed to save configuration',
status: 'error',
duration: 5000,
isClosable: true,
});
} finally {
setIsSaving(false);
}
};
const hasChanges = () => {
if (!currentConfig) return true;
// Create a copy of currentConfig with masked values replaced with empty strings for comparison
const normalizedCurrentConfig = { ...currentConfig };
if (normalizedCurrentConfig.provider === 'webdav' && normalizedCurrentConfig.password === '***') {
normalizedCurrentConfig.password = '';
}
if (normalizedCurrentConfig.provider === 's3') {
if (normalizedCurrentConfig.accessKeyId === '***') {
normalizedCurrentConfig.accessKeyId = '';
}
if (normalizedCurrentConfig.secretAccessKey === '***') {
normalizedCurrentConfig.secretAccessKey = '';
}
}
return JSON.stringify(config) !== JSON.stringify(normalizedCurrentConfig);
};
const renderS3Config = () => (
<VStack spacing={6} align="stretch">
{/* Endpoint */}
<FormControl>
<FormLabel color="white">S3 Endpoint</FormLabel>
<Input
value={config.provider === 's3' ? config.endpoint : ''}
onChange={(e) => handleInputChange('endpoint', e.target.value)}
placeholder="https://s3.amazonaws.com or http://localhost:9000 for MinIO"
bg="gray.700"
borderColor="gray.600"
color="white"
_placeholder={{ color: 'gray.400' }}
_focus={{ borderColor: 'blue.400', boxShadow: 'none' }}
/>
<FormHelperText color="gray.400">
For AWS S3, use: https://s3.amazonaws.com. For MinIO: http://localhost:9000
</FormHelperText>
</FormControl>
{/* Region */}
<FormControl>
<FormLabel color="white">Region</FormLabel>
<Input
value={config.provider === 's3' ? config.region : ''}
onChange={(e) => handleInputChange('region', e.target.value)}
placeholder="us-east-1"
bg="gray.700"
borderColor="gray.600"
color="white"
_placeholder={{ color: 'gray.400' }}
_focus={{ borderColor: 'blue.400', boxShadow: 'none' }}
/>
<FormHelperText color="gray.400">
AWS region (e.g., us-east-1, eu-west-1) or 'us-east-1' for MinIO
</FormHelperText>
</FormControl>
{/* Access Key */}
<FormControl>
<FormLabel color="white">Access Key ID</FormLabel>
<Input
value={config.provider === 's3' ? config.accessKeyId : ''}
onChange={(e) => handleInputChange('accessKeyId', e.target.value)}
placeholder="Your S3 access key"
bg="gray.700"
borderColor="gray.600"
color="white"
_placeholder={{ color: 'gray.400' }}
_focus={{ borderColor: 'blue.400', boxShadow: 'none' }}
/>
</FormControl>
{/* Secret Key */}
<FormControl>
<FormLabel color="white">Secret Access Key</FormLabel>
<Input
type="password"
value={config.provider === 's3' ? config.secretAccessKey : ''}
onChange={(e) => handleInputChange('secretAccessKey', e.target.value)}
placeholder="Your S3 secret key"
bg="gray.700"
borderColor="gray.600"
color="white"
_placeholder={{ color: 'gray.400' }}
_focus={{ borderColor: 'blue.400', boxShadow: 'none' }}
/>
</FormControl>
{/* Bucket Name */}
<FormControl>
<FormLabel color="white">Bucket Name</FormLabel>
<Input
value={config.provider === 's3' ? config.bucketName : ''}
onChange={(e) => handleInputChange('bucketName', e.target.value)}
placeholder="music-files"
bg="gray.700"
borderColor="gray.600"
color="white"
_placeholder={{ color: 'gray.400' }}
_focus={{ borderColor: 'blue.400', boxShadow: 'none' }}
/>
<FormHelperText color="gray.400">
The S3 bucket where music files will be stored
</FormHelperText>
</FormControl>
{/* Use SSL */}
<FormControl display="flex" alignItems="center">
<FormLabel htmlFor="use-ssl" mb="0" color="white">
Use SSL/TLS
</FormLabel>
<Switch
id="use-ssl"
isChecked={config.provider === 's3' ? config.useSSL : false}
onChange={(e) => handleInputChange('useSSL', e.target.checked)}
colorScheme="blue"
/>
<FormHelperText color="gray.400" ml={3}>
Enable for HTTPS connections (recommended for production)
</FormHelperText>
</FormControl>
</VStack>
);
const renderWebDAVConfig = () => (
<VStack spacing={6} align="stretch">
{/* URL */}
<FormControl>
<FormLabel color="white">WebDAV URL</FormLabel>
<Input
value={config.provider === 'webdav' ? config.url : ''}
onChange={(e) => handleInputChange('url', e.target.value)}
placeholder="https://your-nextcloud.com/remote.php/dav/files/username/"
bg="gray.700"
borderColor="gray.600"
color="white"
_placeholder={{ color: 'gray.400' }}
_focus={{ borderColor: 'blue.400', boxShadow: 'none' }}
/>
<FormHelperText color="gray.400">
Your Nextcloud WebDAV URL (usually ends with /remote.php/dav/files/username/)
</FormHelperText>
</FormControl>
{/* Username */}
<FormControl>
<FormLabel color="white">Username</FormLabel>
<Input
value={config.provider === 'webdav' ? config.username : ''}
onChange={(e) => handleInputChange('username', e.target.value)}
placeholder="Your Nextcloud username"
bg="gray.700"
borderColor="gray.600"
color="white"
_placeholder={{ color: 'gray.400' }}
_focus={{ borderColor: 'blue.400', boxShadow: 'none' }}
/>
</FormControl>
{/* Password */}
<FormControl>
<FormLabel color="white">Password</FormLabel>
<Input
type="password"
value={config.provider === 'webdav' ? config.password : ''}
onChange={(e) => handleInputChange('password', e.target.value)}
placeholder="Your Nextcloud password or app password"
bg="gray.700"
borderColor="gray.600"
color="white"
_placeholder={{ color: 'gray.400' }}
_focus={{ borderColor: 'blue.400', boxShadow: 'none' }}
/>
<FormHelperText color="gray.400">
Use your Nextcloud password or create an app password for better security
</FormHelperText>
</FormControl>
{/* Base Path */}
<FormControl>
<FormLabel color="white">Base Path (Optional)</FormLabel>
<Input
value={config.provider === 'webdav' ? config.basePath || '' : ''}
onChange={(e) => handleInputChange('basePath', e.target.value)}
placeholder="/music-files"
bg="gray.700"
borderColor="gray.600"
color="white"
_placeholder={{ color: 'gray.400' }}
_focus={{ borderColor: 'blue.400', boxShadow: 'none' }}
/>
<FormHelperText color="gray.400">
Subfolder within your WebDAV storage where music files will be stored
</FormHelperText>
</FormControl>
</VStack>
);
if (isLoading) {
return (
<Box p={8}>
<VStack spacing={4} align="center">
<Spinner size="xl" />
<Text>Loading storage configuration...</Text>
</VStack>
</Box>
);
}
return (
<Box p={8} maxW="800px" mx="auto">
<VStack spacing={8} align="stretch">
{/* Header */}
<VStack spacing={2} align="start">
<HStack spacing={3}>
<Icon as={FiSettings} w={6} h={6} color="blue.400" />
<Heading size="lg" color="white">Storage Configuration</Heading>
</HStack>
<Text color="gray.400">
Configure your storage provider for music file storage and playback. Choose between S3-compatible storage or WebDAV (Nextcloud).
</Text>
</VStack>
{/* Provider Selection */}
<Card bg="gray.800" borderColor="gray.700">
<CardHeader>
<Heading size="md" color="white">Storage Provider</Heading>
</CardHeader>
<CardBody>
<VStack spacing={4} align="stretch">
<FormControl>
<FormLabel color="white">Select Storage Provider</FormLabel>
<Select
value={config.provider}
onChange={(e) => handleProviderChange(e.target.value as 's3' | 'webdav')}
bg="gray.700"
borderColor="gray.600"
color="white"
_focus={{ borderColor: 'blue.400', boxShadow: 'none' }}
>
<option value="s3">S3-Compatible Storage (AWS S3, MinIO, etc.)</option>
<option value="webdav">WebDAV (Nextcloud, ownCloud, etc.)</option>
</Select>
</FormControl>
<HStack spacing={4} align="center">
<Icon
as={config.provider === 's3' ? FiCloud : FiServer}
w={5} h={5}
color={config.provider === 's3' ? 'blue.400' : 'green.400'}
/>
<Text color="gray.400">
{config.provider === 's3'
? 'S3-compatible storage for scalable cloud storage'
: 'WebDAV for self-hosted solutions like Nextcloud'
}
</Text>
</HStack>
</VStack>
</CardBody>
</Card>
{/* Configuration Form */}
<Card bg="gray.800" borderColor="gray.700">
<CardHeader>
<Heading size="md" color="white">
{config.provider === 's3' ? 'S3 Configuration' : 'WebDAV Configuration'}
</Heading>
</CardHeader>
<CardBody>
{config.provider === 's3' ? renderS3Config() : renderWebDAVConfig()}
</CardBody>
</Card>
{/* Test Connection */}
<Card bg="gray.800" borderColor="gray.700">
<CardHeader>
<HStack spacing={3}>
<Icon as={FiZap} w={5} h={5} color="blue.400" />
<Heading size="md" color="white">Test Connection</Heading>
</HStack>
</CardHeader>
<CardBody>
<VStack spacing={4} align="stretch">
<Text color="gray.400">
Test your {config.provider.toUpperCase()} configuration to ensure it's working properly before saving.
</Text>
<Button
leftIcon={isTesting ? <Spinner size="sm" /> : <FiZap />}
onClick={testConnection}
isLoading={isTesting}
loadingText="Testing..."
colorScheme="blue"
_hover={{ bg: "blue.700" }}
>
Test Connection
</Button>
{testResult && (
<Alert
status={testResult.success ? 'success' : 'error'}
variant="subtle"
flexDirection="column"
alignItems="center"
justifyContent="center"
textAlign="center"
height="auto"
py={4}
>
<AlertIcon boxSize="24px" />
<AlertTitle mt={2} mb={1} fontSize="lg">
{testResult.success ? 'Connection Successful' : 'Connection Failed'}
</AlertTitle>
<AlertDescription maxWidth="sm">
{testResult.message}
</AlertDescription>
{testResult.details && (
<Box mt={2} p={3} bg="gray.700" borderRadius="md" fontSize="sm">
<Text color="gray.300" fontWeight="bold">Details:</Text>
<Text color="gray.400" fontFamily="mono">
{JSON.stringify(testResult.details, null, 2)}
</Text>
</Box>
)}
</Alert>
)}
</VStack>
</CardBody>
</Card>
{/* Save Configuration */}
<Card bg="gray.800" borderColor="gray.700">
<CardHeader>
<HStack spacing={3}>
<Icon as={FiSave} w={5} h={5} color="green.400" />
<Heading size="md" color="white">Save Configuration</Heading>
</HStack>
</CardHeader>
<CardBody>
<VStack spacing={4} align="stretch">
<Text color="gray.400">
Save your {config.provider.toUpperCase()} configuration to use it for music file storage and playback.
</Text>
<HStack spacing={3}>
<Button
leftIcon={isSaving ? <Spinner size="sm" /> : <FiSave />}
onClick={saveConfiguration}
isLoading={isSaving}
loadingText="Saving..."
colorScheme="green"
isDisabled={!hasChanges()}
_hover={{ bg: "green.700" }}
>
Save Configuration
</Button>
{currentConfig && (
<Badge colorScheme="blue" variant="subtle" bg="blue.900" color="blue.200">
Configuration Loaded
</Badge>
)}
</HStack>
{!hasChanges() && currentConfig && (
<Alert status="info" variant="subtle">
<AlertIcon />
<Text color="gray.300">No changes to save</Text>
</Alert>
)}
</VStack>
</CardBody>
</Card>
{/* Help Section */}
<Card bg="gray.800" borderColor="gray.700">
<CardHeader>
<Heading size="md" color="white">Configuration Help</Heading>
</CardHeader>
<CardBody>
<VStack spacing={6} align="stretch">
{/* S3 Help */}
<Box>
<Text color="gray.400" fontWeight="bold" mb={2}>
<Icon as={FiCloud} w={4} h={4} mr={2} />
S3-Compatible Storage
</Text>
<Box pl={4} borderLeft="2px" borderColor="gray.600">
<Text color="gray.400" fontSize="sm" mb={2}>
<strong>For AWS S3:</strong>
</Text>
<Text color="gray.400" fontSize="sm" pl={2}>
• Endpoint: https://s3.amazonaws.com<br/>
• Region: Your AWS region (e.g., us-east-1)<br/>
• Access Key: Your AWS access key<br/>
• Secret Key: Your AWS secret key<br/>
• Bucket: Your S3 bucket name
</Text>
<Text color="gray.400" fontSize="sm" mt={3} mb={2}>
<strong>For MinIO (Local Development):</strong>
</Text>
<Text color="gray.400" fontSize="sm" pl={2}>
• Endpoint: http://localhost:9000<br/>
• Region: us-east-1<br/>
• Access Key: minioadmin<br/>
• Secret Key: minioadmin<br/>
• Bucket: Create a bucket named 'music-files'
</Text>
</Box>
</Box>
<Divider borderColor="gray.600" />
{/* WebDAV Help */}
<Box>
<Text color="gray.400" fontWeight="bold" mb={2}>
<Icon as={FiServer} w={4} h={4} mr={2} />
WebDAV (Nextcloud/ownCloud)
</Text>
<Box pl={4} borderLeft="2px" borderColor="gray.600">
<Text color="gray.400" fontSize="sm" mb={2}>
<strong>For Nextcloud:</strong>
</Text>
<Text color="gray.400" fontSize="sm" pl={2}>
• URL: https://your-nextcloud.com/remote.php/dav/files/username/<br/>
• Username: Your Nextcloud username<br/>
• Password: Your Nextcloud password or app password<br/>
• Base Path: /music-files (optional subfolder)
</Text>
<Text color="gray.400" fontSize="sm" mt={3} mb={2}>
<strong>For ownCloud:</strong>
</Text>
<Text color="gray.400" fontSize="sm" pl={2}>
• URL: https://your-owncloud.com/remote.php/dav/files/username/<br/>
• Username: Your ownCloud username<br/>
• Password: Your ownCloud password or app password<br/>
• Base Path: /music-files (optional subfolder)
</Text>
<Text color="gray.400" fontSize="sm" mt={3}>
<strong>Note:</strong> For better security, create an app password in your Nextcloud/ownCloud settings instead of using your main password.
</Text>
</Box>
</Box>
</VStack>
</CardBody>
</Card>
</VStack>
</Box>
);
};

16
packages/frontend/src/services/api.ts
View File

@ -142,7 +142,7 @@ class Api {
} }
// Background job methods // Background job methods
async startBackgroundJob(type: 'storage-sync' | 'song-matching', options?: any): Promise<{ jobId: string; type: string }> { async startBackgroundJob(type: 's3-sync' | 'song-matching', options?: any): Promise<{ jobId: string; type: string }> {
const response = await fetch(`${API_BASE_URL}/background-jobs/start`, { const response = await fetch(`${API_BASE_URL}/background-jobs/start`, {
method: 'POST', method: 'POST',
headers: { headers: {
@ -155,13 +155,13 @@ class Api {
return response.json(); return response.json();
} }
async startStorageSync(options?: { force?: boolean; clearLinks?: boolean }): Promise<{ jobId: string; type: string }> { async startS3Sync(options?: { force?: boolean; clearLinks?: boolean }): Promise<{ jobId: string; type: string }> {
const response = await fetch(`${API_BASE_URL}/music/sync-s3`, { const response = await fetch(`${API_BASE_URL}/music/sync-s3`, {
method: 'POST', method: 'POST',
headers: { 'Content-Type': 'application/json' }, headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(options || {}) body: JSON.stringify(options || {})
}); });
if (!response.ok) throw new Error('Failed to start storage sync'); if (!response.ok) throw new Error('Failed to start S3 sync');
return response.json(); return response.json();
} }
@ -205,7 +205,15 @@ class Api {
return response.json(); return response.json();
} }
// deleteDuplicateSongs method removed to keep WebDAV integration read-only async deleteDuplicateSongs(targetSongId: string, redundantSongIds: string[], deleteMusicFiles: boolean): Promise<{ deletedSongs: number; unlinkedOrDeletedMusicFiles: number; }>{
const response = await fetch(`${API_BASE_URL}/songs/delete-duplicates`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ targetSongId, redundantSongIds, deleteMusicFiles })
});
if (!response.ok) throw new Error('Failed to delete duplicates');
return response.json();
}
} }
export const api = new Api(); export const api = new Api();