244 lines
8.5 KiB
Markdown
244 lines
8.5 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Development Commands
|
|
|
|
```bash
|
|
# Install dependencies
|
|
npm run install:all
|
|
|
|
# Development (run in separate terminals)
|
|
npm run dev:backend # Starts on :3001 with tsx watch
|
|
npm run dev:frontend # Starts on :3000 with Vite HMR
|
|
|
|
# Within backend/ directory
|
|
cd backend
|
|
npm run dev # Development with tsx watch
|
|
npm run build # TypeScript compilation
|
|
npm start # Run compiled version
|
|
|
|
# Within frontend/ directory
|
|
cd frontend
|
|
npm run dev # Vite dev server
|
|
npm run build # TypeScript + Vite build
|
|
npm run preview # Preview production build
|
|
```
|
|
|
|
## Architecture Overview
|
|
|
|
VidRip is a monorepo with separate backend (Express/SQLite) and frontend (React) apps.
|
|
|
|
### Core Scheduler Logic
|
|
|
|
The scheduler in `backend/src/services/scheduler.ts` operates on two separate timers:
|
|
|
|
1. **Channel Checker**: Cron job (`0 * * * *`) runs every hour to check active channels for new videos
|
|
2. **Download Cycle**: Self-scheduling recursive setTimeout that:
|
|
- Downloads one pending video
|
|
- Calculates next run: `intervalHours * 60 ± random(varianceMinutes)`
|
|
- Schedules itself again with the calculated delay
|
|
|
|
The variance prevents predictable download patterns. The scheduler uses a single global `isDownloading` flag to prevent concurrent downloads.
|
|
|
|
### Database Layer
|
|
|
|
SQLite database (`backend/data.db`) with three tables managed via `better-sqlite3`:
|
|
- `channels`: YouTube channels to monitor
|
|
- `videos`: Individual videos with status tracking (pending/downloading/completed/failed)
|
|
- `config`: Key-value store for app settings
|
|
|
|
All database operations are in `backend/src/db/database.ts` using synchronous better-sqlite3 API. The database auto-initializes on first run.
|
|
|
|
### yt-dlp Integration
|
|
|
|
`backend/src/services/ytdlp.ts` wraps the yt-dlp CLI (must be installed on system):
|
|
- Uses `child_process.spawn` to call yt-dlp
|
|
- `--dump-json` extracts metadata without downloading
|
|
- `--flat-playlist` lists channel videos efficiently
|
|
- Progress tracking via stdout parsing (looks for percentage patterns)
|
|
- Downloads merge best video+audio to MP4 format
|
|
|
|
### Frontend Proxy Setup
|
|
|
|
Vite development server (port 3000) proxies to backend (port 3001):
|
|
- `/api/*` → Backend API endpoints
|
|
- `/downloads/*` → Static video files
|
|
|
|
This is configured in `frontend/vite.config.ts`. Production deployments need a reverse proxy (nginx, etc.).
|
|
|
|
### API Structure
|
|
|
|
Routes are split by domain:
|
|
- `backend/src/routes/channels.ts`: Channel CRUD + refresh
|
|
- `backend/src/routes/videos.ts`: Video CRUD + retry + streaming
|
|
- `backend/src/routes/config.ts`: Settings + scheduler control + WebDAV test
|
|
|
|
Video files are served via `/api/videos/:id/stream` endpoint which handles both local storage and WebDAV proxying.
|
|
|
|
## Key Constraints
|
|
|
|
- Only 1 concurrent download supported (hardcoded in scheduler)
|
|
- Download progress is ephemeral (not persisted to DB, only in-memory)
|
|
- No authentication/authorization
|
|
- yt-dlp must be in PATH (external dependency)
|
|
- Database uses WAL mode by default for better concurrency
|
|
|
|
## Important Implementation Notes
|
|
|
|
### Scheduler Restart Behavior
|
|
|
|
When config changes (e.g., interval or variance), call `restartScheduler()` from `backend/src/services/scheduler.ts`. This stops the cron task and clears the setTimeout chain, then restarts with new settings.
|
|
|
|
### Video Status State Machine
|
|
|
|
Videos progress through states: `pending → downloading → completed/failed`
|
|
- Retry functionality resets `failed` back to `pending`
|
|
- Deleting a channel cascades to delete all its videos and files
|
|
|
|
### Channel Refresh
|
|
|
|
Adding a channel fetches ALL videos from that channel using yt-dlp's `--flat-playlist`. Large channels (1000+ videos) may take time. Videos are created in `pending` state and downloaded according to the schedule.
|
|
|
|
## File Locations
|
|
|
|
- Database: `backend/data.db`
|
|
- Downloaded videos (local mode): `backend/downloads/{videoId}.mp4`
|
|
- Downloaded videos (WebDAV mode): Stored on WebDAV server, proxied through backend
|
|
- Environment: Optional `backend/.env` (PORT, NODE_ENV)
|
|
|
|
## System Requirements
|
|
|
|
- Node.js 18+
|
|
- yt-dlp installed and in PATH (`pip install yt-dlp` or `brew install yt-dlp`)
|
|
|
|
## Troubleshooting 403 Errors
|
|
|
|
If you encounter 403 Forbidden errors when downloading videos:
|
|
|
|
### Built-in Mitigations
|
|
|
|
The app automatically includes these anti-403 measures:
|
|
- Browser-like User-Agent headers
|
|
- YouTube referer
|
|
- Multiple retry attempts (5 extractor retries, 10 connection retries)
|
|
- Request throttling (1-5 second delays between requests)
|
|
- Fragment retries for long videos
|
|
|
|
### Additional Solution: Browser Cookies
|
|
|
|
For persistent 403 errors, export cookies from your browser:
|
|
|
|
1. **Install browser extension**:
|
|
- Chrome/Edge: "Get cookies.txt LOCALLY"
|
|
- Firefox: "cookies.txt"
|
|
|
|
2. **Export YouTube cookies**:
|
|
- Visit youtube.com while logged in
|
|
- Click extension icon and export cookies
|
|
- Save as `cookies.txt`
|
|
|
|
3. **Add to VidRip**:
|
|
- Place `cookies.txt` in `backend/` directory
|
|
- VidRip will automatically use it if present
|
|
- No restart needed
|
|
|
|
4. **Keep cookies fresh**:
|
|
- YouTube cookies expire periodically
|
|
- Re-export if 403 errors return
|
|
- Consider using an account with YouTube Premium to avoid restrictions
|
|
|
|
### Other Tips
|
|
|
|
- Update yt-dlp: `pip install --upgrade yt-dlp` (YouTube changes frequently)
|
|
- Increase download interval to avoid rate limiting
|
|
- Use variance to randomize download times
|
|
- Avoid downloading too many videos from the same channel quickly
|
|
|
|
## WebDAV Storage
|
|
|
|
VidRip supports storing videos on a WebDAV server instead of local storage. This is useful for:
|
|
- Network-attached storage (NAS) devices
|
|
- Cloud storage with WebDAV support (Nextcloud, ownCloud)
|
|
- Remote servers with WebDAV access
|
|
|
|
### Enabling WebDAV
|
|
|
|
1. Go to Settings page in the UI
|
|
2. Enable "WebDAV Storage"
|
|
3. Configure connection details:
|
|
- **Server URL**: Full WebDAV server URL (e.g., `https://nextcloud.example.com/remote.php/dav/files/username`)
|
|
- **Username**: WebDAV authentication username
|
|
- **Password**: WebDAV authentication password
|
|
- **Directory Path**: Subdirectory on server (default: `/vidrip/`)
|
|
4. Click "Test Connection" to verify settings
|
|
5. Save Settings
|
|
|
|
### How WebDAV Works
|
|
|
|
**Download Flow**:
|
|
1. Video is downloaded to `backend/downloads/` temporarily
|
|
2. After successful download, video is uploaded to WebDAV server
|
|
3. Local temporary file is deleted after successful upload
|
|
4. Database stores path as `webdav://{videoId}.mp4`
|
|
|
|
**Playback Flow**:
|
|
1. Frontend requests `/api/videos/:id/stream`
|
|
2. Backend detects WebDAV storage (path starts with `webdav://`)
|
|
3. Backend creates stream from WebDAV server
|
|
4. Stream is proxied to frontend (transparent to user)
|
|
|
|
**Deletion Flow**:
|
|
- When deleting a video, backend automatically removes from WebDAV server
|
|
- Database record is also deleted
|
|
|
|
### Supported WebDAV Servers
|
|
|
|
Tested with:
|
|
- **Nextcloud**: Use WebDAV URL format: `https://your-nextcloud.com/remote.php/dav/files/username`
|
|
- **ownCloud**: Similar to Nextcloud
|
|
- **Generic WebDAV**: Any RFC-compliant WebDAV server
|
|
|
|
### Configuration Details
|
|
|
|
WebDAV settings are stored in the `config` table:
|
|
- `webdavEnabled`: 'true' or 'false'
|
|
- `webdavUrl`: Server URL
|
|
- `webdavUsername`: Username
|
|
- `webdavPassword`: Password (stored in plaintext in database)
|
|
- `webdavPath`: Directory path on server
|
|
|
|
### Troubleshooting WebDAV
|
|
|
|
**Connection Test Fails**:
|
|
- Verify server URL is correct and accessible from backend server
|
|
- Check username/password are correct
|
|
- Ensure WebDAV path exists or backend has permission to create it
|
|
- Check firewall rules if server is remote
|
|
|
|
**Upload Fails**:
|
|
- Check available disk space on WebDAV server
|
|
- Verify write permissions on directory
|
|
- Check WebDAV server logs for detailed errors
|
|
- Ensure network is stable for large uploads
|
|
|
|
**Playback Issues**:
|
|
- Verify WebDAV server is accessible from backend
|
|
- Check network bandwidth between backend and WebDAV
|
|
- Browser console may show streaming errors
|
|
- Try downloading video directly from WebDAV to verify file integrity
|
|
|
|
**Migration**:
|
|
- Existing local videos remain in `backend/downloads/`
|
|
- New videos go to WebDAV when enabled
|
|
- To migrate existing videos, manually upload to WebDAV and update database `filePath`
|
|
- Or delete and re-download videos
|
|
|
|
### Security Considerations
|
|
|
|
- Credentials stored in local database (plaintext)
|
|
- Backend must have network access to WebDAV server
|
|
- Use HTTPS for WebDAV URL to encrypt credentials in transit
|
|
- Consider firewall rules to restrict WebDAV access to backend IP only
|
|
- WebDAV server should require authentication
|