vidrip/PROJECT_CONTEXT.md

# VidRip - Project Context

## Project Overview
VidRip is a respectful YouTube video drip downloader that slowly downloads videos from channels over time to avoid aggressive scraping. Videos are downloaded at configurable intervals with random variance to make the pattern less predictable.

## Architecture

### Tech Stack
- **Backend**: Node.js, Express, TypeScript, SQLite (better-sqlite3), node-cron
- **Frontend**: React, TypeScript, Vite, Tailwind CSS, React Router
- **External**: yt-dlp (must be installed on system)

### Project Structure
```
vidrip/
├── backend/
│   ├── src/
│   │   ├── db/
│   │   │   └── database.ts          # SQLite operations, schema, CRUD
│   │   ├── routes/
│   │   │   ├── channels.ts          # Channel management endpoints
│   │   │   ├── videos.ts            # Video management endpoints
│   │   │   └── config.ts            # Settings & scheduler status
│   │   ├── services/
│   │   │   ├── ytdlp.ts             # yt-dlp wrapper for downloads
│   │   │   └── scheduler.ts         # Download scheduler logic
│   │   ├── types/
│   │   │   └── index.ts             # TypeScript interfaces
│   │   └── server.ts                # Express app entry point
│   ├── downloads/                   # Downloaded video storage
│   ├── data.db                      # SQLite database (auto-created)
│   └── package.json
├── frontend/
│   ├── src/
│   │   ├── components/              # (empty - components in pages)
│   │   ├── pages/
│   │   │   ├── VideosPage.tsx       # List/add/filter videos
│   │   │   ├── ChannelsPage.tsx     # Manage channels
│   │   │   ├── SettingsPage.tsx     # Configure scheduler
│   │   │   └── VideoPlayerPage.tsx  # Watch videos
│   │   ├── services/
│   │   │   └── api.ts               # API client functions
│   │   ├── types/
│   │   │   └── index.ts             # TypeScript interfaces
│   │   ├── App.tsx                  # Router & navigation
│   │   ├── main.tsx                 # React entry point
│   │   └── index.css                # Tailwind imports
│   ├── index.html
│   ├── vite.config.ts               # Proxy to backend
│   └── package.json
└── README.md
```

## Database Schema

### Tables
1. **channels**
   - id, url, name, channelId, addedAt, lastChecked, active
   - Stores YouTube channels to monitor

2. **videos**
   - id, channelId, videoId, title, url, duration, thumbnail, uploadDate
   - status (pending/downloading/completed/failed), filePath, fileSize
   - addedAt, downloadedAt, error
   - Stores individual videos and download status

3. **config**
   - id, key, value
   - Stores app configuration (intervalHours, varianceMinutes, enabled, etc.)

## Key Flows

### Download Scheduler (`backend/src/services/scheduler.ts`)
1. **Channel Checking**: Runs every hour via cron to check channels for new videos
2. **Download Cycle**:
   - Picks next pending video
   - Downloads with progress tracking
   - Marks completed or failed
   - Schedules next download using: `intervalHours ± random(varianceMinutes)`
3. **Restartable**: Can be stopped/restarted when config changes

### Adding a Channel
1. User enters YouTube channel URL in frontend
2. Backend calls `getChannelInfo()` to fetch channel metadata
3. Backend calls `getChannelVideos()` to get all videos (flat playlist)
4. Creates channel record and all video records with status='pending'
5. Scheduler will pick them up based on configuration

### Video Download
1. Scheduler picks next pending video
2. Calls yt-dlp with progress callbacks
3. Downloads to `backend/downloads/{videoId}.mp4`
4. Updates database with file path, size, status

## API Endpoints

### Channels
- `GET /api/channels` - List all channels
- `GET /api/channels/:id` - Get channel details
- `GET /api/channels/:id/videos` - Get channel's videos
- `POST /api/channels` - Add new channel (body: {url})
- `PATCH /api/channels/:id` - Update channel (body: {active})
- `DELETE /api/channels/:id` - Delete channel & videos
- `POST /api/channels/:id/refresh` - Check for new videos

### Videos
- `GET /api/videos` - List all videos (optional ?status=)
- `GET /api/videos/:id` - Get video details
- `POST /api/videos` - Add single video (body: {url})
- `DELETE /api/videos/:id` - Delete video & file
- `POST /api/videos/:id/retry` - Retry failed video

### Config
- `GET /api/config` - Get all settings
- `PATCH /api/config` - Update settings (body: {key: value})
- `GET /api/config/scheduler/status` - Get scheduler status & progress

### Static
- `/downloads/{videoId}.mp4` - Serve downloaded videos

## Configuration Options

Default values in database:
- `intervalHours`: "3" - Average hours between downloads
- `varianceMinutes`: "30" - Random ± minutes to add
- `maxConcurrentDownloads`: "1" - Simultaneous downloads (currently only 1 supported)
- `enabled`: "true" - Whether scheduler is active

## Important Implementation Details

### yt-dlp Integration
- Uses spawn to call yt-dlp CLI (must be in PATH)
- `--dump-json` for metadata extraction
- `--flat-playlist` for channel video lists
- Progress parsing via stdout line parsing
- Downloads as MP4 (merges best video+audio)

### Scheduler Variance
- Prevents predictable download patterns
- Calculates: `baseMinutes = intervalHours * 60`
- Adds random: `variance = random(-varianceMinutes, +varianceMinutes)`
- Uses `setTimeout` for next download, not fixed cron

### Frontend Proxy
- Vite proxies `/api` and `/downloads` to backend (port 3001)
- Allows development without CORS issues
- Production would need reverse proxy (nginx/etc)

## Known Limitations & Future Enhancements

### Current Limitations
1. Only 1 concurrent download supported (hardcoded in scheduler)
2. No authentication/authorization
3. No video queue reordering
4. No bandwidth limiting
5. No retry limit for failed videos
6. No disk space checking
7. No video preview before download
8. Progress tracking only works during active download (not persisted)

### Potential Enhancements
- [ ] User authentication
- [ ] Video quality selection
- [ ] Download queue prioritization
- [ ] Bandwidth throttling
- [ ] Automatic old video cleanup
- [ ] Video search/filtering by title
- [ ] Channel categorization/tagging
- [ ] Download history/statistics
- [ ] Webhook notifications
- [ ] Docker containerization
- [ ] Multiple download quality profiles
- [ ] Subtitle downloading

## Development Commands

```bash
# Install all dependencies
npm run install:all

# Development (run in separate terminals)
npm run dev:backend    # Backend on :3001
npm run dev:frontend   # Frontend on :3000

# Production build
npm run build:backend
npm run build:frontend

# Production run
npm run start:backend
npm run start:frontend
```

## Dependencies to Install

### System Requirements
- Node.js 18+
- yt-dlp (via pip, brew, or binary)

### Installation
```bash
# yt-dlp
pip install yt-dlp
# or
brew install yt-dlp

# Node dependencies
npm run install:all
```

## Troubleshooting

### Common Issues
1. **"yt-dlp not found"**: Ensure yt-dlp is in PATH
2. **Database locked**: Only one backend instance should run
3. **Video won't play**: Check file exists in `backend/downloads/`
4. **Scheduler not running**: Check settings page, ensure enabled=true
5. **Channel refresh fails**: YouTube may be rate limiting, wait and retry

## File Locations

- **Database**: `backend/data.db`
- **Videos**: `backend/downloads/{videoId}.mp4`
- **Logs**: Console output (not persisted to file currently)

## Security Considerations

- No authentication - anyone with access can manage downloads
- Videos stored unencrypted on filesystem
- No rate limiting on API endpoints
- YouTube URLs not validated before passing to yt-dlp
- Consider running behind reverse proxy with auth in production

## Code Entry Points

To understand the codebase quickly:
1. Start with `backend/src/server.ts` - see how routes connect
2. Read `backend/src/services/scheduler.ts` - core business logic
3. Check `frontend/src/App.tsx` - understand page routing
4. Review `backend/src/db/database.ts` - database schema & operations

## Testing

Currently no automated tests. Manual testing checklist:
- [ ] Add channel and verify videos appear
- [ ] Add individual video by URL
- [ ] Watch completed video
- [ ] Change settings and verify scheduler restarts
- [ ] Delete channel and verify cascade delete
- [ ] Retry failed video
- [ ] Refresh channel for new videos

## Environment Variables

Backend supports (optional):
- `PORT` - Server port (default: 3001)
- `NODE_ENV` - Environment (development/production)

Currently no `.env` file required - uses defaults.