Documentation Index Fetch the complete documentation index at: https://docs.grigori.in/llms.txt
Use this file to discover all available pages before exploring further.
System Requirements
Python Python 3.12 or higher required
Memory Minimum 2GB RAM, 4GB+ recommended
Storage 1GB for models, additional space for processing
Redis Redis 6.0+ for job queue and storage
Installation Methods Docker (Recommended)
uv (Development)
pip (Traditional)
The easiest way to get started is with Docker Compose:
Clone Repository
git clone https://github.com/pratyush618/doc_loader.git cd doc-converter
Configure Environment
Edit the .env file: # Basic Configuration REDIS_URL = redis://redis:6379/0 MAX_FILE_SIZE = 104857600 # 100MB # OCR Configuration (optional) MISTRAL_API_KEY = your_api_key_here EASY_OCR_USE_GPU = false PADDLE_OCR_USE_GPU = false # Storage UPLOAD_DIR = ./uploads OUTPUT_DIR = ./outputs
Start Services
This starts:
API server on port 8000
Redis server
Celery worker
Health monitoring
Verify Installation
curl http://localhost:8000/api/v1/health
Expected response: { "status" : "healthy" , "timestamp" : "2024-01-15T10:30:00Z" , "version" : "1.0.0" }
For development with the fast uv package manager:
Install uv
winget install --id=astral-sh.uv -e
Setup Project
git clone https://github.com/pratyush618/doc_loader.git cd doc-converter uv sync # Creates virtual environment and installs dependencies
Configure Environment
cp .env.example .env # Edit .env file as needed
Start Redis
brew services start redis
Run Services
# Terminal 1: API Server uv run python run_api.py # Terminal 2: Worker uv run python run_worker.py
Traditional Python installation:
Create Virtual Environment
python -m venv venv # Activate virtual environment # Windows venv\Scripts\activate # macOS/Linux source venv/bin/activate
Install Dependencies
pip install -r requirements.txt
Setup Environment
cp .env.example .env # Configure .env file
Start Services
# Start Redis first redis-server # Then start API and worker python run_api.py # Terminal 1 python run_worker.py # Terminal 2
Environment Configuration
# Application APP_NAME = doc-converter APP_ENV = production DEBUG = false # API Configuration API_HOST = 0.0.0.0 API_PORT = 8000 API_PREFIX = /api/v1 # Security SECRET_KEY = your-secret-key-here ALLOWED_ORIGINS = *
# Redis Connection REDIS_URL = redis://localhost:6379/0 # Celery Configuration CELERY_BROKER_URL = redis://localhost:6379/0 CELERY_RESULT_BACKEND = redis://localhost:6379/0 CELERY_TASK_SERIALIZER = json CELERY_RESULT_SERIALIZER = json CELERY_ACCEPT_CONTENT = ["json" ] CELERY_TIMEZONE = UTC
# Storage Paths UPLOAD_DIR = ./uploads OUTPUT_DIR = ./outputs # File Limits MAX_FILE_SIZE = 104857600 # 100MB in bytes FILE_TTL_HOURS = 24 # How long to keep files # Image Processing IMAGE_COMPRESSION_QUALITY = 95 IMAGE_MAX_WIDTH = 2048 IMAGE_MAX_HEIGHT = 2048
# PaddleOCR PADDLE_OCR_USE_GPU = false PADDLE_OCR_LANG = en PADDLE_MODELS_DIR = ./models/paddle_models # EasyOCR EASY_OCR_USE_GPU = false EASY_OCR_LANG = en EASY_OCR_MODEL_STORAGE = ./models/easy_ocr_models EASY_OCR_TEXT_THRESHOLD = 0.7 EASY_OCR_LINK_THRESHOLD = 0.4 EASY_OCR_LOW_TEXT = 0.4 # Mistral AI MISTRAL_API_KEY = your_api_key_here MISTRAL_API_URL = https://api.mistral.ai/v1/chat/completions MISTRAL_MODEL = pixtral-12b-2409
# Webhook Configuration DEFAULT_WEBHOOK_URL = https://your-site.com/webhook WEBHOOK_TIMEOUT = 30 WEBHOOK_MAX_RETRIES = 3
Model Downloads The application will automatically download required models on first use:
PaddleOCR Models Downloaded to ./models/paddle_models/ Size: ~200MB
EasyOCR Models Downloaded to ./models/easy_ocr_models/ Size: ~150MB per language
Mistral AI API-based, no local models Requires API key
Verification After installation, verify everything is working:
Health Check
curl http://localhost:8000/api/v1/health
Should return: { "status" : "healthy" , "timestamp" : "2024-01-15T10:30:00Z" , "version" : "1.0.0" }
Readiness Check
curl http://localhost:8000/api/v1/ready
Should return: { "status" : "ready" , "services" : { "redis" : true , "celery" : true } }
Test Conversion
# Create a test file echo "Hello, World!" > test.txt # Convert it curl -X POST "http://localhost:8000/api/v1/jobs" \ -F "[email protected] " \ -F "output_format=md"
Should return job details with a valid ID. Prerequisites:
Windows 10/11 or Windows Server 2019+
Visual Studio Build Tools (for some dependencies)
Redis for Windows (or use Docker)
Common Issues:
PyTorch installation may require specific versions
Use pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu for CPU-only
EasyOCR may have compatibility issues - PaddleOCR is recommended
Performance Tips:
Use WSL2 for better performance
Consider Docker Desktop for easier Redis setup
Prerequisites:
macOS 10.15+ (Catalina or later)
Xcode Command Line Tools: xcode-select --install
Homebrew for package management
Installation Commands: # Install system dependencies brew install redis [email protected] # Start Redis brew services start redis
M1/M2 Macs:
Use native ARM builds when available
Some ML dependencies may require Rosetta 2
Prerequisites:
Ubuntu 20.04+ / CentOS 8+ / Debian 11+
Python 3.12+ development headers
Redis server
Ubuntu/Debian: sudo apt update sudo apt install python3.12 python3.12-dev python3.12-venv sudo apt install redis-server sudo systemctl start redis sudo systemctl enable redis
CentOS/RHEL: sudo dnf install python3.12 python3.12-devel sudo dnf install redis sudo systemctl start redis sudo systemctl enable redis
CPU Optimization # Set CPU-specific optimizations export OMP_NUM_THREADS = 4 export MKL_NUM_THREADS = 4 # For production export CELERY_WORKER_CONCURRENCY = 2
Memory Management # Limit memory usage export MALLOC_ARENA_MAX = 2 # For containers export CELERY_WORKER_MAX_MEMORY_PER_CHILD = 500000
GPU Acceleration # Enable GPU for OCR (if available) EASY_OCR_USE_GPU = true PADDLE_OCR_USE_GPU = true # Requires CUDA installation
File System # Use SSD for better performance UPLOAD_DIR = /fast/storage/uploads OUTPUT_DIR = /fast/storage/outputs # Clean up old files FILE_TTL_HOURS = 12
Troubleshooting
Symptoms: redis.exceptions.ConnectionErrorSolutions:
Check if Redis is running: redis-cli ping
Verify Redis URL in .env
Check firewall settings
For Docker: ensure Redis container is running
# Test Redis connection redis-cli -u $REDIS_URL ping
OCR Installation Problems
Symptoms: OCR providers fail to initializeSolutions:
EasyOCR : Often PyTorch compatibility issuespip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
PaddleOCR : Model download issues# Pre-download models python -c "from paddleocr import PaddleOCR; PaddleOCR(use_angle_cls=True, lang='en')"
Mistral : API key issues# Test API key curl -H "Authorization: Bearer $MISTRAL_API_KEY " https://api.mistral.ai/v1/models
Symptoms: 413 Request Entity Too LargeSolutions:
Increase file size limit in .env
Check web server limits (nginx, Apache)
Verify disk space
# Check current limits curl -v http://localhost:8000/api/v1/health
Symptoms: Worker crashes, out of memory errorsSolutions:
Reduce worker concurrency
Increase system memory
Enable memory limits
# Monitor memory usage docker stats # For Docker htop # For native installation
Next Steps
Test Your Installation Run your first document conversion
Configure OCR Set up OCR providers for better text extraction
Production Deployment Deploy to production with monitoring
API Documentation Explore the complete API reference
