RAGProcessor is a high-performance document processing and semantic search pipeline. It automatically processes various document types (text, images, audio), creates optimized text chunks, generates embeddings using OpenAI models, and stores them in a Qdrant vector database for efficient semantic search.
β
Multi-format Document Processing - Supports text (.txt, .md, .json, .csv, .html, .xml), code (.py, .js, .css), images (.png, .jpg, .jpeg, .webp), and audio/video files (.mp3, .mp4, .wav, .webm)
β
Intelligent Text Chunking - Advanced token-aware text splitting with 3-10x performance optimization through caching
β
Image Processing - Automatic image description using OpenAI Vision API (gpt-4.1-mini)
β
Audio Transcription - Audio/video content extraction using OpenAI Whisper API
β
Semantic Search - High-quality embeddings with OpenAI text-embedding-3-large model
β
Vector Storage - Efficient storage and retrieval using Qdrant vector database
β
Parallel Processing - Multi-threaded document processing for optimal performance
β
Directory Analysis - Smart file discovery with processing time estimation
β
Colored Logging - Enhanced console output with color-coded log levels (warnings in yellow, errors in red)
β
Performance Monitoring - Detailed statistics and cache performance metrics
β
Strict Token Limits - Configurable chunk size enforcement with 5% tolerance
β
Advanced Filtering - Support for metadata-based filtering in semantic search queries
β
Automatic Indexing - Smart payload field indexing for optimized query performance
β
Enhanced Metadata - Improved keyword extraction with list-based output format
Install dependencies using Poetry (Python β₯ 3.10 recommended):
poetry installOr with pip:
pip install qdrant-client openai python-dotenv tiktoken pydanticConfigure environment variables in .env file:
OPENAI_API_KEY=your_openai_api_key_here
QDRANT_URL=your_qdrant_url_here
QDRANT_API_KEY=your_qdrant_api_key_here # Optional for local QdrantYou will need:
- OpenAI API access for embeddings and multimodal processing
- Qdrant instance (local or cloud) for vector storage
from ragprocessor import VectorStoreService
# Initialize the service
vector_store = VectorStoreService(
collection_name="my_documents",
chunk_size=500,
embedding_model="text-embedding-3-large",
verbose=True,
strict_chunking=True
)
# Create collection
vector_store.create_collection()
# Process documents from directory
stats = vector_store.add_documents(
documents_path="path/to/your/documents",
batch_size=5,
chunk_validation=True
)
# Semantic search with optional filtering
results = vector_store.query(
query="What is the main topic discussed?",
k=5,
score_threshold=0.7,
filter_values=[{"key": "keywords", "value": "technology"}] # Optional metadata filtering
)
for result in results:
print(f"Score: {result['score']:.4f}")
print(f"Text: {result['text'][:200]}...")
print(f"File: {result['file_name']}")# Search with metadata filtering
results = vector_store.query(
query="artificial intelligence topics",
filter_values=[
{"key": "keywords", "value": "AI"},
{"key": "data_type", "value": "text/plain"}
],
k=10,
score_threshold=0.75
)
# Get collection information with index status
collection_info = vector_store.get_collection_info()
print(f"Collection has {collection_info['points_count']} documents")# Analyze directory before processing
dir_info = vector_store.get_directory_info("path/to/documents")
print(f"Found {dir_info['total_files']} files ({dir_info['total_size_mb']:.2f} MB)")
# Estimate processing time
estimates = vector_store.estimate_processing_time("path/to/documents")
print(f"Estimated time: {estimates['estimated_processing_time_minutes']:.1f} minutes")from ragprocessor import TextSplitter
# Configure optimized text splitter
splitter = TextSplitter(
model_name="text-embedding-3-large",
max_chunk_size=500,
enable_debug=False # Enable for detailed logging
)
# Split text with performance optimization
chunks = splitter.split_text(text, strict_limit=True)
# Get chunking statistics
stats = splitter.get_chunking_stats(chunks)
print(f"Cache efficiency: {stats['cache_hit_rate']:.1f}%")RAGProcessor/
βββ src/ragprocessor/
β βββ __init__.py # Package initialization
β βββ vector_store_service.py # Main service class
β βββ text_splitter.py # Optimized text chunking
β βββ document_service.py # Multi-format document processing
βββ prompts/
β βββ describe_image.md # Image analysis prompt template
βββ logs/ # Application logs (timestamped)
βββ pyproject.toml # Poetry configuration
βββ README.md # This file
βββ .env # Environment variables (not versioned)
The system provides:
- Semantic Search Results with relevance scores, text content, and metadata
- Processing Statistics including chunk counts, token counts, and performance metrics
- Colored Console Logs for better debugging experience
- Detailed Analytics on file types, sizes, and processing estimates
- Token Caching - 3-10x performance improvement through intelligent caching
- Parallel Processing - Multi-threaded document handling
- Memory Optimization - Efficient chunk size management with binary search algorithms
- Smart Chunking - Context-aware text splitting respecting sentence and paragraph boundaries
- Background Processing - Non-blocking operations for large document sets
| Category | Extensions | Processing Method |
|---|---|---|
| Text Documents | .txt, .md, .json, .csv, .html, .xml, .rst | Direct text extraction |
| Code Files | .py, .js, .css, .es | Syntax-aware processing |
| Images | .png, .jpg, .jpeg, .webp | OpenAI Vision API description |
| Audio/Video | .mp3, .mp4, .wav, .webm, .m4a, .mpeg, .mpga | OpenAI Whisper transcription |
The following features are planned for future releases:
-
Support for PDF Files
Add robust PDF parsing and chunking, including extraction of text, images, and metadata.
Planned: Integration with libraries such as PyPDF2 or pdfplumber. -
Support for Microsoft Word Documents (.docx)
Enable processing and semantic search for Word files, including tables and embedded images.
Planned: Use of python-docx for extraction. -
Support for Other Binary Formats
Expand to additional formats such as PowerPoint (.pptx), Excel (.xlsx), and OpenDocument files. -
Advanced Metadata Extraction
Extract and index document metadata (author, creation date, etc.) for improved search and filtering.
Feel free to suggest or remove any features from this roadmap according to your needs!
Made with β€οΈ by MichaΕ KamiΕski
This project is licensed under the MIT License.
You are free to use, modify, and distribute it as you wish.
Enhancements:
- β¨ Enhanced VectorStoreService with improved index management and metadata filtering
- π§ Updated Keyword Extraction -
_get_keywords()method now returns a proper list instead of string - π New Index Management - Added
add_indexes()method for automatic payload field indexing - π Advanced Query Filtering - Modified
query()method to support metadata-based filtering - π Improved Logging - Enhanced chunk statistics logging for better performance monitoring
Technical Changes:
- Modified
_get_keywords()method to returnList[str]instead of comma-separated string - Added automatic payload indexing for fields:
file_name,version_id,data_type,keywords,tokens,headers,urls,images,indexes - Enhanced
query()method withfilter_valuesparameter for metadata filtering - Improved performance logging with detailed chunk statistics
- π Initial stable release with full multi-format document processing
- π Complete vector storage and semantic search pipeline
- π Comprehensive documentation and examples
RAGProcessor v1.0.1 - High-Performance Document Processing & Semantic Search Pipeline