Skip to content

About

High-performance document processing and semantic search pipeline

Resources

Stars

5 stars

Watchers

0 watching

Forks

Latest commit

Β 

History

3 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

πŸ” RAGProcessor

RAGProcessor is a high-performance document processing and semantic search pipeline. It automatically processes various document types (text, images, audio), creates optimized text chunks, generates embeddings using OpenAI models, and stores them in a Qdrant vector database for efficient semantic search.

πŸš€ Features

βœ… Multi-format Document Processing - Supports text (.txt, .md, .json, .csv, .html, .xml), code (.py, .js, .css), images (.png, .jpg, .jpeg, .webp), and audio/video files (.mp3, .mp4, .wav, .webm)
βœ… Intelligent Text Chunking - Advanced token-aware text splitting with 3-10x performance optimization through caching
βœ… Image Processing - Automatic image description using OpenAI Vision API (gpt-4.1-mini)
βœ… Audio Transcription - Audio/video content extraction using OpenAI Whisper API
βœ… Semantic Search - High-quality embeddings with OpenAI text-embedding-3-large model
βœ… Vector Storage - Efficient storage and retrieval using Qdrant vector database
βœ… Parallel Processing - Multi-threaded document processing for optimal performance
βœ… Directory Analysis - Smart file discovery with processing time estimation
βœ… Colored Logging - Enhanced console output with color-coded log levels (warnings in yellow, errors in red)
βœ… Performance Monitoring - Detailed statistics and cache performance metrics
βœ… Strict Token Limits - Configurable chunk size enforcement with 5% tolerance
βœ… Advanced Filtering - Support for metadata-based filtering in semantic search queries
βœ… Automatic Indexing - Smart payload field indexing for optimized query performance
βœ… Enhanced Metadata - Improved keyword extraction with list-based output format

πŸ›  Requirements

Install dependencies using Poetry (Python β‰₯ 3.10 recommended):

poetry install

Or with pip:

pip install qdrant-client openai python-dotenv tiktoken pydantic

Configure environment variables in .env file:

OPENAI_API_KEY=your_openai_api_key_here
QDRANT_URL=your_qdrant_url_here
QDRANT_API_KEY=your_qdrant_api_key_here  # Optional for local Qdrant

You will need:

  • OpenAI API access for embeddings and multimodal processing
  • Qdrant instance (local or cloud) for vector storage

πŸ“Ί How to use

Basic Usage

from ragprocessor import VectorStoreService

# Initialize the service
vector_store = VectorStoreService(
    collection_name="my_documents",
    chunk_size=500,
    embedding_model="text-embedding-3-large",
    verbose=True,
    strict_chunking=True
)

# Create collection
vector_store.create_collection()

# Process documents from directory
stats = vector_store.add_documents(
    documents_path="path/to/your/documents",
    batch_size=5,
    chunk_validation=True
)

# Semantic search with optional filtering
results = vector_store.query(
    query="What is the main topic discussed?",
    k=5,
    score_threshold=0.7,
    filter_values=[{"key": "keywords", "value": "technology"}]  # Optional metadata filtering
)

for result in results:
    print(f"Score: {result['score']:.4f}")
    print(f"Text: {result['text'][:200]}...")
    print(f"File: {result['file_name']}")

Advanced Filtering and Search

# Search with metadata filtering
results = vector_store.query(
    query="artificial intelligence topics",
    filter_values=[
        {"key": "keywords", "value": "AI"},
        {"key": "data_type", "value": "text/plain"}
    ],
    k=10,
    score_threshold=0.75
)

# Get collection information with index status
collection_info = vector_store.get_collection_info()
print(f"Collection has {collection_info['points_count']} documents")

Directory Analysis

# Analyze directory before processing
dir_info = vector_store.get_directory_info("path/to/documents")
print(f"Found {dir_info['total_files']} files ({dir_info['total_size_mb']:.2f} MB)")

# Estimate processing time
estimates = vector_store.estimate_processing_time("path/to/documents")
print(f"Estimated time: {estimates['estimated_processing_time_minutes']:.1f} minutes")

Advanced Text Splitting

from ragprocessor import TextSplitter

# Configure optimized text splitter
splitter = TextSplitter(
    model_name="text-embedding-3-large",
    max_chunk_size=500,
    enable_debug=False  # Enable for detailed logging
)

# Split text with performance optimization
chunks = splitter.split_text(text, strict_limit=True)

# Get chunking statistics
stats = splitter.get_chunking_stats(chunks)
print(f"Cache efficiency: {stats['cache_hit_rate']:.1f}%")

πŸ”Ή Project Structure

RAGProcessor/
β”œβ”€β”€ src/ragprocessor/
β”‚   β”œβ”€β”€ __init__.py                    # Package initialization
β”‚   β”œβ”€β”€ vector_store_service.py        # Main service class
β”‚   β”œβ”€β”€ text_splitter.py              # Optimized text chunking
β”‚   └── document_service.py           # Multi-format document processing
β”œβ”€β”€ prompts/
β”‚   └── describe_image.md              # Image analysis prompt template
β”œβ”€β”€ logs/                              # Application logs (timestamped)
β”œβ”€β”€ pyproject.toml                     # Poetry configuration
β”œβ”€β”€ README.md                          # This file
└── .env                              # Environment variables (not versioned)

πŸ“‚ Output

The system provides:

  • Semantic Search Results with relevance scores, text content, and metadata
  • Processing Statistics including chunk counts, token counts, and performance metrics
  • Colored Console Logs for better debugging experience
  • Detailed Analytics on file types, sizes, and processing estimates

🎯 Performance Features

  • Token Caching - 3-10x performance improvement through intelligent caching
  • Parallel Processing - Multi-threaded document handling
  • Memory Optimization - Efficient chunk size management with binary search algorithms
  • Smart Chunking - Context-aware text splitting respecting sentence and paragraph boundaries
  • Background Processing - Non-blocking operations for large document sets

πŸ“ Supported File Types

Category Extensions Processing Method
Text Documents .txt, .md, .json, .csv, .html, .xml, .rst Direct text extraction
Code Files .py, .js, .css, .es Syntax-aware processing
Images .png, .jpg, .jpeg, .webp OpenAI Vision API description
Audio/Video .mp3, .mp4, .wav, .webm, .m4a, .mpeg, .mpga OpenAI Whisper transcription

πŸš€ Planned Features & Roadmap

The following features are planned for future releases:

  • Support for PDF Files
    Add robust PDF parsing and chunking, including extraction of text, images, and metadata.
    Planned: Integration with libraries such as PyPDF2 or pdfplumber.

  • Support for Microsoft Word Documents (.docx)
    Enable processing and semantic search for Word files, including tables and embedded images.
    Planned: Use of python-docx for extraction.

  • Support for Other Binary Formats
    Expand to additional formats such as PowerPoint (.pptx), Excel (.xlsx), and OpenDocument files.

  • Advanced Metadata Extraction
    Extract and index document metadata (author, creation date, etc.) for improved search and filtering.

Feel free to suggest or remove any features from this roadmap according to your needs!

πŸ‘€ Author

Made with ❀️ by MichaΕ‚ KamiΕ„ski

🧾 License

This project is licensed under the MIT License.
You are free to use, modify, and distribute it as you wish.

πŸ“‹ Changelog

Version 1.0.1 (Latest)

Enhancements:

  • ✨ Enhanced VectorStoreService with improved index management and metadata filtering
  • πŸ”§ Updated Keyword Extraction - _get_keywords() method now returns a proper list instead of string
  • πŸš€ New Index Management - Added add_indexes() method for automatic payload field indexing
  • πŸ” Advanced Query Filtering - Modified query() method to support metadata-based filtering
  • πŸ“Š Improved Logging - Enhanced chunk statistics logging for better performance monitoring

Technical Changes:

  • Modified _get_keywords() method to return List[str] instead of comma-separated string
  • Added automatic payload indexing for fields: file_name, version_id, data_type, keywords, tokens, headers, urls, images, indexes
  • Enhanced query() method with filter_values parameter for metadata filtering
  • Improved performance logging with detailed chunk statistics

Version 1.0.0

  • πŸŽ‰ Initial stable release with full multi-format document processing
  • πŸš€ Complete vector storage and semantic search pipeline
  • πŸ“ Comprehensive documentation and examples

RAGProcessor v1.0.1 - High-Performance Document Processing & Semantic Search Pipeline

About

High-performance document processing and semantic search pipeline

Resources

Stars

5 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages