C3PO - INFO to Text Converter

A powerful and efficient tool for converting INFO format files to clean text format while preserving content integrity.

Features

• Efficient Processing: Handles large files with parallel processing and automatic chunking
• Content Validation: Ensures the converted output maintains content integrity with detailed metrics
• Format Cleaning: Removes headers, separators, menu items, and control characters
• Semantic Markup: Optional feature to identify and mark semantic structures (tables, code blocks, lists, headings)
• Comprehensive Logging: Detailed logging system with configurable levels and color-coded output
• Customizable: Adjustable parameters for processing and validation

Installation

Using uv (Recommended)

# Clone the repository
git clone https://github.com/yourusername/c3po.git
cd c3po

# Create and activate virtual environment with uv
uv venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
uv pip install -r requirements.txt

Using pip

# Clone the repository
git clone https://github.com/yourusername/c3po.git
cd c3po

# Create and activate virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

Usage

python c3po.py input_file.info output_file.txt [options]

Options

• --no-verify: Skip content validation
• --workers=N: Set the number of worker threads (default: CPU count)
• --debug: Enable debug mode
• --log=LEVEL: Set log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
• --enable-semantic-markers: Enable semantic structure identification and marking

Examples

Basic conversion:

python c3po.py document.info document.txt

Convert with 4 worker threads and skip validation:

python c3po.py large_document.info large_document.txt --workers=4 --no-verify

Convert with semantic markup:

python c3po.py document.info document.txt --enable-semantic-markers

Convert with debug mode and detailed logging:

python c3po.py document.info document.txt --debug --log=DEBUG

Content Validation

The tool validates converted content using several metrics:

• Paragraph Ratio: Ensures paragraph structure is maintained (ideal: 0.5-1.5)
• Content Length Ratio: Checks overall content length preservation (ideal: 0.7-1.2)
• Keyword Retention: Verifies important keywords are preserved (ideal: >25%)
• Content Similarity: Samples content to ensure similarity (ideal: >10%)

Semantic Markup

When enabled, the semantic processor identifies and marks various structures in the text:

• SQL Code Blocks: Identifies and formats SQL code sections
• Tables: Detects tabular data and preserves its structure
• Lists: Recognizes and formats bullet points and numbered lists
• Headings: Identifies section headings and maintains hierarchy

Performance

• Processes files at approximately 1.5MB/second on modern hardware
• Memory usage scales efficiently with file size through chunked processing
• Parallel processing utilizes available CPU cores for optimal performance
• Automatic chunk size adjustment based on file size and available memory

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• The tqdm library for progress bar functionality
• Python’s concurrent.futures for parallel processing capabilities