Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

57 Commits

Folders and files

Repository files navigation

Import Knowledge and Memory Portability (import-kb)

A utility package for importing distilled knowledge into a ChromaDB-based Long-Term Memory (LTM) system and backing up or restoring Omega user memory.

Purpose

The import-kb package is designed to bridge the gap between static knowledge files (JSONL, MeTTa) and an active agent's memory. It processes structured knowledge, generates vector embeddings, and upserts them into a ChromaDB collection, enabling semantic search and retrieval for AI agents.

The package also provides memory_portability, a programmatic interface for exporting and restoring Omega conversation history, user LTM records, skills, and workflow space. Omega Core remains responsible for its CLI, container lifecycle, transfer-directory mount, and user-facing decisions.

Supported Embedding Models

This package supports the following embedding modes:

  • OpenAI (Cloud):
    • Default model: text-embedding-3-large
    • High accuracy but requires an internet connection and an API key.
  • ASI Cloud:
    • Default model: WhereIsAI/UAE-Large-V1
    • Uses an OpenAI-compatible endpoint hosted by ASI Cloud, requires an API key.
  • SentenceTransformers (Local):
    • Default model: intfloat/e5-large-v2
    • Runs locally; the model may need to be downloaded before it can be used offline.
    • Can be configured to use any model compatible with the sentence-transformers library (e.g., all-MiniLM-L6-v2).

Installation

You can install the package directly from PyPI:

pip install import-kb

Or install it locally in editable mode:

git clone <repository-url>
cd import-knowledge-package
pip install -e .

Setup

Create a .env file in your project root or set the following environment variables:

  • OPENAI_API_KEY: Required if using OpenAI embeddings.
  • ASI_API_KEY: Required if using ASI Cloud embeddings (--provider asicloud).
  • CHROMA_DB_PATH: (Optional) Custom path to your Chroma database. Defaults to looking for /PeTTa/chroma_db or a local chroma_db folder.

How to Run

Command Line Interface (CLI)

After installation, you can run the import via the provided entry point:

# Use OpenAI embeddings (default)
import-knowledge

# Same as above, made explicit
import-knowledge --provider openai

# Override OpenAI model
import-knowledge --provider openai --model "text-embedding-3-small"

# Use ASI Cloud embeddings
import-knowledge --provider asicloud

# Override the ASI Cloud model
import-knowledge --provider asicloud --model "<model-name>"

# Use Local embeddings
import-knowledge --local

# Use a specific local model
import-knowledge --local --model "all-MiniLM-L6-v2"

Alternatively, run it as a module:

python3 -m import_knowledge.import_knowledge --local

Programmatic Usage

You can initialize the embedding system and embed text from your Python scripts:

from import_knowledge.import_knowledge import embed_batch, initLocalEmbedding

# Initialize for local use
initLocalEmbedding(model_name="intfloat/e5-large-v2")

# Generate embeddings for a batch of texts
embeddings = embed_batch(["Example text"])

To import the bundled knowledge files, use the CLI shown above. Its main() function parses command-line arguments and initializes the selected provider.

Memory portability

The host application owns memory configuration and must pass the resolved paths to the package explicitly:

from pathlib import Path

from memory_portability import MemoryStore, MemoryTransfer

store = MemoryStore(
    memory_dir=Path("/path/to/omega/memory"),
    chroma_path=Path("/path/to/chroma_db"),
    collection_name="memories",
    skills_dir=Path("/path/to/omega/memory/workflow/custom-skills"),
    space_dir=Path("/path/to/omega/memory/workflow/space"),
)
transfer = MemoryTransfer(
    transfer_dir=Path("/path/to/memory-transfer"),
    store=store,
)

exported = transfer.export(component="all")
transfer.import_archive(exported["filename"])
transfer.recover()

MemoryStore does not infer Omega paths or read them from environment variables. Resolve these values in the host application's configuration layer.

The default export component is now "all" (history, LTM, and skills), replacing the previous "both" option. Both skills_dir and space_dir must be configured for export(), export("all"), or export("skills"); missing configuration raises ValueError before any export directories are created or memory is read. These paths remain optional for history-only and LTM-only operations: use export("history") or export("ltm"). To migrate an application that previously exported both history and LTM, configure both workflow paths and use "all", or export history and LTM separately if workflow storage is not available.

Skills imports reject symlinks in either destination root or its parents. Append also rejects symlinks at incoming file and directory paths before changing memory. Overwrite safely removes existing trees, including nested symlinks, and rollback preserves those links without writing to their targets.

Default exports use omega-memory-<UTC timestamp>-<ID>.tar.gz, where the ID is the first eight hexadecimal characters of a UUIDv4.

Dependencies

  • openai: For cloud-based embeddings.
  • sentence-transformers: For local, offline embeddings.
  • chromadb: Vector database for storage.
  • python-dotenv: Management of environment variables.
  • tqdm: Progress bars for batch processing.

Publishing to PyPI

The publishing workflow builds and validates a source distribution and wheel when a GitHub Release is published, then uploads them to PyPI using Trusted Publishing. No PyPI API token or GitHub secret is needed.

One-time setup:

  1. Create a GitHub environment named pypi in the repository settings. Configure required reviewers if releases should require approval before uploading.
  2. In the PyPI import-kb project's Publishing settings, add a GitHub trusted publisher with the following values:
    • Owner: the GitHub user or organization that will publish releases (blackhammer116 for this fork, or singnet for the upstream repository).
    • Repository: import-knowledge-package.
    • Workflow filename: publish.yml.
    • Environment: pypi.

To release, update project.version in pyproject.toml to an unused version, commit the change, and publish a GitHub Release tagged with that version (for example, v0.2.5 or 0.2.5). The tag must point to the commit containing the version update and this workflow. Draft releases and tag pushes alone do not publish; publishing a GitHub prerelease also triggers the workflow, so use a corresponding Python prerelease version such as 0.3.0rc1.

The workflow rejects tags that do not match project.version. PyPI does not allow an existing release file to be overwritten; use a new version for each release.

License

This project is licensed under the Apache License 2.0. See LICENSE for details.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages