A utility package for importing distilled knowledge into a ChromaDB-based Long-Term Memory (LTM) system and backing up or restoring Omega user memory.
The import-kb package is designed to bridge the gap between static knowledge files (JSONL, MeTTa) and an active agent's memory. It processes structured knowledge, generates vector embeddings, and upserts them into a ChromaDB collection, enabling semantic search and retrieval for AI agents.
The package also provides memory_portability, a programmatic interface for exporting and restoring Omega conversation history, user LTM records, skills, and workflow space. Omega Core remains responsible for its CLI, container lifecycle, transfer-directory mount, and user-facing decisions.
This package supports the following embedding modes:
- OpenAI (Cloud):
- Default model:
text-embedding-3-large - High accuracy but requires an internet connection and an API key.
- Default model:
- ASI Cloud:
- Default model:
WhereIsAI/UAE-Large-V1 - Uses an OpenAI-compatible endpoint hosted by ASI Cloud, requires an API key.
- Default model:
- SentenceTransformers (Local):
- Default model:
intfloat/e5-large-v2 - Runs locally; the model may need to be downloaded before it can be used offline.
- Can be configured to use any model compatible with the
sentence-transformerslibrary (e.g.,all-MiniLM-L6-v2).
- Default model:
You can install the package directly from PyPI:
pip install import-kbOr install it locally in editable mode:
git clone <repository-url>
cd import-knowledge-package
pip install -e .Create a .env file in your project root or set the following environment variables:
OPENAI_API_KEY: Required if using OpenAI embeddings.ASI_API_KEY: Required if using ASI Cloud embeddings (--provider asicloud).CHROMA_DB_PATH: (Optional) Custom path to your Chroma database. Defaults to looking for/PeTTa/chroma_dbor a localchroma_dbfolder.
After installation, you can run the import via the provided entry point:
# Use OpenAI embeddings (default)
import-knowledge
# Same as above, made explicit
import-knowledge --provider openai
# Override OpenAI model
import-knowledge --provider openai --model "text-embedding-3-small"
# Use ASI Cloud embeddings
import-knowledge --provider asicloud
# Override the ASI Cloud model
import-knowledge --provider asicloud --model "<model-name>"
# Use Local embeddings
import-knowledge --local
# Use a specific local model
import-knowledge --local --model "all-MiniLM-L6-v2"Alternatively, run it as a module:
python3 -m import_knowledge.import_knowledge --localYou can initialize the embedding system and embed text from your Python scripts:
from import_knowledge.import_knowledge import embed_batch, initLocalEmbedding
# Initialize for local use
initLocalEmbedding(model_name="intfloat/e5-large-v2")
# Generate embeddings for a batch of texts
embeddings = embed_batch(["Example text"])To import the bundled knowledge files, use the CLI shown above. Its main()
function parses command-line arguments and initializes the selected provider.
The host application owns memory configuration and must pass the resolved paths to the package explicitly:
from pathlib import Path
from memory_portability import MemoryStore, MemoryTransfer
store = MemoryStore(
memory_dir=Path("/path/to/omega/memory"),
chroma_path=Path("/path/to/chroma_db"),
collection_name="memories",
skills_dir=Path("/path/to/omega/memory/workflow/custom-skills"),
space_dir=Path("/path/to/omega/memory/workflow/space"),
)
transfer = MemoryTransfer(
transfer_dir=Path("/path/to/memory-transfer"),
store=store,
)
exported = transfer.export(component="all")
transfer.import_archive(exported["filename"])
transfer.recover()MemoryStore does not infer Omega paths or read them from environment
variables. Resolve these values in the host application's configuration layer.
The default export component is now "all" (history, LTM, and skills), replacing
the previous "both" option. Both skills_dir and space_dir must be configured
for export(), export("all"), or export("skills"); missing configuration raises
ValueError before any export directories are created or memory is read.
These paths remain optional for history-only and LTM-only operations: use
export("history") or export("ltm"). To migrate an application that previously
exported both history and LTM, configure both workflow paths and use "all", or
export history and LTM separately if workflow storage is not available.
Skills imports reject symlinks in either destination root or its parents. Append also rejects symlinks at incoming file and directory paths before changing memory. Overwrite safely removes existing trees, including nested symlinks, and rollback preserves those links without writing to their targets.
Default exports use omega-memory-<UTC timestamp>-<ID>.tar.gz, where the
ID is the first eight hexadecimal characters of a UUIDv4.
openai: For cloud-based embeddings.sentence-transformers: For local, offline embeddings.chromadb: Vector database for storage.python-dotenv: Management of environment variables.tqdm: Progress bars for batch processing.
The publishing workflow builds and validates a source distribution and wheel when a GitHub Release is published, then uploads them to PyPI using Trusted Publishing. No PyPI API token or GitHub secret is needed.
One-time setup:
- Create a GitHub environment named
pypiin the repository settings. Configure required reviewers if releases should require approval before uploading. - In the PyPI
import-kbproject's Publishing settings, add a GitHub trusted publisher with the following values:- Owner: the GitHub user or organization that will publish releases
(
blackhammer116for this fork, orsingnetfor the upstream repository). - Repository:
import-knowledge-package. - Workflow filename:
publish.yml. - Environment:
pypi.
- Owner: the GitHub user or organization that will publish releases
(
To release, update project.version in pyproject.toml to an unused version,
commit the change, and publish a GitHub Release tagged with that version
(for example, v0.2.5 or 0.2.5). The tag must point to the commit containing
the version update and this workflow. Draft releases and tag pushes alone do
not publish; publishing a GitHub prerelease also triggers the workflow, so use
a corresponding Python prerelease version such as 0.3.0rc1.
The workflow rejects tags that do not match project.version. PyPI does not
allow an existing release file to be overwritten; use a new version for each
release.
This project is licensed under the Apache License 2.0. See LICENSE for details.