Semantic Compression Layer

LoreTokens Technical Overview

LoreTokens are an AI-native serialization format designed to compress meaning, not bytes. A single LoreToken line can encode domain, concept, subject, and output shape, which modern LLMs expand into detailed responses using their existing knowledge. High-level overview: loretokens.ai.

1. Repositories & Spec

LoreTokens format kit: github.com/apolloraines/LoreTokens
SAIQL Engine (LoreToken-aware DB): github.com/apolloraines/SAIQL-Engine_v0.2.1

The README in the LoreTokens repo provides the narrative history and long-form design notes: README.md .

2. Formats & Glyphs

LoreTokens come in three main flavors: Symbolic (.sym), Standard (.lt), and Ultra (.ltu). All three share the same semantic core but trade off machine-friendliness vs readability.

Format overview: docs/FORMATS.md
Full format reference: docs/LORETOKEN_FORMATS.md
Glyph cheat-sheet: docs/SYMBOLS.md
Complete symbol legend: docs/COMPLETE_SYMBOL_LEGEND.md

LoreTokens do not try to be a human-first config language. They are for AI systems and operators who need consistent, compact patterns that models can learn once and reuse.

3. Examples & Datasets

Example files show how real schemas and datasets look once they have been LoreToken-ised. These are ideal for both human inspection and LLM fine-tuning.

Example schemas: examples/Schema-LoreTokenised.txt
Large LoreToken dataset: datasets/ALLDATA-LoreTokened.txt
Use-case overview: docs/LoreToken_UseCases.md

Simple conceptual example

PROFILE.SKILLS:[Mohit+AI,Cloud,Trading>>profile_record,ACTIVE]

This single LoreToken line captures:

Domain: PROFILE
Concept: SKILLS
Subject: Mohit
Attributes: AI, Cloud, Trading
Output shape: profile_record
Status: ACTIVE

A modern LLM can expand this into a structured description, JSON object, documentation, or code that operates on this profile – all from one short line.

4. Tooling, GPU Hook, and SAIQL Integration

LoreTokens ship with tools ranging from simple converters to a CUDA-based GPU hook. The goal is to keep compression as close as possible to where compute happens.

Converter & translator scripts: scripts/loretoken_converter.py , scripts/loretoken_translator.py .
GPU hook overview: docs/GPU_HOOK.md .
Semantic compression proofs: docs/GPU_FINAL_SUMMARY.md and LORETOKEN_PROOF_FOR_CLAUDE.md .
SAIQL integration tests: SAIQL test_loretoken_integration.py .

LoreTokens and SAIQL are two halves of the same stack: LoreTokens compress the semantics, and SAIQL provides a queryable, durable database engine to operate on them. For the database story, see saiql.ai/saiql-docs.html.

5. License & Patent Notes

Patent notes: docs/PATENT.md
Open Lore License text: docs/LICENSE.txt

LoreTokens and SAIQL are governed by the Open Lore License (OLL): small teams and early-stage projects can experiment freely up to a revenue cap; larger commercial deployments must coordinate licensing.