Vector Index (vector)

Category: data | Form: | Symbol: Ve

Store and search vector embeddings

Vector storage for embedding-based similarity search. Supports insert, search (k-nearest neighbors), embed, and list operations. Backed by pgvector (PostgreSQL extension). Attach to action elements to inject the connection.

Guide

Store and search vector embeddings

What It Does

Vector Index is a data element that stores embedding vectors and searches them by similarity. It is backed by pgvector (a PostgreSQL extension), and each vector element is a self-contained embedding namespace scoped by element_id within the shared public.element_vectors table.

Beyond raw k-nearest-neighbor search, the element can generate embeddings on demand from text via the LLM gateway (so callers do not have to compute vectors client-side), perform hybrid vector + keyword search, and bulk embed-and-insert large document collections. Searches can be narrowed with metadata filters, and ingestion can be made idempotent via upsert.

Vector elements are attached to action elements to inject the connection — an attached python (or other action) element receives the resource so it can read and write vectors directly. Vector is a leaf atom: it is not wirable and holds data (has_data: true).

Element Definition

Property	Value
Type	vector
Category	data
Form	atom
Symbol	Ve
Icon	hub / #10B981
Storage backend	pgvector

There are no user-editable spec fields. properties.yaml defines VectorProperties with an empty fields map — properties (such as the embedding model) are resolved at runtime from the platform embedding configuration, not configured per element.

States

State	Meaning
provisioned	Initial state after creation
active	Index is in service
error	Index is in an error condition

Capabilities

Capability	Description
similarity-search	k-nearest-neighbor search via pgvector with cosine, L2, or inner-product distance
embedding-generation	On-demand text embedding via the LLM gateway (embed op)
hybrid-search	Combined vector similarity + BM25 keyword search when Tantivy is available; falls back to vector-only scoring when the BM25 leg is unavailable or fails
batch-embed	Bulk embed-and-insert for large document collections
metadata-filter	Pre-filter search candidates by metadata key-value predicates
upsert	Insert-or-update vectors by content hash for idempotent ingestion

Attachable Modifiers

Modifier	Why
rate-limit	Throttle per-op invocation rate (search is CPU/GPU intensive)
auth-policy	Element-scoped auth requirements

Error Codes

Code	Class	Retryable	Description
VECTOR_DIMENSION_MISMATCH	validation	No	Query or insert vector dimensions do not match the index’s stored dimensions
VECTOR_GATEWAY_UNAVAILABLE	internal	Yes	LLM gateway not configured — cannot auto-embed text (embed / batch_embed ops)
VECTOR_EMPTY_QUERY	validation	No	search op received an empty query vector (length 0)
VECTOR_MISSING_VECTOR_FIELD	validation	No	insert / upsert payload entry is missing the required vector field
VECTOR_MISSING_TEXT_FIELD	validation	No	embed op requires a non-empty text string
VECTOR_SEARCH_FAILED	internal	Yes	pgvector similarity search failed — transient backend error
VECTOR_EMBEDDING_FAILED	internal	Yes	LLM gateway returned an error or empty embedding response

Operations

All operations are POST to /api/{circle}/{slug}/ops/{path}.

insert — POST .../ops/insert

Inserts one or more vectors with optional metadata. Each entry requires a vector (array of floats); id, metadata, and content are optional. An ID is auto-generated if not provided, and duplicate IDs are upserted. Vector dimensions must match the index configuration. Auth level: write. Returns inserted (count) and ids.

search — POST .../ops/search

Finds the k most similar vectors to the query vector using cosine (default), inner_product, or l2 distance. Optionally filter by metadata and set include_vectors. Auth level: read. Returns results (each with id, score, metadata, optional content) and count.

embed — POST .../ops/embed

Generates an embedding vector from a text string using the configured embedding model, without storing it. Uses the LLM gateway. Auth level: write. Returns vector, model, and dimensions.

hybrid_search — POST .../ops/hybrid_search

Combines vector similarity (cosine) with BM25 keyword matching, ranking results by a weighted combination. Requires a text query; vector is auto-generated from query if omitted, and vector_weight (default 0.7) balances the two signals. Auth level: read. Returns results.

upsert — POST .../ops/upsert

Like insert but explicitly overwrites existing vectors with matching IDs. If no ID is provided, behaves like insert. Requires vectors. Auth level: write. Returns upserted (count).

batch_embed — POST .../ops/batch_embed

Like embed but optimized for large batches — processes texts in chunks (batch_size, default 32) to avoid LLM gateway timeouts, embedding and inserting each. Auth level: write. Returns inserted and batches.

reindex — POST .../ops/reindex

Drops and recreates the vector index. Useful after bulk inserts or when switching distance metrics; may take time for large collections. Auth level: admin. Returns reindexed (boolean) and vector_count.

Quick Start

Creating via API

POST /api/{circle}/{project}/
Content-Type: application/json

{
  "element_type": "vector",
  "slug": "my-vectors",
  "name": "My Vector Index"
}

Inserting vectors

POST /api/{circle}/my-vectors/ops/insert
Content-Type: application/json

{
  "vectors": [
    {
      "id": "doc-1",
      "vector": [0.12, 0.04, 0.91, 0.33],
      "content": "The quick brown fox.",
      "metadata": { "source": "manual", "lang": "en" }
    }
  ]
}

Searching by similarity

POST /api/{circle}/my-vectors/ops/search
Content-Type: application/json

{
  "vector": [0.10, 0.05, 0.88, 0.30],
  "k": 5,
  "metric": "cosine",
  "filter": { "lang": "en" }
}

Embedding text without computing vectors client-side

POST /api/{circle}/my-vectors/ops/embed
Content-Type: application/json

{ "text": "Sentence to embed." }

Hybrid search from a text query

POST /api/{circle}/my-vectors/ops/hybrid_search
Content-Type: application/json

{ "query": "brown fox", "k": 10, "vector_weight": 0.7 }

Common Mistakes

Mixing incompatible vector dimensions. All vectors in an index must share the dimensionality the index was first populated with. Inserting or querying with a differently sized vector raises VECTOR_DIMENSION_MISMATCH. Switching the embedding model after data has been inserted changes the dimension and causes the same error — pick the model before ingestion.

Searching with an empty query vector. search rejects a zero-length query vector with VECTOR_EMPTY_QUERY. Supply a real query embedding, or use hybrid_search with a query string to let the element auto-embed.

Omitting the vector field on insert/upsert. Each entry in vectors must include a vector array; a missing field raises VECTOR_MISSING_VECTOR_FIELD. (id, metadata, and content are optional.)

Calling embed / batch_embed or auto-embed hybrid search without a configured gateway. Text embedding goes through the LLM gateway. If it is not configured, those ops return the retryable VECTOR_GATEWAY_UNAVAILABLE; an embedding error or empty response surfaces as VECTOR_EMBEDDING_FAILED.

Relationships

• Attaches to: rate-limit, auth-policy

Capabilities

• similarity-search: k-nearest-neighbor search via pgvector with cosine, L2, or inner-product distance
• embedding-generation: On-demand text embedding via the LLM gateway (embed op)
• hybrid-search: Combined vector similarity + BM25 keyword search when Tantivy is available; falls back to vector-only scoring when the BM25 leg is unavailable or fails
• batch-embed: Bulk embed-and-insert for large document collections
• metadata-filter: Pre-filter search candidates by metadata key-value predicates
• upsert: Insert-or-update vectors by content hash for idempotent ingestion

Operations

activity

Get /ops/activity | Auth: Read

Get activity events for this element

Scope depends on element capabilities: individual elements query by element_id, project-form elements with activity-scope-members include member activities, circle-level elements with activity-scope-all query the entire circle. Gracefully returns empty list if activities table is missing (old circles).

attachments

Get /ops/attachments | Auth: Read

List all modifiers and resources attached to this element

Returns both modifiers (policy enforcement) and resources (data injection) with is_modifier flag to distinguish. Items in the generated MODIFIER_TYPES list are modifiers; everything else is a resource. Includes cascade_policy and version pin info.

batch_embed

Post /ops/batch_embed | Auth: Write

Batch embed and insert text documents

Like embed but optimized for large batches. Processes texts in chunks to avoid LLM gateway timeouts.

batch_stats

Get /ops/batch_stats | Auth: Read

Get per-element statistics for all children of this element

Returns per-child stats plus an aggregate. Most meaningful on compound or manifest form elements (repositories, circles, projects); atoms have no children so the result is an empty children array with a zeroed aggregate. Uses efficient GROUP BY SQL. Weighted averages for eval scores.

compose

Post /ops/compose | Auth: Execute

Batch add and remove modifiers on this element in a single call

Declarative composition: add modifiers by ref path (slug or path@version) and remove by attachment ID, all in one atomic call on the target element. Each ‘add’ entry resolves the source element, validates topology, attaches with optional priority and cascade policy. Each ‘remove’ entry deletes the attachment row. Returns a summary of what was added and removed. Example: compose({ add: [{ref: “my-prompt”}, {ref: “rate-limit/api@v2”, priority: 50}], remove: [{attachment_id: “uuid”}] })

context

Get /ops/context | Auth: Read

Get connected elements (graph traversal)

Graph traversal showing all connected elements with their relationship type (contains, contained_by, references, referenced_by, attaches, etc.). Use ?depth=N to control traversal depth (default 1) and ?types=actor,data to filter by element types.

create

Post /ops/create | Auth: Write

Create child element

POST to the parent path — element_type goes in the request body, NOT the URL. Both element_type and slug are required and must be non-empty. Name is derived from slug if omitted. Writes to both Git and PostgreSQL. All elements are stored flat under the circle — no intermediate library wrapper rows.

delete

Delete /ops/delete | Auth: Admin

Delete element (soft delete)

Soft delete — sets state to ‘deleted’ but retains the record. Cannot delete elements that have children (has_no_bond precondition) or active runs. Requires admin auth and confirmation.

disable

Post /ops/disable | Auth: Admin

Disable element (hides and prevents use)

Idempotent — safe to call on already-disabled elements. Optionally pass a reason string. Disabled elements cannot be invoked or executed. Inverse of enable.

embed

Post /ops/embed | Auth: Write

Generate embedding vector from text without storing

Generates vector embeddings from text using the configured embedding model. Returns the embedding vector without storing it. Uses the LLM gateway for embedding generation.

enable

Post /ops/enable | Auth: Admin

Enable element (makes usable and visible)

Idempotent — safe to call on already-enabled elements. Transitions element to ready/enabled state. Cannot enable deleted elements. Inverse of disable.

export_bundle

Get /ops/export/bundle | Auth: Read

Export element as downloadable git bundle

On non-root-namespace elements, returns a binary git bundle. On root-namespace (circle) elements, dispatch hands off to the circle’s own export_bundle op, which returns a multi-element JSON envelope with one base64 bundle per child element — this is intentional, not an error.

get

Get /ops/get | Auth: Read

Get element details

Element is already resolved by the routing layer — this returns the cached element, not a fresh DB query. Use the path /api/{circle}/{slug} to address elements.

hybrid_search

Post /ops/hybrid_search | Auth: Read

Combined vector + keyword search

Performs hybrid search combining vector similarity (cosine) with BM25 keyword matching. Returns results ranked by a weighted combination of both scores.

import_bundle

Post /ops/import/bundle | Auth: Write

Import git bundle into element

Accepts a base64-encoded git bundle in the JSON bundle_base64 field. Use overwrite=true to replace existing elements with same slug (default skips duplicates). Imported elements get new UUIDs. Returns counts of imported/skipped elements and any errors.

insert

Post /ops/insert | Auth: Write

Insert vectors with metadata

Inserts one or more vectors into the index. Each entry requires a vector (array of floats) and optional metadata (JSON object for filtering). An ID is auto-generated if not provided. Duplicate IDs are upserted. Vector dimensions must match the index configuration.

intention

Get /ops/intention | Auth: Read

Get element intention with full inheritance chain

Returns three levels: direct (this element’s intention), inherited (from category and root), and resolved (final merged intention). Useful for understanding an element’s purpose in context of its hierarchy.

promote

Post /ops/promote | Auth: Admin

Promote element configuration to a target environment

Only for manifest-form elements (projects). Environments advance: dev → demo → live. dev→demo requires member+ role, demo→live requires admin. Freezes member versions at promotion time (creates snapshot). Persists environment config to spec.environments.

readme

Get /ops/readme | Auth: Read

Get element README.md content

Reads README.md from the element’s git repository. Returns empty content (not an error) if no README exists. Always returns markdown format.

readme_update

Post /ops/readme_update | Auth: Write

Update element README.md content

Creates or overwrites README.md in the element’s git repo. Commits to the draft branch. Content must be provided as a markdown string.

reindex

Post /ops/reindex | Auth: Admin

Rebuild the vector index

Drops and recreates the vector index. Useful after bulk inserts or when switching distance metrics. May take time for large collections.

remove-modifier

Post /ops/remove-modifier | Auth: Execute

Remove an attached modifier from this element by attachment ID

Removes a modifier/resource attachment by its row ID. The ID comes from the attachments or context API. This is the reverse of attach — called on the target element, not the source.

restore

Post /ops/restore | Auth: Admin

Restore element to a specific version

Automatically snapshots the current state before restoring (creates a ‘Before restore to vN’ version entry). Writes restored spec to git as .triform/spec.yaml. Git failures warn but don’t fail the operation — DB state is authoritative. Cannot restore deleted elements.

schema

Get /ops/schema | Auth: Read

Get element input/output schema (MCP tools/list compatible)

Returns type-level port schemas from the TypeRegistry — not instance-specific overrides. Includes direction (input/output), required flag, and JSON schema per port. Useful for understanding what data an element accepts and produces.

search

Post /ops/search | Auth: Read

Search for similar vectors (k-nearest neighbors)

Finds the k most similar vectors to the query vector using cosine similarity (default), inner product, or L2 distance. Filter by metadata to narrow results. Returns vectors with similarity scores, metadata, and optional content.

source

Get /ops/source | Auth: Read

Get any file’s content from the element’s git repository

Reads an arbitrary file from the element’s CAS-backed git tree by its relative path. Same store as readme, just generalized. Path safety: rejects .. traversal, leading /, and null bytes. Use this to view main.py for action elements, asset files for SPAs, etc. Returns empty content (not an error) if the file doesn’t exist.

source_branches

Get /ops/source/branches | Auth: Read

List Source branches for this element

Returns the standard draft/demo/live Source branches, their current commits, and promotion relationships. Use GET /api/{element_path}/ops/source/branches.

source_fixtures

Post /ops/source/fixtures | Auth: Write

Dry-run or apply approved Source seed fixtures

Scans .triform/fixtures/ manifests from the addressed data element Source repo. Defaults to dry_run=true and never imports live runtime data. Apply requires dry_run=false plus confirm=true and dispatches approved records through existing generated element ops.

source_promote

Post /ops/source/promote | Auth: Write

Promote Source branch forward

Promotes draft to demo or demo to live through the generated element op path. Direct Git pushes to demo/live are blocked by Source policy.

source_repair

Post /ops/source/repair | Auth: Write

Inspect or repair the element Source index

Runs Source repair through the element operation path. Defaults to dry_run=true; set dry_run=false only after reviewing a dry-run report.

source_status

Get /ops/source/status | Auth: Read

Get Source control status for this element

Returns the branch-aware clone URL, checkout commands, current draft commit, child source-link count, portable export summary, Source health, warnings, and auth hints for the addressed element. Use the element-first path: GET /api/{element_path}/ops/source/status.

source_validate

Post /ops/source/validate | Auth: Read

Validate Source branch contents

Validates a Source branch before accepting local Git workflow changes or promotion. Defaults to branch=draft and rejects runtime data, generated output, secret material, and unreadable CAS refs.

stats

Get /ops/stats | Auth: Read

Get aggregate statistics for this element

Health status is computed: error if errors_per_day > 5 or success_rate < 0.8, warning if errors_per_day > 0 or success_rate < 0.95. Firing alerts escalate health to error/warning. Default period is ‘day’. Returns runs_per_day, success_rate, avg_duration_ms, and more.

tree

Get /ops/tree | Auth: Read

Get the element’s position in the graph — ancestors, children, references, and subtree statistics

Uses per-circle ElementGraph cache for O(1) lookups. Returns ancestors (containment chain), children (direct), members (references), referenced_by (reverse refs), attachments, and subtree stats. Default depth is 3, max is 10. Pass ?include_metadata=true for name/state on each node.

update

Patch /ops/update | Auth: Write

Update element

Partial update — send only the fields you want to change. spec, name, and intention are all independently optional. spec MUST be a JSON object when present; deep-merged into the existing spec by default. Empty {"spec":{}} preserves existing spec content but still records a new version (no-op for content, not for version state). To clear/replace the entire spec wholesale send {"spec":{...},"deep":false}. List-typed spec fields use replace semantics (the patch list replaces the existing list, no array merging). Coordinates Git + DB writes. Slug cannot be changed after creation.

update_meta

Patch /ops/update_meta | Auth: Write

Update element metadata (lightweight merge — does NOT bump version or snapshot spec)

Shallow JSONB merge into element.meta. Top-level keys in the provided value replace existing meta values; other keys are preserved. Used for UI metadata like canvas positions, panel state, viewer preferences. Wire-shape op_name is update_meta (distinct from update) so SSE subscribers + the cache auto-invalidator can distinguish lightweight metadata changes from spec edits without inspecting the payload. The MutatingElementStore wrapper stamps this op_name on the lifecycle event emitted by update_element_meta storage calls.

upsert

Post /ops/upsert | Auth: Write

Insert or update vectors by ID

Like insert but explicitly overwrites existing vectors with matching IDs. If no ID is provided, behaves like insert.

version

Get /ops/version | Auth: Read

Get current version or full history

Returns current version by default. Pass ?history=true for full version history (up to ?limit=N, default 50). Versions are backed by the element_versions table. Every spec update creates a new version entry.

Error Codes

Code	Class	Retryable	Description
VECTOR_DIMENSION_MISMATCH	validation	no	Query or insert vector dimensions do not match the index’s stored dimensions
VECTOR_GATEWAY_UNAVAILABLE	internal	yes	LLM gateway not configured — cannot auto-embed text (embed / batch_embed ops)
VECTOR_EMPTY_QUERY	validation	no	search op received an empty query vector (length 0)
VECTOR_MISSING_VECTOR_FIELD	validation	no	insert / upsert payload entry is missing the required ‘vector’ field
VECTOR_MISSING_TEXT_FIELD	validation	no	embed op requires a non-empty ‘text’ string
VECTOR_SEARCH_FAILED	internal	yes	pgvector similarity search failed — transient backend error
VECTOR_EMBEDDING_FAILED	internal	yes	LLM gateway returned an error or empty embedding response