Code Indexing

Automatic code indexing transforms your codebase into searchable knowledge, enabling AI assistants to understand and work with your WordPress projects using semantic search.

Overview

Code indexing in WDG:

Automatic: Triggered by git commits and pulls
Incremental: Only indexes changed files
Semantic: Understands code meaning, not just keywords
Project-Scoped: Each project has isolated collections
Fast: Local embeddings require no external API calls

How It Works

Indexing Pipeline

%%{init: {'theme':'neutral'}}%%
sequenceDiagram
    participant Git
    participant Hook as post-commit hook
    participant Service as indexer service (:8666)
    participant Indexer as WDGCodeIndexer
    participant Qdrant

    Git->>Hook: git commit
    Hook->>Service: POST /index/incremental {files, project}
    Service->>Indexer: index_file() per path
    Indexer->>Indexer: chunk_code() + embed in-process (all-MiniLM-L6-v2)
    Indexer->>Qdrant: upsert (batches of 50)
    Service->>Hook: IndexResponse

What Gets Indexed

PHP Files

Functions:

php

// Indexed as complete unit with context
function get_user_posts($user_id, $post_type = 'post') {
    global $wpdb;
    return $wpdb->get_results(
        $wpdb->prepare(
            "SELECT * FROM {$wpdb->posts}
             WHERE post_author = %d
             AND post_type = %s
             AND post_status = 'publish'",
            $user_id,
            $post_type
        )
    );
}

Classes:

php

// Indexed with all methods and properties
class CustomPostType {
    private $post_type;

    public function __construct($type) {
        $this->post_type = $type;
        $this->register();
    }

    public function register() {
        // Method implementation
    }
}

WordPress Hooks:

php

// Indexed with full context
add_action('init', function() {
    register_post_type('custom_type', [
        'public' => true,
        'supports' => ['title', 'editor']
    ]);
});

JavaScript Files

Functions:

javascript

// Regular functions
function validateEmail(email) {
    return /^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(email);
}

// Arrow functions
const fetchUserData = async (userId) => {
    const response = await fetch(`/api/users/${userId}`);
    return response.json();
};

React Components:

jsx

// Indexed with props and hooks
function UserProfile({ userId, showEmail = false }) {
    const [user, setUser] = useState(null);

    useEffect(() => {
        fetchUserData(userId).then(setUser);
    }, [userId]);

    return (
        <div className="user-profile">
            <h2>{user?.name}</h2>
            {showEmail && <p>{user?.email}</p>}
        </div>
    );
}

CSS/SCSS Files

scss

// Chunked by logical sections
.user-profile {
    display: flex;
    padding: 2rem;

    &__header {
        font-size: 1.5rem;
        color: var(--primary-color);
    }

    @media (max-width: 768px) {
        flex-direction: column;
    }
}

Block Configurations

json

{
  "name": "wdg/custom-block",
  "title": "Custom Block",
  "category": "wdg-blocks",
  "attributes": {
    "content": {
      "type": "string",
      "default": ""
    }
  }
}

Indexing Strategies

Automatic Indexing (Recommended)

Git hooks automatically trigger indexing:

bash

# After committing changes
git add .
git commit -m "Add new feature"
# → Automatically indexes changed files

# After pulling updates
git pull origin main
# → Automatically indexes merged changes

Manual Indexing

bash

# Index Wikit framework (default target)
wdg index

# Index a specific project's repositories
wdg index my-site

# Index the platform codebase (CLI, MCP server, indexer, dashboard, docs)
wdg index --platform

# Index Wikit documentation
wdg index docs

# Pull latest Wikit repos, then re-index
wdg index --update

There are no --types, --path, --exclude, --force, or --all flags on wdg index. Re-indexing simply re-runs the relevant target; the indexer upserts points by a deterministic id (md5(file_path:line_number:name)), so re-running overwrites the existing vectors for unchanged components. The set of indexable files and the directories that are skipped are fixed by the indexer engine (see Indexer Internals), not selectable per run.

Vector Collections

Collection Structure

Each project gets its own collection in Qdrant:

Collections:
├── wdg_framework          # Wikit core framework
├── platform               # CLI, MCP server, indexer, dashboard, docs
├── project_my_site        # Project: my-site
├── project_client_site    # Project: client-site
└── project_demo           # Project: demo

Vector Metadata

Each indexed chunk is a Qdrant point with a flat payload (no nested metadata object) and an id of md5(file_path:line_number:name). Core keys:

json

{
  "id": "md5(file_path:line_number:name)",
  "vector": [0.1, 0.2, 0.3, "..."],
  "payload": {
    "project": "my-site",
    "file_path": "wp-content/themes/custom/functions.php",
    "line_number": 45,
    "name": "get_user_posts",
    "component_type": "function",
    "language": "php",
    "content": "function get_user_posts($user_id...) {...}"
  }
}

The indexer also stores file_extension, is_wikit, and repository, plus component-specific optional fields (e.g. calls, extends, implements, ACF/hook metadata). There is no line_start/line_end, docblock, commit_hash, branch, or indexed_at field.

Indexing Performance

Initial Indexing

bash

# Wikit Framework (~5,000 files)
Time: 5-7 minutes
Vectors created: ~15,000
Disk space: ~50MB

# Typical project (~500 files)
Time: 30-60 seconds
Vectors created: ~1,500
Disk space: ~5MB

Incremental Updates

bash

# Single file change
Time: <1 second
Vectors updated: 1-10
Overhead: Minimal

# Pull with 20 changed files
Time: 5-10 seconds
Vectors updated: 50-200
Overhead: Negligible

Performance Optimization

Batch Processing:

python

# Indexer processes files in batches
batch_size = 32
embeddings = model.encode(
    code_chunks,
    batch_size=batch_size,
    show_progress_bar=True
)

Caching:

python

# Only re-index if file changed
if file_hash != cached_hash:
    index_file(file)
else:
    skip_file(file)

Search Capabilities

Semantic Search

Find code by meaning, not just keywords:

bash

# Search query: "validate user email address"
# Finds:
- is_valid_email($email)
- validateEmailAddress(email)
- checkUserEmailFormat()
- /^[^\s@]+@[^\s@]+\.[^\s@]+$/

Cross-Language Search

bash

# Search: "fetch data from API"
# Finds across languages:
PHP:  wp_remote_get($url)
JS:   fetch(url).then(r => r.json())
JS:   axios.get(url)

Pattern Recognition

bash

# Search: "custom post type registration"
# Finds all register_post_type() calls with context:
- Portfolio custom post type
- Testimonials CPT
- Events post type
- Product catalog

Code Chunking Strategy

PHP Chunking

php

// Chunk 1: Function with full body
function calculate_total($items) {
    $total = 0;
    foreach ($items as $item) {
        $total += $item->price;
    }
    return $total;
}

// Chunk 2: Separate function
function apply_discount($total, $discount) {
    return $total * (1 - $discount);
}

JavaScript Chunking

javascript

// Chunk 1: Component definition
function ProductCard({ product }) {
    return (
        <div className="product-card">
            <h3>{product.name}</h3>
            <p>{product.price}</p>
        </div>
    );
}

// Chunk 2: Helper function
const formatPrice = (price) => {
    return `$${price.toFixed(2)}`;
};

CSS Chunking

scss

// Chunk 1: Component styles
.product-card {
    display: flex;
    padding: 1rem;

    h3 {
        font-size: 1.2rem;
    }
}

// Chunk 2: Media queries
@media (max-width: 768px) {
    .product-card {
        flex-direction: column;
    }
}

Managing Collections

List Collections

bash

wdg collections list

Output (counts are illustrative):

Available collections:
  - wdg_framework: 15234 vectors
  - platform: 4120 vectors
  - project_my_site: 1450 vectors
  - project_client_site: 3892 vectors

Delete Collection

bash

# Delete project collection
wdg collections delete project_old_site

# Re-create by re-indexing
wdg index old-site

Git Hook Integration

Two hooks ship in the platform's hooks/ directory — post-commit and post-merge. Both POST the changed file list to the running indexer service rather than shelling out to wdg index. (See Git Hooks for the full implementation.)

Post-Commit Hook (summary)

bash

#!/usr/bin/env bash
# .git/hooks/post-commit  (excerpt)

# Skip silently unless the indexer container is up
docker ps --format '{{.Names}}' | grep -q "wdg-indexer" || exit 0

# Changed files, with host paths rewritten to the container's /workspace
CHANGED_FILES=$(git diff-tree --no-commit-id --name-only -r HEAD \
    | while read -r f; do echo "$REPO_DIR/$f"; done \
    | sed "s|${WDG_ROOT}|/workspace|g")
FILES_JSON=$(echo "$CHANGED_FILES" | jq -R -s -c 'split("\n") | map(select(length > 0))')

# Fire-and-forget POST to the indexer (backgrounded so the commit isn't blocked)
(curl -s --max-time 300 -X POST http://localhost:8666/index/incremental \
    -H "Content-Type: application/json" \
    -d "{\"files\": $FILES_JSON, \"project\": \"$INDEX_PROJECT\"}" \
    > /dev/null 2>&1) &

The post-merge hook is identical except it diffs from ORIG_HEAD to capture everything a pull/merge changed.

Installing Hooks

bash

# Hooks are installed automatically when:
# 1. Creating a project with --init-wikit
wdg create my-site --init-wikit

# 2. Adding a repository to a project
wdg my-site repo add https://github.com/client/repo

# 3. Manually (the two shipped hooks)
hooks/install-hooks.sh /path/to/repo
# or
cp hooks/post-commit hooks/post-merge /path/to/repo/.git/hooks/
chmod +x /path/to/repo/.git/hooks/post-commit /path/to/repo/.git/hooks/post-merge

Indexing Best Practices

1. Commit Frequently

bash

# Each commit triggers incremental indexing
git commit -m "Add user authentication"  # Indexes auth code
git commit -m "Add email validation"     # Indexes validation

2. Use Descriptive Commits

bash

# Good: AI can understand context
git commit -m "Add custom post type for portfolio items"

# Bad: Less context for AI
git commit -m "Update code"

3. Structure Code Well

php

// Good: Clear function separation
function get_user() { }
function validate_user() { }
function save_user() { }

// Bad: Monolithic function (harder to search)
function handle_user() {
    // 200 lines of mixed logic
}

4. Include DocBlocks

php

/**
 * Calculate discounted price for user
 *
 * @param float $price Original price
 * @param int $user_id User ID for discount lookup
 * @return float Discounted price
 */
function calculate_discount($price, $user_id) {
    // Implementation
}

5. Regular Maintenance

bash

# Weekly: Update and re-index framework
wdg update

# Monthly: validate, then clean orphaned collections
wdg collections validate
wdg collections clean --confirm

# Quarterly: re-index each scope
wdg index            # Wikit framework
wdg index --platform # platform code
wdg index my-site    # each project

Troubleshooting

Indexing Not Triggering

bash

# Check if hooks are installed
ls -la .git/hooks/post-commit

# Verify hook is executable
chmod +x .git/hooks/post-commit

# Test hook manually
.git/hooks/post-commit

Slow Indexing

bash

# Check system resources
docker stats wdg-indexer

# Use faster model
# Edit .env: EMBEDDING_MODEL=all-MiniLM-L6-v2

# Restart indexer
docker-compose restart indexer

Missing Results

bash

# Verify collection exists
wdg collections list

# Check vector count
curl http://localhost:6333/collections/project_my_site

# Re-index if needed (overwrites existing vectors)
wdg index my-site

Out of Disk Space

bash

# Check collection sizes
wdg collections list

# Delete old collections
wdg collections delete project_old_*

# Prune Docker volumes
docker system prune -v

Advanced Configuration

Custom Embedding Model

bash

# Edit .env
EMBEDDING_MODEL=all-mpnet-base-v2  # Higher quality, slower
# or
EMBEDDING_MODEL=all-MiniLM-L6-v2  # Faster, default

# Restart indexer service to load the new model
docker compose restart indexer

# Re-index each scope with the new model
wdg index            # Wikit framework
wdg index --platform # platform code
wdg index my-site    # each project

Indexing Filters

Filtering is not configurable per project — there is no .wdg/indexing.json file, and there are no chunk_size/overlap settings. The indexer engine applies a fixed policy:

Indexable extensions are fixed (php, js, jsx, ts, tsx, css, scss, sass, json, md, yml, yaml, py, sh).
Skipped directories are hardcoded: node_modules, vendor, dist, build, .git, __pycache__, the various venv/cache dirs, and worktrees (git and agent worktrees). Third-party /plugins/ and the bundled twenty* themes are also excluded; mu-plugins are kept.
Chunking is fixed: semantic components (functions, classes, hooks, blocks, etc.) where the language supports it, otherwise 50-line windows. There is no configurable chunk size or overlap.

See Indexer Internals for the authoritative extension list and exclusion rules.

Integration with AI Assistants

The indexed code becomes instantly searchable by AI:

bash

# AI can now answer:
"Where do we register custom post types?"
"Show me how we handle user authentication"
"Find similar implementations of email validation"
"What Wikit blocks are used in this project?"

Monitoring Indexing

View Indexing Logs

bash

# Real-time logs
wdg logs indexer --follow

# Last 100 lines
wdg logs indexer --tail 100

Indexing Status

bash

# Overall status
wdg status

# Project-specific status
wdg status my-site

Next Steps:

Learn about Local Embeddings
Configure MCP Server
Review Git Hooks

Code Indexing ​

Overview ​

How It Works ​

Indexing Pipeline ​

What Gets Indexed ​

PHP Files ​

JavaScript Files ​

CSS/SCSS Files ​

Block Configurations ​

Indexing Strategies ​

Automatic Indexing (Recommended) ​

Manual Indexing ​

Vector Collections ​

Collection Structure ​

Vector Metadata ​

Indexing Performance ​

Initial Indexing ​

Incremental Updates ​

Performance Optimization ​

Search Capabilities ​

Semantic Search ​

Cross-Language Search ​

Pattern Recognition ​

Code Chunking Strategy ​

PHP Chunking ​

JavaScript Chunking ​

CSS Chunking ​

Managing Collections ​

List Collections ​

Delete Collection ​

Git Hook Integration ​

Post-Commit Hook (summary) ​

Installing Hooks ​

Indexing Best Practices ​

1. Commit Frequently ​

2. Use Descriptive Commits ​

3. Structure Code Well ​

4. Include DocBlocks ​

5. Regular Maintenance ​

Troubleshooting ​

Indexing Not Triggering ​

Slow Indexing ​

Missing Results ​

Out of Disk Space ​

Advanced Configuration ​

Custom Embedding Model ​

Indexing Filters ​

Integration with AI Assistants ​

Monitoring Indexing ​

View Indexing Logs ​

Indexing Status ​

Code Indexing

Overview

How It Works

Indexing Pipeline

What Gets Indexed

PHP Files

JavaScript Files

CSS/SCSS Files

Block Configurations

Indexing Strategies

Automatic Indexing (Recommended)

Manual Indexing

Vector Collections

Collection Structure

Vector Metadata

Indexing Performance

Initial Indexing

Incremental Updates

Performance Optimization

Search Capabilities

Semantic Search

Cross-Language Search

Pattern Recognition

Code Chunking Strategy

PHP Chunking

JavaScript Chunking

CSS Chunking

Managing Collections

List Collections

Delete Collection

Git Hook Integration

Post-Commit Hook (summary)

Installing Hooks

Indexing Best Practices

1. Commit Frequently

2. Use Descriptive Commits

3. Structure Code Well

4. Include DocBlocks

5. Regular Maintenance

Troubleshooting

Indexing Not Triggering

Slow Indexing

Missing Results

Out of Disk Space

Advanced Configuration

Custom Embedding Model

Indexing Filters

Integration with AI Assistants

Monitoring Indexing

View Indexing Logs

Indexing Status