redmine/docs/semantic_index_deployment_runbook.md

Semantic Index Deployment Runbook

This runbook captures the current deployment shape for the Redmine semantic index. It is written for the LAN test server first, with the same steps intended to carry forward to production after paths and secrets are adjusted. The latest LAN validation record is in docs/semantic_index_predeployment_validation.md.

Deployable Files

Copy or update these tracked paths together:

• semantic_index/
• tests/semantic_index/
• deploy/semantic-index/
• docs/semantic_index_production_notes.md
• docs/semantic_index_deployment_runbook.md
• docs/semantic_index_predeployment_validation.md
• docs/redmine_issue_api_helpdesk_include.md

The Helpdesk contact metadata dependency is the Redmine plugin API patch documented in docs/redmine_issue_api_helpdesk_include.md. Deploy that plugin patch before expecting Helpdesk contact fields in indexed results.

Do not copy local-only runtime files:

• semantic_index/.env
• .cache/
• .venv/
• __pycache__/
• Qdrant storage snapshots or rollback tarballs unless deliberately restoring

Runtime Prerequisites

Python runtime dependencies:

pip install openai qdrant-client fastapi uvicorn

Qdrant is expected to run on the larger host and be reachable from the semantic index host through QDRANT_URL. The current collection default is redmine_semantic_sample.

Qdrant Docker example:

docker run -p 6333:6333 -p 6334:6334 \
  -v qdrant_storage:/qdrant/storage \
  qdrant/qdrant

Before destructive maintenance, create a Qdrant snapshot or preserve the Docker volume.

WireGuard Topology

Current WireGuard endpoints:

• production server: 10.11.0.100
• LAN server: 10.11.0.105

When Qdrant is hosted on the LAN server, keep it reachable on the WireGuard address and point production semantic-index traffic at:

QDRANT_URL=http://10.11.0.105:6333

Do not bind production-hosted Qdrant to 10.11.0.105; that address belongs to the LAN host.

Environment

For a production-style install, use:

• code: /opt/semantic-index
• environment file: /etc/semantic-index.env
• refresh state: /var/lib/semantic-index/refresh_state.json
• refresh logs: /var/log/semantic-index

Create /etc/semantic-index.env from deploy/semantic-index/semantic-index.env.example and fill secrets on the target host:

OPENAI_API_KEY=
QDRANT_URL=http://10.11.0.105:6333
QDRANT_API_KEY=
QDRANT_COLLECTION=redmine_semantic_sample
REDMINE_URL=http://redmine-host
REDMINE_API_KEY=
REDMINE_PROJECT_IDENTIFIER=
REDMINE_SAMPLE_LIMIT=500
SEMANTIC_INDEX_HOST=127.0.0.1
SEMANTIC_INDEX_PORT=8787
SEMANTIC_INDEX_API_KEY=
SEMANTIC_INDEX_REFRESH_STATE_PATH=/var/lib/semantic-index/refresh_state.json

Recommended production-style refresh overrides:

SEMANTIC_INDEX_PROJECT_LIMITS='customer-service=500,hiring=200,todo-jason=200,sales-inbox=100,business-development=100,dock-scheduling=100,prep-standardization=100'
SEMANTIC_INDEX_LOG_DIR=/var/log/semantic-index
SEMANTIC_INDEX_STATE_PATH=/var/lib/semantic-index/refresh_state.json
SEMANTIC_INDEX_OVERLAP_MINUTES=15

Keep SEMANTIC_INDEX_API_KEY set when binding outside localhost. Do not commit API keys or .env files.

Systemd Templates

Templates live in deploy/semantic-index/:

install.sh
semantic-index.service
semantic-index-refresh.service
semantic-index-refresh.timer
semantic-index.env.example

Use the installer first. It defaults to dry-run:

deploy/semantic-index/install.sh

Apply the install:

deploy/semantic-index/install.sh --apply

Optionally start only the HTTP service after installing:

deploy/semantic-index/install.sh --apply --start

The installer creates /opt/semantic-index, /var/lib/semantic-index, and /var/log/semantic-index; copies the deploy unit; creates /etc/semantic-index.env only if it does not already exist; installs systemd unit files; and runs local validation. It does not run backfill, does not enable the refresh timer, and never passes --force-rebuild.

Manual install shape, if the installer cannot be used:

sudo mkdir -p /opt/semantic-index /var/lib/semantic-index /var/log/semantic-index
sudo rsync -a \
  --exclude '.env' \
  --exclude '__pycache__/' \
  --exclude '*.pyc' \
  semantic_index tests docs deploy dist /opt/semantic-index/
sudo cp deploy/semantic-index/semantic-index.env.example /etc/semantic-index.env
sudo install -m 0644 deploy/semantic-index/semantic-index.service /etc/systemd/system/semantic-index.service
sudo install -m 0644 deploy/semantic-index/semantic-index-refresh.service /etc/systemd/system/semantic-index-refresh.service
sudo install -m 0644 deploy/semantic-index/semantic-index-refresh.timer /etc/systemd/system/semantic-index-refresh.timer

After editing /etc/semantic-index.env, validate manually before enabling the timer:

sudo systemctl daemon-reload
sudo systemctl start semantic-index.service
sudo systemctl status semantic-index.service
sudo systemctl start semantic-index-refresh.service
sudo journalctl -u semantic-index-refresh.service -n 100 --no-pager

Enable the timer only after manual dry-run and --apply logs look normal:

sudo systemctl enable --now semantic-index-refresh.timer

Initial Validation

Run syntax and test checks after copying code:

.venv/bin/python -m py_compile semantic_index/*.py
.venv/bin/python -m unittest discover -s tests/semantic_index
bash -n semantic_index/refresh.sh

Confirm service startup:

uvicorn semantic_index.app:app --host 127.0.0.1 --port 8787
curl -sS http://127.0.0.1:8787/health

If SEMANTIC_INDEX_API_KEY is set:

curl -sS -H "Authorization: Bearer $SEMANTIC_INDEX_API_KEY" \
  http://127.0.0.1:8787/projects

Initial Backfill

Preview Redmine mapping before writing to Qdrant:

.venv/bin/python -m semantic_index inspect preview-redmine \
  --project customer-service \
  --limit 5

Backfill the current balanced sample:

.venv/bin/python -m semantic_index --backfill-redmine-projects \
  --project-limits customer-service=500,hiring=200,todo-jason=200,sales-inbox=100,business-development=100,dock-scheduling=100,prep-standardization=100

Audit the result:

.venv/bin/python -m semantic_index inspect audit --source redmine --limit 5000
.venv/bin/python -m semantic_index inspect smoke-search --project customer-service

Expected broad shape for the current LAN sample is roughly:

• Customer Service is the largest project.
• Helpdesk tickets have contact metadata.
• Internal projects may have no Helpdesk contact metadata.
• attachments=0.

Routine Refresh

Use the wrapper for production-style refresh. It defaults to dry-run:

semantic_index/refresh.sh

Small smoke check:

SEMANTIC_INDEX_PROJECT_LIMITS='customer-service=5' semantic_index/refresh.sh

Apply refresh manually:

semantic_index/refresh.sh --apply

Installed wrappers can also be called by absolute path, for example /opt/semantic-index/semantic_index/refresh.sh. The wrapper uses its own install root as the working directory and reads defaults from /etc/semantic-index.env when that file is readable.

Review the log path printed by the wrapper. For a healthy routine run after state exists, expect:

• scanned_issues greater than or equal to detail_fetched_issues
• old issues counted under skipped_issues
• would_embed_documents and embedded_documents near zero when Redmine has not changed
• no scheduled use of --force-rebuild

Only schedule the wrapper after manual dry-run and apply logs look normal.

Cron shape, when ready:

*/30 * * * * cd /home/iadnah/redmine && semantic_index/refresh.sh --apply

Search Validation

HTTP search:

semantic_index/search.sh "goods return" customer-service 3
semantic_index/search.sh "candidate follow up" hiring 5

CLI inspection:

.venv/bin/python -m semantic_index inspect search "goods return" \
  --project customer-service \
  --limit 3

.venv/bin/python -m semantic_index inspect list \
  --source redmine \
  --project customer-service \
  --limit 10

MCP stdio:

.venv/bin/python -m semantic_index --mcp-stdio

Available tools:

• semantic_search
• semantic_get_document
• semantic_list_projects
• semantic_backfill_redmine_sample
• semantic_refresh_redmine

Rollback

Code rollback:

• Stop uvicorn or the service manager unit.
• Restore the previous semantic_index/ code.
• Restore the previous Redmine Helpdesk plugin patch if contact metadata broke.
• Restart the service.

Index rollback options:

• Restore a Qdrant snapshot or preserved Docker volume.
• Or rebuild from Redmine with the known-good code using the multi-project backfill command above.

Refresh rollback:

• Disable cron/systemd schedule if enabled.
• Preserve the failing log file for diagnosis.
• If the refresh state is wrong, move the state file aside rather than editing it in place:

mv .cache/semantic_index/refresh_state.json .cache/semantic_index/refresh_state.json.bad

The next refresh will behave like a first refresh for state purposes, while the source_hash guard still prevents embedding unchanged documents.

Production Readiness Checklist

• Redmine API key is scoped appropriately and stored outside git.
• Qdrant URL and collection are confirmed.
• Qdrant snapshot/export path is known.
• Helpdesk API patch is deployed and validated.
• HTTP service is bound only to trusted localhost/LAN as intended.
• SEMANTIC_INDEX_API_KEY is set for non-localhost use.
• Initial backfill audit and smoke searches pass.
• Refresh dry-run and apply logs show expected low embedding counts.
• --force-rebuild is documented as manual-only.