redmine/docs/semantic_index_deployment_runbook.md

# Semantic Index Deployment Runbook

This runbook captures the current deployment shape for the Redmine semantic
index. It is written for the LAN test server first, with the same steps intended
to carry forward to production after paths and secrets are adjusted.
The latest LAN validation record is in
`docs/semantic_index_predeployment_validation.md`.

## Deployable Files

Copy or update these tracked paths together:

- `semantic_index/`
- `tests/semantic_index/`
- `deploy/semantic-index/`
- `docs/semantic_index_production_notes.md`
- `docs/semantic_index_deployment_runbook.md`
- `docs/semantic_index_predeployment_validation.md`
- `docs/redmine_issue_api_helpdesk_include.md`

The Helpdesk contact metadata dependency is the Redmine plugin API patch
documented in `docs/redmine_issue_api_helpdesk_include.md`. Deploy that plugin
patch before expecting Helpdesk contact fields in indexed results.

Do not copy local-only runtime files:

- `semantic_index/.env`
- `.cache/`
- `.venv/`
- `__pycache__/`
- Qdrant storage snapshots or rollback tarballs unless deliberately restoring

## Runtime Prerequisites

Python runtime dependencies:

```sh
pip install openai qdrant-client fastapi uvicorn
```

Qdrant is expected to run on the larger host and be reachable from the semantic
index host through `QDRANT_URL`. The current collection default is
`redmine_semantic_sample`.

Qdrant Docker example:

```sh
docker run -p 6333:6333 -p 6334:6334 \
  -v qdrant_storage:/qdrant/storage \
  qdrant/qdrant
```

Before destructive maintenance, create a Qdrant snapshot or preserve the Docker
volume.

## WireGuard Topology

Current WireGuard endpoints:

- production server: `10.11.0.100`
- LAN server: `10.11.0.105`

When Qdrant is hosted on the LAN server, keep it reachable on the WireGuard
address and point production semantic-index traffic at:

```sh
QDRANT_URL=http://10.11.0.105:6333
```

Do not bind production-hosted Qdrant to `10.11.0.105`; that address belongs to
the LAN host.

## Environment

For a production-style install, use:

- code: `/opt/semantic-index`
- environment file: `/etc/semantic-index.env`
- refresh state: `/var/lib/semantic-index/refresh_state.json`
- refresh logs: `/var/log/semantic-index`

Create `/etc/semantic-index.env` from
`deploy/semantic-index/semantic-index.env.example` and fill secrets on the
target host:

```sh
OPENAI_API_KEY=
QDRANT_URL=http://10.11.0.105:6333
QDRANT_API_KEY=
QDRANT_COLLECTION=redmine_semantic_sample
REDMINE_URL=http://redmine-host
REDMINE_API_KEY=
REDMINE_PROJECT_IDENTIFIER=
REDMINE_SAMPLE_LIMIT=500
SEMANTIC_INDEX_HOST=127.0.0.1
SEMANTIC_INDEX_PORT=8787
SEMANTIC_INDEX_API_KEY=
SEMANTIC_INDEX_REFRESH_STATE_PATH=/var/lib/semantic-index/refresh_state.json
```

Recommended production-style refresh overrides:

```sh
SEMANTIC_INDEX_PROJECT_LIMITS='customer-service=500,hiring=200,todo-jason=200,sales-inbox=100,business-development=100,dock-scheduling=100,prep-standardization=100'
SEMANTIC_INDEX_LOG_DIR=/var/log/semantic-index
SEMANTIC_INDEX_STATE_PATH=/var/lib/semantic-index/refresh_state.json
SEMANTIC_INDEX_OVERLAP_MINUTES=15
```

Keep `SEMANTIC_INDEX_API_KEY` set when binding outside localhost. Do not commit
API keys or `.env` files.

## Systemd Templates

Templates live in `deploy/semantic-index/`:

```text
install.sh
semantic-index.service
semantic-index-refresh.service
semantic-index-refresh.timer
semantic-index.env.example
```

Use the installer first. It defaults to dry-run:

```sh
deploy/semantic-index/install.sh
```

Apply the install:

```sh
deploy/semantic-index/install.sh --apply
```

Optionally start only the HTTP service after installing:

```sh
deploy/semantic-index/install.sh --apply --start
```

The installer creates `/opt/semantic-index`, `/var/lib/semantic-index`, and
`/var/log/semantic-index`; copies the deploy unit; creates
`/etc/semantic-index.env` only if it does not already exist; installs systemd
unit files; and runs local validation. It does not run backfill, does not enable
the refresh timer, and never passes `--force-rebuild`.

Manual install shape, if the installer cannot be used:

```sh
sudo mkdir -p /opt/semantic-index /var/lib/semantic-index /var/log/semantic-index
sudo rsync -a \
  --exclude '.env' \
  --exclude '__pycache__/' \
  --exclude '*.pyc' \
  semantic_index tests docs deploy dist /opt/semantic-index/
sudo cp deploy/semantic-index/semantic-index.env.example /etc/semantic-index.env
sudo install -m 0644 deploy/semantic-index/semantic-index.service /etc/systemd/system/semantic-index.service
sudo install -m 0644 deploy/semantic-index/semantic-index-refresh.service /etc/systemd/system/semantic-index-refresh.service
sudo install -m 0644 deploy/semantic-index/semantic-index-refresh.timer /etc/systemd/system/semantic-index-refresh.timer
```

After editing `/etc/semantic-index.env`, validate manually before enabling the
timer:

```sh
sudo systemctl daemon-reload
sudo systemctl start semantic-index.service
sudo systemctl status semantic-index.service
sudo systemctl start semantic-index-refresh.service
sudo journalctl -u semantic-index-refresh.service -n 100 --no-pager
```

Enable the timer only after manual dry-run and `--apply` logs look normal:

```sh
sudo systemctl enable --now semantic-index-refresh.timer
```

## Initial Validation

Run syntax and test checks after copying code:

```sh
.venv/bin/python -m py_compile semantic_index/*.py
.venv/bin/python -m unittest discover -s tests/semantic_index
bash -n semantic_index/refresh.sh
```

Confirm service startup:

```sh
uvicorn semantic_index.app:app --host 127.0.0.1 --port 8787
curl -sS http://127.0.0.1:8787/health
```

If `SEMANTIC_INDEX_API_KEY` is set:

```sh
curl -sS -H "Authorization: Bearer $SEMANTIC_INDEX_API_KEY" \
  http://127.0.0.1:8787/projects
```

## Initial Backfill

Preview Redmine mapping before writing to Qdrant:

```sh
.venv/bin/python -m semantic_index inspect preview-redmine \
  --project customer-service \
  --limit 5
```

Backfill the current balanced sample:

```sh
.venv/bin/python -m semantic_index --backfill-redmine-projects \
  --project-limits customer-service=500,hiring=200,todo-jason=200,sales-inbox=100,business-development=100,dock-scheduling=100,prep-standardization=100
```

Audit the result:

```sh
.venv/bin/python -m semantic_index inspect audit --source redmine --limit 5000
.venv/bin/python -m semantic_index inspect smoke-search --project customer-service
```

Expected broad shape for the current LAN sample is roughly:

- Customer Service is the largest project.
- Helpdesk tickets have contact metadata.
- Internal projects may have no Helpdesk contact metadata.
- `attachments=0`.

## Routine Refresh

Use the wrapper for production-style refresh. It defaults to dry-run:

```sh
semantic_index/refresh.sh
```

Small smoke check:

```sh
SEMANTIC_INDEX_PROJECT_LIMITS='customer-service=5' semantic_index/refresh.sh
```

Apply refresh manually:

```sh
semantic_index/refresh.sh --apply
```

Installed wrappers can also be called by absolute path, for example
`/opt/semantic-index/semantic_index/refresh.sh`. The wrapper uses its own
install root as the working directory and reads defaults from
`/etc/semantic-index.env` when that file is readable.

Review the log path printed by the wrapper. For a healthy routine run after
state exists, expect:

- `scanned_issues` greater than or equal to `detail_fetched_issues`
- old issues counted under `skipped_issues`
- `would_embed_documents` and `embedded_documents` near zero when Redmine has
  not changed
- no scheduled use of `--force-rebuild`

Only schedule the wrapper after manual dry-run and apply logs look normal.

Cron shape, when ready:

```cron
*/30 * * * * cd /home/iadnah/redmine && semantic_index/refresh.sh --apply
```

## Search Validation

HTTP search:

```sh
semantic_index/search.sh "goods return" customer-service 3
semantic_index/search.sh "candidate follow up" hiring 5
```

CLI inspection:

```sh
.venv/bin/python -m semantic_index inspect search "goods return" \
  --project customer-service \
  --limit 3

.venv/bin/python -m semantic_index inspect list \
  --source redmine \
  --project customer-service \
  --limit 10
```

MCP stdio:

```sh
.venv/bin/python -m semantic_index --mcp-stdio
```

Available tools:

- `semantic_search`
- `semantic_get_document`
- `semantic_list_projects`
- `semantic_backfill_redmine_sample`
- `semantic_refresh_redmine`

## Rollback

Code rollback:

- Stop `uvicorn` or the service manager unit.
- Restore the previous `semantic_index/` code.
- Restore the previous Redmine Helpdesk plugin patch if contact metadata broke.
- Restart the service.

Index rollback options:

- Restore a Qdrant snapshot or preserved Docker volume.
- Or rebuild from Redmine with the known-good code using the multi-project
  backfill command above.

Refresh rollback:

- Disable cron/systemd schedule if enabled.
- Preserve the failing log file for diagnosis.
- If the refresh state is wrong, move the state file aside rather than editing
  it in place:

```sh
mv .cache/semantic_index/refresh_state.json .cache/semantic_index/refresh_state.json.bad
```

The next refresh will behave like a first refresh for state purposes, while the
`source_hash` guard still prevents embedding unchanged documents.

## Production Readiness Checklist

- Redmine API key is scoped appropriately and stored outside git.
- Qdrant URL and collection are confirmed.
- Qdrant snapshot/export path is known.
- Helpdesk API patch is deployed and validated.
- HTTP service is bound only to trusted localhost/LAN as intended.
- `SEMANTIC_INDEX_API_KEY` is set for non-localhost use.
- Initial backfill audit and smoke searches pass.
- Refresh dry-run and apply logs show expected low embedding counts.
- `--force-rebuild` is documented as manual-only.