Add semantic-index service, deployment assets, and tests
This commit is contained in:
Jason Thistlethwaite
@@ -0,0 +1,336 @@
|
||||
# Semantic Index Deployment Runbook
|
||||
|
||||
This runbook captures the current deployment shape for the Redmine semantic
|
||||
index. It is written for the LAN test server first, with the same steps intended
|
||||
to carry forward to production after paths and secrets are adjusted.
|
||||
The latest LAN validation record is in
|
||||
`docs/semantic_index_predeployment_validation.md`.
|
||||
|
||||
## Deployable Files
|
||||
|
||||
Copy or update these tracked paths together:
|
||||
|
||||
- `semantic_index/`
|
||||
- `tests/semantic_index/`
|
||||
- `deploy/semantic-index/`
|
||||
- `docs/semantic_index_production_notes.md`
|
||||
- `docs/semantic_index_deployment_runbook.md`
|
||||
- `docs/semantic_index_predeployment_validation.md`
|
||||
- `docs/redmine_issue_api_helpdesk_include.md`
|
||||
|
||||
The Helpdesk contact metadata dependency is the Redmine plugin API patch
|
||||
documented in `docs/redmine_issue_api_helpdesk_include.md`. Deploy that plugin
|
||||
patch before expecting Helpdesk contact fields in indexed results.
|
||||
|
||||
Do not copy local-only runtime files:
|
||||
|
||||
- `semantic_index/.env`
|
||||
- `.cache/`
|
||||
- `.venv/`
|
||||
- `__pycache__/`
|
||||
- Qdrant storage snapshots or rollback tarballs unless deliberately restoring
|
||||
|
||||
## Runtime Prerequisites
|
||||
|
||||
Python runtime dependencies:
|
||||
|
||||
```sh
|
||||
pip install openai qdrant-client fastapi uvicorn
|
||||
```
|
||||
|
||||
Qdrant is expected to run on the larger host and be reachable from the semantic
|
||||
index host through `QDRANT_URL`. The current collection default is
|
||||
`redmine_semantic_sample`.
|
||||
|
||||
Qdrant Docker example:
|
||||
|
||||
```sh
|
||||
docker run -p 6333:6333 -p 6334:6334 \
|
||||
-v qdrant_storage:/qdrant/storage \
|
||||
qdrant/qdrant
|
||||
```
|
||||
|
||||
Before destructive maintenance, create a Qdrant snapshot or preserve the Docker
|
||||
volume.
|
||||
|
||||
## Environment
|
||||
|
||||
For a production-style install, use:
|
||||
|
||||
- code: `/opt/semantic-index`
|
||||
- environment file: `/etc/semantic-index.env`
|
||||
- refresh state: `/var/lib/semantic-index/refresh_state.json`
|
||||
- refresh logs: `/var/log/semantic-index`
|
||||
|
||||
Create `/etc/semantic-index.env` from
|
||||
`deploy/semantic-index/semantic-index.env.example` and fill secrets on the
|
||||
target host:
|
||||
|
||||
```sh
|
||||
OPENAI_API_KEY=
|
||||
QDRANT_URL=http://qdrant-host:6333
|
||||
QDRANT_API_KEY=
|
||||
QDRANT_COLLECTION=redmine_semantic_sample
|
||||
REDMINE_URL=http://redmine-host
|
||||
REDMINE_API_KEY=
|
||||
REDMINE_PROJECT_IDENTIFIER=
|
||||
REDMINE_SAMPLE_LIMIT=500
|
||||
SEMANTIC_INDEX_HOST=127.0.0.1
|
||||
SEMANTIC_INDEX_PORT=8787
|
||||
SEMANTIC_INDEX_API_KEY=
|
||||
SEMANTIC_INDEX_REFRESH_STATE_PATH=/var/lib/semantic-index/refresh_state.json
|
||||
```
|
||||
|
||||
Recommended production-style refresh overrides:
|
||||
|
||||
```sh
|
||||
SEMANTIC_INDEX_PROJECT_LIMITS='customer-service=500,hiring=200,todo-jason=200,sales-inbox=100,business-development=100,dock-scheduling=100,prep-standardization=100'
|
||||
SEMANTIC_INDEX_LOG_DIR=/var/log/semantic-index
|
||||
SEMANTIC_INDEX_STATE_PATH=/var/lib/semantic-index/refresh_state.json
|
||||
SEMANTIC_INDEX_OVERLAP_MINUTES=15
|
||||
```
|
||||
|
||||
Keep `SEMANTIC_INDEX_API_KEY` set when binding outside localhost. Do not commit
|
||||
API keys or `.env` files.
|
||||
|
||||
## Systemd Templates
|
||||
|
||||
Templates live in `deploy/semantic-index/`:
|
||||
|
||||
```text
|
||||
install.sh
|
||||
semantic-index.service
|
||||
semantic-index-refresh.service
|
||||
semantic-index-refresh.timer
|
||||
semantic-index.env.example
|
||||
```
|
||||
|
||||
Use the installer first. It defaults to dry-run:
|
||||
|
||||
```sh
|
||||
deploy/semantic-index/install.sh
|
||||
```
|
||||
|
||||
Apply the install:
|
||||
|
||||
```sh
|
||||
deploy/semantic-index/install.sh --apply
|
||||
```
|
||||
|
||||
Optionally start only the HTTP service after installing:
|
||||
|
||||
```sh
|
||||
deploy/semantic-index/install.sh --apply --start
|
||||
```
|
||||
|
||||
The installer creates `/opt/semantic-index`, `/var/lib/semantic-index`, and
|
||||
`/var/log/semantic-index`; copies the deploy unit; creates
|
||||
`/etc/semantic-index.env` only if it does not already exist; installs systemd
|
||||
unit files; and runs local validation. It does not run backfill, does not enable
|
||||
the refresh timer, and never passes `--force-rebuild`.
|
||||
|
||||
Manual install shape, if the installer cannot be used:
|
||||
|
||||
```sh
|
||||
sudo mkdir -p /opt/semantic-index /var/lib/semantic-index /var/log/semantic-index
|
||||
sudo rsync -a \
|
||||
--exclude '.env' \
|
||||
--exclude '__pycache__/' \
|
||||
--exclude '*.pyc' \
|
||||
semantic_index tests docs deploy dist /opt/semantic-index/
|
||||
sudo cp deploy/semantic-index/semantic-index.env.example /etc/semantic-index.env
|
||||
sudo install -m 0644 deploy/semantic-index/semantic-index.service /etc/systemd/system/semantic-index.service
|
||||
sudo install -m 0644 deploy/semantic-index/semantic-index-refresh.service /etc/systemd/system/semantic-index-refresh.service
|
||||
sudo install -m 0644 deploy/semantic-index/semantic-index-refresh.timer /etc/systemd/system/semantic-index-refresh.timer
|
||||
```
|
||||
|
||||
After editing `/etc/semantic-index.env`, validate manually before enabling the
|
||||
timer:
|
||||
|
||||
```sh
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl start semantic-index.service
|
||||
sudo systemctl status semantic-index.service
|
||||
sudo systemctl start semantic-index-refresh.service
|
||||
sudo journalctl -u semantic-index-refresh.service -n 100 --no-pager
|
||||
```
|
||||
|
||||
Enable the timer only after manual dry-run and `--apply` logs look normal:
|
||||
|
||||
```sh
|
||||
sudo systemctl enable --now semantic-index-refresh.timer
|
||||
```
|
||||
|
||||
## Initial Validation
|
||||
|
||||
Run syntax and test checks after copying code:
|
||||
|
||||
```sh
|
||||
.venv/bin/python -m py_compile semantic_index/*.py
|
||||
.venv/bin/python -m unittest discover -s tests/semantic_index
|
||||
bash -n semantic_index/refresh.sh
|
||||
```
|
||||
|
||||
Confirm service startup:
|
||||
|
||||
```sh
|
||||
uvicorn semantic_index.app:app --host 127.0.0.1 --port 8787
|
||||
curl -sS http://127.0.0.1:8787/health
|
||||
```
|
||||
|
||||
If `SEMANTIC_INDEX_API_KEY` is set:
|
||||
|
||||
```sh
|
||||
curl -sS -H "Authorization: Bearer $SEMANTIC_INDEX_API_KEY" \
|
||||
http://127.0.0.1:8787/projects
|
||||
```
|
||||
|
||||
## Initial Backfill
|
||||
|
||||
Preview Redmine mapping before writing to Qdrant:
|
||||
|
||||
```sh
|
||||
.venv/bin/python -m semantic_index inspect preview-redmine \
|
||||
--project customer-service \
|
||||
--limit 5
|
||||
```
|
||||
|
||||
Backfill the current balanced sample:
|
||||
|
||||
```sh
|
||||
.venv/bin/python -m semantic_index --backfill-redmine-projects \
|
||||
--project-limits customer-service=500,hiring=200,todo-jason=200,sales-inbox=100,business-development=100,dock-scheduling=100,prep-standardization=100
|
||||
```
|
||||
|
||||
Audit the result:
|
||||
|
||||
```sh
|
||||
.venv/bin/python -m semantic_index inspect audit --source redmine --limit 5000
|
||||
.venv/bin/python -m semantic_index inspect smoke-search --project customer-service
|
||||
```
|
||||
|
||||
Expected broad shape for the current LAN sample is roughly:
|
||||
|
||||
- Customer Service is the largest project.
|
||||
- Helpdesk tickets have contact metadata.
|
||||
- Internal projects may have no Helpdesk contact metadata.
|
||||
- `attachments=0`.
|
||||
|
||||
## Routine Refresh
|
||||
|
||||
Use the wrapper for production-style refresh. It defaults to dry-run:
|
||||
|
||||
```sh
|
||||
semantic_index/refresh.sh
|
||||
```
|
||||
|
||||
Small smoke check:
|
||||
|
||||
```sh
|
||||
SEMANTIC_INDEX_PROJECT_LIMITS='customer-service=5' semantic_index/refresh.sh
|
||||
```
|
||||
|
||||
Apply refresh manually:
|
||||
|
||||
```sh
|
||||
semantic_index/refresh.sh --apply
|
||||
```
|
||||
|
||||
Installed wrappers can also be called by absolute path, for example
|
||||
`/opt/semantic-index/semantic_index/refresh.sh`. The wrapper uses its own
|
||||
install root as the working directory and reads defaults from
|
||||
`/etc/semantic-index.env` when that file is readable.
|
||||
|
||||
Review the log path printed by the wrapper. For a healthy routine run after
|
||||
state exists, expect:
|
||||
|
||||
- `scanned_issues` greater than or equal to `detail_fetched_issues`
|
||||
- old issues counted under `skipped_issues`
|
||||
- `would_embed_documents` and `embedded_documents` near zero when Redmine has
|
||||
not changed
|
||||
- no scheduled use of `--force-rebuild`
|
||||
|
||||
Only schedule the wrapper after manual dry-run and apply logs look normal.
|
||||
|
||||
Cron shape, when ready:
|
||||
|
||||
```cron
|
||||
*/30 * * * * cd /home/iadnah/redmine && semantic_index/refresh.sh --apply
|
||||
```
|
||||
|
||||
## Search Validation
|
||||
|
||||
HTTP search:
|
||||
|
||||
```sh
|
||||
semantic_index/search.sh "goods return" customer-service 3
|
||||
semantic_index/search.sh "candidate follow up" hiring 5
|
||||
```
|
||||
|
||||
CLI inspection:
|
||||
|
||||
```sh
|
||||
.venv/bin/python -m semantic_index inspect search "goods return" \
|
||||
--project customer-service \
|
||||
--limit 3
|
||||
|
||||
.venv/bin/python -m semantic_index inspect list \
|
||||
--source redmine \
|
||||
--project customer-service \
|
||||
--limit 10
|
||||
```
|
||||
|
||||
MCP stdio:
|
||||
|
||||
```sh
|
||||
.venv/bin/python -m semantic_index --mcp-stdio
|
||||
```
|
||||
|
||||
Available tools:
|
||||
|
||||
- `semantic_search`
|
||||
- `semantic_get_document`
|
||||
- `semantic_list_projects`
|
||||
- `semantic_backfill_redmine_sample`
|
||||
- `semantic_refresh_redmine`
|
||||
|
||||
## Rollback
|
||||
|
||||
Code rollback:
|
||||
|
||||
- Stop `uvicorn` or the service manager unit.
|
||||
- Restore the previous `semantic_index/` code.
|
||||
- Restore the previous Redmine Helpdesk plugin patch if contact metadata broke.
|
||||
- Restart the service.
|
||||
|
||||
Index rollback options:
|
||||
|
||||
- Restore a Qdrant snapshot or preserved Docker volume.
|
||||
- Or rebuild from Redmine with the known-good code using the multi-project
|
||||
backfill command above.
|
||||
|
||||
Refresh rollback:
|
||||
|
||||
- Disable cron/systemd schedule if enabled.
|
||||
- Preserve the failing log file for diagnosis.
|
||||
- If the refresh state is wrong, move the state file aside rather than editing
|
||||
it in place:
|
||||
|
||||
```sh
|
||||
mv .cache/semantic_index/refresh_state.json .cache/semantic_index/refresh_state.json.bad
|
||||
```
|
||||
|
||||
The next refresh will behave like a first refresh for state purposes, while the
|
||||
`source_hash` guard still prevents embedding unchanged documents.
|
||||
|
||||
## Production Readiness Checklist
|
||||
|
||||
- Redmine API key is scoped appropriately and stored outside git.
|
||||
- Qdrant URL and collection are confirmed.
|
||||
- Qdrant snapshot/export path is known.
|
||||
- Helpdesk API patch is deployed and validated.
|
||||
- HTTP service is bound only to trusted localhost/LAN as intended.
|
||||
- `SEMANTIC_INDEX_API_KEY` is set for non-localhost use.
|
||||
- Initial backfill audit and smoke searches pass.
|
||||
- Refresh dry-run and apply logs show expected low embedding counts.
|
||||
- `--force-rebuild` is documented as manual-only.
|
||||
Reference in New Issue
Block a user