ldr/redmine
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add semantic-index service, deployment assets, and tests

Browse Source
This commit is contained in:
Jason Thistlethwaite
2026-05-04 09:50:03 -04:00
parent faad70872b
commit b305544f63
42 changed files with 5059 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/semantic_index_deployment_runbook.md
+336
View File
@@ -0,0 +1,336 @@
# Semantic Index Deployment Runbook
This runbook captures the current deployment shape for the Redmine semantic
index. It is written for the LAN test server first, with the same steps intended
to carry forward to production after paths and secrets are adjusted.
The latest LAN validation record is in
`docs/semantic_index_predeployment_validation.md`.
## Deployable Files
Copy or update these tracked paths together:
- `semantic_index/`
- `tests/semantic_index/`
- `deploy/semantic-index/`
- `docs/semantic_index_production_notes.md`
- `docs/semantic_index_deployment_runbook.md`
- `docs/semantic_index_predeployment_validation.md`
- `docs/redmine_issue_api_helpdesk_include.md`
The Helpdesk contact metadata dependency is the Redmine plugin API patch
documented in `docs/redmine_issue_api_helpdesk_include.md`. Deploy that plugin
patch before expecting Helpdesk contact fields in indexed results.
Do not copy local-only runtime files:
- `semantic_index/.env`
- `.cache/`
- `.venv/`
- `__pycache__/`
- Qdrant storage snapshots or rollback tarballs unless deliberately restoring
## Runtime Prerequisites
Python runtime dependencies:
```sh
pip install openai qdrant-client fastapi uvicorn
```
Qdrant is expected to run on the larger host and be reachable from the semantic
index host through `QDRANT_URL`. The current collection default is
`redmine_semantic_sample`.
Qdrant Docker example:
```sh
docker run -p 6333:6333 -p 6334:6334 \
-v qdrant_storage:/qdrant/storage \
qdrant/qdrant
```
Before destructive maintenance, create a Qdrant snapshot or preserve the Docker
volume.
## Environment
For a production-style install, use:
- code: `/opt/semantic-index`
- environment file: `/etc/semantic-index.env`
- refresh state: `/var/lib/semantic-index/refresh_state.json`
- refresh logs: `/var/log/semantic-index`
Create `/etc/semantic-index.env` from
`deploy/semantic-index/semantic-index.env.example` and fill secrets on the
target host:
```sh
OPENAI_API_KEY=
QDRANT_URL=http://qdrant-host:6333
QDRANT_API_KEY=
QDRANT_COLLECTION=redmine_semantic_sample
REDMINE_URL=http://redmine-host
REDMINE_API_KEY=
REDMINE_PROJECT_IDENTIFIER=
REDMINE_SAMPLE_LIMIT=500
SEMANTIC_INDEX_HOST=127.0.0.1
SEMANTIC_INDEX_PORT=8787
SEMANTIC_INDEX_API_KEY=
SEMANTIC_INDEX_REFRESH_STATE_PATH=/var/lib/semantic-index/refresh_state.json
```
Recommended production-style refresh overrides:
```sh
SEMANTIC_INDEX_PROJECT_LIMITS='customer-service=500,hiring=200,todo-jason=200,sales-inbox=100,business-development=100,dock-scheduling=100,prep-standardization=100'
SEMANTIC_INDEX_LOG_DIR=/var/log/semantic-index
SEMANTIC_INDEX_STATE_PATH=/var/lib/semantic-index/refresh_state.json
SEMANTIC_INDEX_OVERLAP_MINUTES=15
```
Keep `SEMANTIC_INDEX_API_KEY` set when binding outside localhost. Do not commit
API keys or `.env` files.
## Systemd Templates
Templates live in `deploy/semantic-index/`:
```text
install.sh
semantic-index.service
semantic-index-refresh.service
semantic-index-refresh.timer
semantic-index.env.example
```
Use the installer first. It defaults to dry-run:
```sh
deploy/semantic-index/install.sh
```
Apply the install:
```sh
deploy/semantic-index/install.sh --apply
```
Optionally start only the HTTP service after installing:
```sh
deploy/semantic-index/install.sh --apply --start
```
The installer creates `/opt/semantic-index`, `/var/lib/semantic-index`, and
`/var/log/semantic-index`; copies the deploy unit; creates
`/etc/semantic-index.env` only if it does not already exist; installs systemd
unit files; and runs local validation. It does not run backfill, does not enable
the refresh timer, and never passes `--force-rebuild`.
Manual install shape, if the installer cannot be used:
```sh
sudo mkdir -p /opt/semantic-index /var/lib/semantic-index /var/log/semantic-index
sudo rsync -a \
--exclude '.env' \
--exclude '__pycache__/' \
--exclude '*.pyc' \
semantic_index tests docs deploy dist /opt/semantic-index/
sudo cp deploy/semantic-index/semantic-index.env.example /etc/semantic-index.env
sudo install -m 0644 deploy/semantic-index/semantic-index.service /etc/systemd/system/semantic-index.service
sudo install -m 0644 deploy/semantic-index/semantic-index-refresh.service /etc/systemd/system/semantic-index-refresh.service
sudo install -m 0644 deploy/semantic-index/semantic-index-refresh.timer /etc/systemd/system/semantic-index-refresh.timer
```
After editing `/etc/semantic-index.env`, validate manually before enabling the
timer:
```sh
sudo systemctl daemon-reload
sudo systemctl start semantic-index.service
sudo systemctl status semantic-index.service
sudo systemctl start semantic-index-refresh.service
sudo journalctl -u semantic-index-refresh.service -n 100 --no-pager
```
Enable the timer only after manual dry-run and `--apply` logs look normal:
```sh
sudo systemctl enable --now semantic-index-refresh.timer
```
## Initial Validation
Run syntax and test checks after copying code:
```sh
.venv/bin/python -m py_compile semantic_index/*.py
.venv/bin/python -m unittest discover -s tests/semantic_index
bash -n semantic_index/refresh.sh
```
Confirm service startup:
```sh
uvicorn semantic_index.app:app --host 127.0.0.1 --port 8787
curl -sS http://127.0.0.1:8787/health
```
If `SEMANTIC_INDEX_API_KEY` is set:
```sh
curl -sS -H "Authorization: Bearer $SEMANTIC_INDEX_API_KEY" \
http://127.0.0.1:8787/projects
```
## Initial Backfill
Preview Redmine mapping before writing to Qdrant:
```sh
.venv/bin/python -m semantic_index inspect preview-redmine \
--project customer-service \
--limit 5
```
Backfill the current balanced sample:
```sh
.venv/bin/python -m semantic_index --backfill-redmine-projects \
--project-limits customer-service=500,hiring=200,todo-jason=200,sales-inbox=100,business-development=100,dock-scheduling=100,prep-standardization=100
```
Audit the result:
```sh
.venv/bin/python -m semantic_index inspect audit --source redmine --limit 5000
.venv/bin/python -m semantic_index inspect smoke-search --project customer-service
```
Expected broad shape for the current LAN sample is roughly:
- Customer Service is the largest project.
- Helpdesk tickets have contact metadata.
- Internal projects may have no Helpdesk contact metadata.
- `attachments=0`.
## Routine Refresh
Use the wrapper for production-style refresh. It defaults to dry-run:
```sh
semantic_index/refresh.sh
```
Small smoke check:
```sh
SEMANTIC_INDEX_PROJECT_LIMITS='customer-service=5' semantic_index/refresh.sh
```
Apply refresh manually:
```sh
semantic_index/refresh.sh --apply
```
Installed wrappers can also be called by absolute path, for example
`/opt/semantic-index/semantic_index/refresh.sh`. The wrapper uses its own
install root as the working directory and reads defaults from
`/etc/semantic-index.env` when that file is readable.
Review the log path printed by the wrapper. For a healthy routine run after
state exists, expect:
- `scanned_issues` greater than or equal to `detail_fetched_issues`
- old issues counted under `skipped_issues`
- `would_embed_documents` and `embedded_documents` near zero when Redmine has
not changed
- no scheduled use of `--force-rebuild`
Only schedule the wrapper after manual dry-run and apply logs look normal.
Cron shape, when ready:
```cron
*/30 * * * * cd /home/iadnah/redmine && semantic_index/refresh.sh --apply
```
## Search Validation
HTTP search:
```sh
semantic_index/search.sh "goods return" customer-service 3
semantic_index/search.sh "candidate follow up" hiring 5
```
CLI inspection:
```sh
.venv/bin/python -m semantic_index inspect search "goods return" \
--project customer-service \
--limit 3
.venv/bin/python -m semantic_index inspect list \
--source redmine \
--project customer-service \
--limit 10
```
MCP stdio:
```sh
.venv/bin/python -m semantic_index --mcp-stdio
```
Available tools:
- `semantic_search`
- `semantic_get_document`
- `semantic_list_projects`
- `semantic_backfill_redmine_sample`
- `semantic_refresh_redmine`
## Rollback
Code rollback:
- Stop `uvicorn` or the service manager unit.
- Restore the previous `semantic_index/` code.
- Restore the previous Redmine Helpdesk plugin patch if contact metadata broke.
- Restart the service.
Index rollback options:
- Restore a Qdrant snapshot or preserved Docker volume.
- Or rebuild from Redmine with the known-good code using the multi-project
backfill command above.
Refresh rollback:
- Disable cron/systemd schedule if enabled.
- Preserve the failing log file for diagnosis.
- If the refresh state is wrong, move the state file aside rather than editing
it in place:
```sh
mv .cache/semantic_index/refresh_state.json .cache/semantic_index/refresh_state.json.bad
```
The next refresh will behave like a first refresh for state purposes, while the
`source_hash` guard still prevents embedding unchanged documents.
## Production Readiness Checklist
- Redmine API key is scoped appropriately and stored outside git.
- Qdrant URL and collection are confirmed.
- Qdrant snapshot/export path is known.
- Helpdesk API patch is deployed and validated.
- HTTP service is bound only to trusted localhost/LAN as intended.
- `SEMANTIC_INDEX_API_KEY` is set for non-localhost use.
- Initial backfill audit and smoke searches pass.
- Refresh dry-run and apply logs show expected low embedding counts.
- `--force-rebuild` is documented as manual-only.
docs/semantic_index_predeployment_validation.md
+182
View File
@@ -0,0 +1,182 @@
# Semantic Index Pre-Deployment Validation
Validation date: `2026-04-25`
This records the current LAN pre-deployment checks for the semantic index. It
does not include secrets.
## Deploy Unit
Semantic-index deployable files are documented in:
- `dist/semantic-index-v1-predeployment-20260425T150000Z.MANIFEST.md`
- `docs/semantic_index_deployment_runbook.md`
Current known unrelated worktree changes are outside the semantic-index deploy
unit and should not be mixed into the semantic-index release package:
- `redMCP/README.md`
- `redMCP/app/McpDispatcher.php`
- `redMCP/app/RedmineClient.php`
- `redMCP/composer.json`
- `redMCP/bin/test-redmine-structure.php`
- `TODO.md`
## Local Verification
Passed:
```sh
.venv/bin/python -m py_compile semantic_index/*.py
.venv/bin/python -m unittest discover -s tests/semantic_index
bash -n semantic_index/refresh.sh
```
Observed semantic test result:
```text
Ran 65 tests in 1.041s
OK
```
## LAN Redmine Preview
Passed:
```sh
.venv/bin/python -m semantic_index inspect preview-redmine \
--project customer-service \
--limit 5
```
Observed:
- Helpdesk issue chunks include contact id, name, email, and company metadata.
- Issue `39779` includes Callum Mackeonis and `callum@safetagtracking.com`.
- Journals are present as separate indexed documents.
- Contact documents are present as separate indexed documents.
## Qdrant Audit
Passed:
```sh
.venv/bin/python -m semantic_index inspect audit --source redmine --limit 5000 --json
```
Observed:
```text
total_documents=2947
doc_type contact=714
doc_type issue=1208
doc_type journal=1025
project business-development=66
project customer-service=1684
project dock-scheduling=63
project hiring=409
project prep-standardization=25
project sales-inbox=192
project todo-jason=508
contact_metadata=2232
helpdesk_contact_metadata=2232/2232
attachments=0
```
## HTTP Validation
Passed:
```sh
curl -sS http://127.0.0.1:8787/health
```
Observed:
```json
{"status":"ok"}
```
Unauthenticated `/projects` correctly returned unauthorized when
`SEMANTIC_INDEX_API_KEY` was configured.
Authenticated `/projects` passed and returned the expected seven projects:
```text
business-development
customer-service
dock-scheduling
hiring
prep-standardization
sales-inbox
todo-jason
```
HTTP search passed:
```sh
semantic_index/search.sh "goods return" customer-service 3
```
Observed:
- Top result was `redmine:issue:39779:chunk:0`.
- Citation included project `customer-service`.
- Citation included contact id `1890`, contact name, contact email, and Redmine
URL.
## Refresh Validation
Passed safe dry-run smoke check:
```sh
SEMANTIC_INDEX_PROJECT_LIMITS='customer-service=5' semantic_index/refresh.sh
```
Observed:
```text
mode=dry-run
issues=5
scanned_issues=5
detail_fetched_issues=0
skipped_issues=5
would_embed_documents=0
embedded_documents=0
```
This confirms the refresh state prefilter skips old issues before Redmine detail
fetch and before embedding.
## Qdrant Validation
Read-only collection check passed:
```text
collection=redmine_semantic_sample
status=green
vector_size=1536
distance=Cosine
points_count=2947
update_queue.length=0
```
Read-only snapshot listing endpoint responded successfully:
```text
/collections/redmine_semantic_sample/snapshots
result=[]
```
No snapshot was created during this validation.
## Remaining Pre-Deployment Items
- Decide final target host paths for logs and refresh state.
- Decide service manager shape: manual `uvicorn`, systemd service, or another
supervisor.
- Create or confirm a Qdrant snapshot immediately before production backfill.
- Package only the semantic-index deploy unit, keeping unrelated `redMCP`
worktree changes out of the release.
- Keep scheduled refresh disabled until manual dry-run and `--apply` logs are
reviewed on the target host.
docs/semantic_index_production_notes.md
+76
View File
@@ -0,0 +1,76 @@
# Semantic Index Production Notes
These notes capture the current production direction for the Redmine semantic
index. The service is still local-agent oriented, but the refresh command is now
shaped so it can later be run by cron or systemd without changing the command.
Use `docs/semantic_index_deployment_runbook.md` for the full deploy, validation,
and rollback checklist.
## Routine Refresh
Use the wrapper from the repository root:
```sh
semantic_index/refresh.sh
```
By default this is a dry-run. It does not call OpenAI for document embeddings
and does not write to Qdrant. To apply a rolling refresh:
```sh
semantic_index/refresh.sh --apply
```
The wrapper writes a timestamped log under `.cache/semantic_index/logs` and uses
`.cache/semantic_index/refresh_state.json` for rolling refresh state.
## Production Overrides
Use environment variables rather than editing the script:
```sh
SEMANTIC_INDEX_PROJECT_LIMITS='customer-service=500,hiring=200,todo-jason=200,sales-inbox=100,business-development=100,dock-scheduling=100,prep-standardization=100'
SEMANTIC_INDEX_LOG_DIR=/var/log/semantic-index
SEMANTIC_INDEX_STATE_PATH=/var/lib/semantic-index/refresh_state.json
SEMANTIC_INDEX_OVERLAP_MINUTES=15
```
Keep `OPENAI_API_KEY`, `QDRANT_URL`, `REDMINE_URL`, and `REDMINE_API_KEY` in the
existing `.env` workflow or in the service manager environment.
For production-style deployment, use `/opt/semantic-index` for code,
`/etc/semantic-index.env` for service environment, `/var/lib/semantic-index`
for refresh state, and `/var/log/semantic-index` for refresh logs. Systemd
templates live in `deploy/semantic-index/`.
## Embedding Cost Guard
Normal refresh embeds only documents that are new or whose Redmine-derived
`source_hash` changed. Unchanged documents are left alone. Stale indexed
documents for refreshed issues are deleted without embedding.
Do not schedule `--force-rebuild`. Use it only as a manual maintenance action
when intentionally re-embedding unchanged documents.
## Cron Shape
A later cron entry can call the same wrapper:
```cron
*/30 * * * * cd /home/iadnah/redmine && semantic_index/refresh.sh --apply
```
Before adding a real schedule, run the wrapper manually and confirm the log
shows expected `embedded_documents`, `unchanged_documents`, and
`skipped_issues` counts.
For a quick wrapper smoke check, reduce the project limits:
```sh
SEMANTIC_INDEX_PROJECT_LIMITS='customer-service=5' semantic_index/refresh.sh
```
After refresh state exists, routine dry-runs should show old issues as
`skipped_issues` without matching `detail_fetched_issues`. That indicates the
refresh is avoiding unnecessary Redmine detail requests before it reaches the
embedding cost guard.