Initial Redmine tooling and local plugin forks

2026-04-24 22:01:18 +00:00
commit 9f682af0eb
683 changed files with 56878 additions and 0 deletions
@@ -0,0 +1,338 @@
+# AGENT.md
+
+This file is a restart brief for future agent sessions on this repository.
+
+## Project Identity
+
+This repository supports a legacy, business-critical Redmine 3.4.4
+installation. The real purpose is to make CRM/helpdesk communication data
+searchable and automatable without a risky near-term platform migration.
+
+The LAN copy used for testing is:
+
+```text
+http://192.168.50.170/
+```
+
+This is not a greenfield app. Treat it as a local-fork-and-integration project
+around an existing Redmine install and old RedmineUP plugins.
+
+Tracked local plugin source lives in `plugins/`. The full `redmine-copy/` tree
+is ignored and should be treated as a working/reference copy, not the source of
+truth for local plugin changes.
+
+## Original Motivation
+
+The business stores most external communication inside Redmine through:
+
+- `redmine_contacts`
+- `redmine_contacts_helpdesk`
+
+The default UI/API is not good enough for:
+
+- efficient contact search
+- light contact maintenance
+- helpdesk message search
+- customer communication timeline lookup
+- future semantic/vector search
+
+The key realization from prior work:
+
+```text
+Helpdesk-created issues may have Anonymous issue authors.
+The real customer identity lives in helpdesk_tickets, journal_messages, and contacts.
+```
+
+Never design the search/index layer as if `issues.author` were enough.
+
+## High-Level Architecture Decision
+
+The intended architecture is:
+
+1. Redmine writes local outbox rows.
+2. External worker reads those rows.
+3. Worker enriches from read-only MySQL joins.
+4. Worker builds ticket/message documents.
+5. External index stores/searches those documents.
+
+Safety rule:
+
+```text
+External failures must not break Redmine saves.
+```
+
+That is why the event boundary is local DB outbox, not request-time webhooks or
+embedding calls.
+
+## Redmine And Plugin Baseline
+
+- Redmine version: `3.4.4`
+- Old RedmineUP plugin versions in use:
+  - `redmine_contacts` 4.1.2 PRO
+  - `redmine_contacts_helpdesk` 3.0.9 PRO
+- Near-term upgrade to newer Redmine/RedmineUP is not the current goal.
+- Treat these plugins as local legacy code when necessary.
+
+## Repository Landmarks
+
+Top-level docs:
+
+- [README.md](/home/iadnah/redmine/README.md:1)
+- [docs/event_outbox_spec.md](/home/iadnah/redmine/docs/event_outbox_spec.md:1)
+- [docs/redmineup_local_fork_changelog.md](/home/iadnah/redmine/docs/redmineup_local_fork_changelog.md:1)
+- [docs/pre_existing_issues.md](/home/iadnah/redmine/docs/pre_existing_issues.md:1)
+
+Main scripts:
+
+- [redmine_contacts.py](/home/iadnah/redmine/redmine_contacts.py:1)
+- [redmine_helpdesk_search.py](/home/iadnah/redmine/redmine_helpdesk_search.py:1)
+- [redmine_outbox_worker.py](/home/iadnah/redmine/redmine_outbox_worker.py:1)
+
+Local Redmine copy:
+
+- [redmine-copy](/home/iadnah/redmine/redmine-copy)
+
+Important local plugin paths:
+
+- [redmine-copy/plugins/redmine_event_outbox](/home/iadnah/redmine/redmine-copy/plugins/redmine_event_outbox)
+- [redmine-copy/plugins/redmine_contacts_helpdesk](/home/iadnah/redmine/redmine-copy/plugins/redmine_contacts_helpdesk)
+
+## What Has Already Been Done
+
+### Contact CLI
+
+`redmine_contacts.py` exists and supports:
+
+- contact fetch to local cache
+- fuzzy-ish contact search
+- light contact updates
+- helpdesk read API calls:
+  - ticket by issue
+  - issues by contact
+  - messages by issue
+  - contact timeline
+
+### Event Outbox Plugin
+
+`redmine_event_outbox` exists and the known-good archive is:
+
+- `dist/redmine_event_outbox-0.0.1-known-good-20260421T143957Z.tar.gz`
+
+Known-good tested LAN events:
+
+- `issue.created`
+- `issue.updated`
+- `journal.created`
+- `contact.created`
+- `contact.updated`
+
+Local code also adds optional helpdesk events:
+
+- `helpdesk_ticket.created`
+- `helpdesk_ticket.updated`
+- `journal_message.created`
+- `journal_message.updated`
+
+These helpdesk event paths are implemented locally but should still be treated
+as needing fuller end-to-end validation.
+
+### Helpdesk Read API In Local Plugin Fork
+
+The local `redmine_contacts_helpdesk` fork includes:
+
+- `helpdesk_search_controller.rb`
+- routes under `/helpdesk_search/*`
+- alias/usage routes to avoid noisy routing errors
+
+This was deployed to the LAN copy and route-loaded successfully.
+
+### Helpdesk Export/Search CLI
+
+`redmine_helpdesk_search.py` was created to prove the data model and export
+path. It:
+
+- SSHes to the LAN Redmine host
+- reads remote MySQL credentials from `config/database.yml`
+- runs read-only MySQL queries
+- exports ticket/message docs to local JSONL cache
+- provides rough local search/timeline/issues-by-contact commands
+
+Important:
+
+```text
+This script is a diagnostic/export tool, not the final search architecture.
+```
+
+Do not get dragged into treating local CLI search speed as the core mission.
+
+### External Outbox Worker Prototype
+
+`redmine_outbox_worker.py` now exists as the first worker/indexer boundary.
+It:
+
+- reads and optionally claims pending `event_outbox_events` over SSH/MySQL
+- supports `--dry-run` for non-mutating previews
+- uses `locked_at`, `locked_by`, `processed_at`, `attempts`, and `last_error`
+- enriches helpdesk ticket/message/contact events with read-only joins
+- writes derived JSONL to `.cache/redmine_outbox/derived_documents.jsonl`
+- marks rows processed only after a successful local write
+
+This is still a prototype output target, not the final vector index.
+
+## LAN Host Facts
+
+These were verified previously:
+
+- SSH host/user: `reddev@192.168.50.170`
+- SSH key used previously: `/tmp/reddev`
+- remote Redmine path: `/usr/share/redmine`
+- remote Ruby: `2.5.1`
+- remote Bundler: `1.17.3`
+- remote DB: MySQL 5.7
+- remote plugin rollback archives were stored in:
+  - `/home/reddev/redmine-plugin-backups/`
+
+If using those exact credentials again, verify they still exist before relying
+on them.
+
+## Remote Plugin Changes Previously Deployed
+
+Changes were copied to the LAN Redmine host for:
+
+- `plugins/redmine_event_outbox/...`
+- `plugins/redmine_contacts_helpdesk/...`
+
+Passenger was restarted with:
+
+```text
+touch /usr/share/redmine/tmp/restart.txt
+```
+
+Routes were verified by remote `rake routes`.
+
+## Fork Hygiene Rules
+
+Before editing RedmineUP plugin code:
+
+1. create a rollback archive in `dist/`
+2. record the change in:
+   - `docs/redmineup_local_fork_changelog.md`
+   - plugin-local changelog if appropriate
+3. keep edits scoped
+4. prefer read-only APIs/endpoints first
+5. validate on the LAN copy before claiming the change is done
+
+Existing rollback archives:
+
+- `dist/redmine_contacts-4.1.2-local-before-helpdesk-search-20260421T215548Z.tar.gz`
+- `dist/redmine_contacts_helpdesk-3.0.9-local-before-helpdesk-search-20260421T215548Z.tar.gz`
+
+## Important Pre-Existing Issues
+
+Read:
+
+- [docs/pre_existing_issues.md](/home/iadnah/redmine/docs/pre_existing_issues.md:1)
+
+Especially remember:
+
+1. duplicate `id="avatar"` is likely a real long-standing UI bug
+2. `attachments/contacts_thumbnail` digest warning is probably log noise
+3. `acts_as_list` warning is an upgrade-compatibility note, not a current
+   blocker
+
+Do not confuse those with the main search deliverable unless they directly block
+the work in front of you.
+
+## Current Strategic State
+
+The project is past the "can we get data out?" phase.
+
+We now know:
+
+- contact JSON access exists
+- helpdesk/customer identity can be extracted
+- outbox events can be recorded safely
+- helpdesk read API routes can be added locally
+- full ticket/message export is feasible
+
+The most useful next step is:
+
+```text
+Build the external worker/indexer pipeline.
+```
+
+Not:
+
+- polishing a local CLI into a full search product
+- building a big Redmine admin UI
+- chasing unrelated legacy plugin cleanup
+
+## Recommended Next Steps For A Future Session
+
+1. Re-read:
+   - `README.md`
+   - `docs/event_outbox_spec.md`
+   - `docs/redmineup_local_fork_changelog.md`
+   - `docs/pre_existing_issues.md`
+2. Verify local and remote state:
+   - plugin files present
+   - LAN host reachable
+   - remote routes still load
+3. Validate `redmine_outbox_worker.py` end to end against the LAN copy
+4. Refine read-only joins that enrich:
+   - `helpdesk_tickets`
+   - `journal_messages`
+   - `contacts`
+   - `issues`
+   - `journals`
+5. Confirm deterministic ticket/message/contact JSONL output
+6. Only after that, connect Qdrant and embeddings
+
+## Query/Data Model Notes
+
+Authoritative helpdesk context lives in:
+
+- `helpdesk_tickets`
+  - `issue_id`
+  - `contact_id`
+  - `from_address`
+  - `to_address`
+  - `cc_address`
+  - `message_id`
+  - `ticket_date`
+  - `source`
+- `journal_messages`
+  - `journal_id`
+  - `contact_id`
+  - `from_address`
+  - `to_address`
+  - `cc_address`
+  - `bcc_address`
+  - `message_id`
+  - `is_incoming`
+  - `message_date`
+- `contacts`
+- `issues`
+- `journals`
+
+The worker/indexer should prefer direct DB joins as the source of truth for this
+layer.
+
+## What To Be Careful About
+
+- Do not assume Redmine core issue API data is enough for helpdesk search.
+- Do not treat old RedmineUP code as untouchable vendor code.
+- Do not claim a plugin change is validated just because `ruby -c` passes.
+- Do not burn time optimizing local CLI search if the real goal is external
+  indexing.
+- Do not revert unrelated repo changes; the Redmine tree may be dirty.
+
+## If You Need A One-Paragraph Summary
+
+This project exists to safely extract and search real CRM/helpdesk communication
+history from a legacy Redmine 3.4.4 install. We already have a contact CLI, a
+local event outbox plugin, local helpdesk read API routes, and a rough
+helpdesk-export CLI. The next meaningful milestone is an external worker that
+consumes outbox rows, enriches from helpdesk/contact MySQL tables, and builds
+deterministic ticket/message documents for a future Qdrant/OpenAI-backed search
+index.