Initial Redmine tooling and local plugin forks

2026-04-24 22:01:18 +00:00
commit 9f682af0eb
683 changed files with 56878 additions and 0 deletions
@@ -0,0 +1,588 @@
+# Redmine Event Outbox Spec
+
+## Purpose
+
+Add a low-risk event boundary around the legacy Redmine install so external tools
+can react to Redmine changes without polling heavily and without making Redmine
+dependent on a message bus, search service, or experimental automation code.
+
+The first version should capture issue and journal activity into a local database
+outbox table. A separate worker can later publish those rows to Redis, RabbitMQ,
+a webhook, or a search/indexing service.
+
+## Goals
+
+- Record issue creation and update events.
+- Record issue journal/comment events.
+- Record contact creation and update events from `redmine_contacts`.
+- Keep Redmine issue saves working even if external messaging/search systems are
+  down.
+- Make event processing replayable from a durable local table.
+- Keep the plugin small and compatible with Redmine 3.4.4-era plugin patterns.
+- Support future indexing for queries like "recent events for customer A".
+
+## Non-Goals For V1
+
+- No direct Redis/RabbitMQ publish inside Redmine request callbacks.
+- No semantic/vector search inside Redmine.
+- No replacement of Redmine email handling in the first version.
+- No guarantee that external consumers receive each event exactly once.
+- No broad UI in Redmine beyond optional admin/status views later.
+- No binary image indexing inside Redmine.
+
+## Safety Rule
+
+Redmine must not fail an issue create/update because a broker, worker, search
+index, DNS lookup, network call, or external service failed.
+
+V1 should therefore only write to Redmine's own database from inside Redmine. All
+network publishing happens outside Redmine in a worker.
+
+There is one important tradeoff:
+
+- If outbox insert failures are rescued, ticket saves are maximally protected but
+  rare event loss is possible.
+- If outbox rows are written transactionally and failures are not rescued, event
+  durability is stronger but ticket saves can fail because the outbox write
+  failed.
+
+For this project, prefer protecting ticket saves. The outbox insert should be
+simple and local, and failures should be logged with enough detail to diagnose.
+
+## Proposed Plugin
+
+Plugin name:
+
+```text
+redmine_event_outbox
+```
+
+Directory:
+
+```text
+redmine-copy/plugins/redmine_event_outbox/
+```
+
+Initial structure:
+
+```text
+init.rb
+db/migrate/001_create_event_outbox_events.rb
+app/models/event_outbox_event.rb
+lib/redmine_event_outbox.rb
+lib/redmine_event_outbox/hooks/issues_hook.rb
+lib/redmine_event_outbox/patches/contact_patch.rb
+lib/redmine_event_outbox/patches/journal_patch.rb
+lib/tasks/redmine_event_outbox.rake
+```
+
+## Event Table
+
+Table name:
+
+```text
+event_outbox_events
+```
+
+Columns:
+
+```text
+id                integer primary key
+event_type        string, required
+source_type       string, required
+source_id         integer, required
+project_id        integer, nullable
+issue_id          integer, nullable
+journal_id        integer, nullable
+user_id           integer, nullable
+occurred_at       datetime, required
+payload           text/json, required
+processed_at      datetime, nullable
+attempts          integer, default 0
+last_error        text, nullable
+locked_at         datetime, nullable
+locked_by         string, nullable
+created_at        datetime
+updated_at        datetime
+```
+
+Production Redmine currently uses MySQL. Store `payload` as `text` for Redmine
+3.4.4 compatibility and serialize JSON in application code. Do not rely on
+native MySQL JSON behavior in V1.
+
+Useful indexes:
+
+```text
+index_event_outbox_events_on_processed_at_id
+index_event_outbox_events_on_event_type
+index_event_outbox_events_on_issue_id
+index_event_outbox_events_on_project_id
+index_event_outbox_events_on_occurred_at
+```
+
+## V1 Event Types
+
+### issue.created
+
+Created after a new issue is saved.
+
+Trigger candidate found in Redmine 3.4.4:
+
+```ruby
+controller_issues_new_after_save
+```
+
+Payload:
+
+```json
+{
+  "event_type": "issue.created",
+  "issue_id": 123,
+  "project_id": 4,
+  "tracker_id": 1,
+  "status_id": 1,
+  "priority_id": 2,
+  "author_id": 5,
+  "author_name": "Jane User",
+  "assigned_to_id": null,
+  "assigned_to_name": null,
+  "subject": "Example",
+  "created_on": "2026-04-21T12:00:00Z",
+  "updated_on": "2026-04-21T12:00:00Z"
+}
+```
+
+### issue.updated
+
+Created after an existing issue update succeeds.
+
+Trigger candidate found in Redmine 3.4.4:
+
+```ruby
+controller_issues_edit_after_save
+```
+
+Payload should include issue identity plus the current journal id when present:
+
+```json
+{
+  "event_type": "issue.updated",
+  "issue_id": 123,
+  "journal_id": 456,
+  "project_id": 4,
+  "status_id": 2,
+  "assigned_to_id": 7,
+  "assigned_to_name": "Support User",
+  "actor_id": 5,
+  "actor_name": "Jane User",
+  "subject": "Example",
+  "updated_on": "2026-04-21T12:15:00Z"
+}
+```
+
+### journal.created
+
+Created when a journal row for an issue is committed.
+
+Trigger candidate found in Redmine 3.4.4:
+
+```ruby
+Journal.after_commit :on => :create
+```
+
+The plugin can patch `Journal` with a separate `after_commit` callback.
+
+Payload:
+
+```json
+{
+  "event_type": "journal.created",
+  "journal_id": 456,
+  "issue_id": 123,
+  "project_id": 4,
+  "user_id": 5,
+  "user_name": "Jane User",
+  "subject": "Example",
+  "private_notes": false,
+  "has_notes": true,
+  "changed_fields": ["status_id", "assigned_to_id"],
+  "created_on": "2026-04-21T12:15:00Z"
+}
+```
+
+V1 should not put full private note text into the payload. The indexing worker can
+fetch detail with appropriate credentials if needed.
+
+### contact.created
+
+Created after a contact is committed.
+
+Trigger candidate:
+
+```ruby
+Contact.after_commit :on => :create
+```
+
+The plugin can patch `Contact` from `redmine_contacts` when that plugin is
+installed.
+
+Payload:
+
+```json
+{
+  "event_type": "contact.created",
+  "contact_id": 321,
+  "project_ids": [4],
+  "is_company": false,
+  "name": "Customer Name",
+  "company": "Customer Company",
+  "author_id": 5,
+  "author_name": "Jane User",
+  "assigned_to_id": 7,
+  "assigned_to_name": "Support User",
+  "created_on": "2026-04-21T12:20:00Z",
+  "updated_on": "2026-04-21T12:20:00Z"
+}
+```
+
+### contact.updated
+
+Created after a contact update is committed.
+
+Trigger candidate:
+
+```ruby
+Contact.after_commit :on => :update
+```
+
+Payload:
+
+```json
+{
+  "event_type": "contact.updated",
+  "contact_id": 321,
+  "project_ids": [4],
+  "is_company": false,
+  "name": "Customer Name",
+  "company": "Customer Company",
+  "actor_id": 5,
+  "actor_name": "Jane User",
+  "assigned_to_id": 7,
+  "assigned_to_name": "Support User",
+  "updated_on": "2026-04-21T12:25:00Z"
+}
+```
+
+Contact payloads should include enough human context for downstream consumers to
+decide whether to fetch the full contact. Avoid embedding all phone/email/address
+data in V1 payloads unless a consumer proves it needs those fields inline.
+
+### helpdesk_ticket.created
+
+Created after a `HelpdeskTicket` row is committed when
+`redmine_contacts_helpdesk` is installed.
+
+Payload:
+
+```json
+{
+  "event_type": "helpdesk_ticket.created",
+  "helpdesk_ticket_id": 10,
+  "issue_id": 123,
+  "project_id": 4,
+  "contact_id": 321,
+  "message_id": "<message@example>",
+  "is_incoming": true,
+  "source": 0,
+  "from_address": "customer@example.com",
+  "to_address": "support@example.com",
+  "cc_address": null,
+  "subject": "Example",
+  "ticket_date": "2026-04-21T12:00:00Z"
+}
+```
+
+### helpdesk_ticket.updated
+
+Created after an existing `HelpdeskTicket` row is updated. This is useful when
+the contact, source, or message metadata is corrected after ticket creation.
+
+### journal_message.created
+
+Created after a `JournalMessage` row is committed. This is the authoritative
+per-email metadata layer for helpdesk conversation search.
+
+Payload:
+
+```json
+{
+  "event_type": "journal_message.created",
+  "journal_message_id": 20,
+  "journal_id": 456,
+  "issue_id": 123,
+  "project_id": 4,
+  "contact_id": 321,
+  "message_id": "<reply@example>",
+  "is_incoming": false,
+  "source": 0,
+  "from_address": "support@example.com",
+  "to_address": "customer@example.com",
+  "cc_address": null,
+  "has_bcc_address": false,
+  "private_notes": false,
+  "has_notes": true,
+  "message_date": "2026-04-21T12:15:00Z"
+}
+```
+
+The event records whether BCC metadata exists but does not store the BCC address
+itself.
+
+### journal_message.updated
+
+Created after an existing `JournalMessage` row is updated. This lets downstream
+indexes repair message-level documents when message metadata is corrected.
+
+## Payload Context Policy
+
+Payloads should be lightweight but useful. Include:
+
+- ids needed for follow-up fetches
+- event type and timestamp
+- issue/contact subject or display name
+- user id and user name when known
+- project id(s)
+- changed field names when available
+
+Avoid:
+
+- full private notes
+- large descriptions/background fields
+- attachments or binary content
+- full email bodies
+- BCC addresses
+- large custom field dumps
+
+This gives consumers enough information to decide whether they care about an
+event while keeping the outbox table compact and lower-risk.
+
+## Delivery Semantics
+
+V1 should provide at-least-once processing from the outbox to downstream systems.
+
+Consumers must be idempotent. Use the outbox row id as the event id:
+
+```json
+{
+  "event_id": 98765,
+  "event_type": "issue.updated"
+}
+```
+
+Duplicates are acceptable. Silent message loss is not acceptable once an outbox
+row exists.
+
+## Worker/Rake Task
+
+Current implemented worker-facing command:
+
+```sh
+bundle exec rake redmine_event_outbox:dump RAILS_ENV=production
+```
+
+This prints pending rows as JSON and does not mark them processed.
+
+Next worker command:
+
+```sh
+bundle exec rake redmine_event_outbox:publish RAILS_ENV=production
+```
+
+V1 modes:
+
+```sh
+# Print pending rows as JSON without marking processed.
+bundle exec rake redmine_event_outbox:dump
+
+# Process pending rows and mark success.
+bundle exec rake redmine_event_outbox:publish
+
+# Retry failed/unprocessed rows.
+bundle exec rake redmine_event_outbox:publish RETRY=1
+```
+
+Initial publisher target can be stdout or a local JSONL file. That lets us test
+the Redmine side before choosing Redis Streams or RabbitMQ.
+
+The next implementation should keep the worker small and conservative:
+
+- select pending rows in id order
+- lock or claim a bounded batch
+- publish each row
+- mark `processed_at` only after publish succeeds
+- increment `attempts` and write `last_error` on failure
+- leave failed rows available for retry
+- make duplicate delivery acceptable to consumers
+
+Later publisher targets:
+
+- Redis Streams
+- RabbitMQ
+- webhook HTTP POST
+- local search/indexing service
+
+## Search/Indexing Direction
+
+The semantic/fuzzy search system should be external to Redmine.
+
+The first external search index should have strong vector-search support because
+future work will use embeddings heavily, including image embeddings. Redmine
+should only emit events and identifiers; the external indexer should fetch,
+transform, chunk, embed, and store searchable material.
+
+Likely derived entities:
+
+- contacts
+- issues
+- journals
+- helpdesk email messages
+- status/assignee changes
+- timestamps and project links
+- future image/document embeddings
+
+Likely query examples:
+
+```text
+recent events for customer A
+open issues involving customer A
+recent emails from customer A
+status changes for customer A this week
+```
+
+The outbox does not need to contain all searchable text. It only needs enough
+identity and timestamps for a worker to fetch and update the external index.
+
+Candidate vector-capable search/index options to evaluate later:
+
+- PostgreSQL with `pgvector`
+- Qdrant
+- Weaviate
+- OpenSearch/Elasticsearch with vector fields
+- LanceDB
+
+Since production Redmine uses MySQL, the search/index database should be treated
+as a separate derived system rather than as a feature of the Redmine database.
+
+## Email Handling Direction
+
+Email handling should be handled after the basic issue/journal outbox is stable.
+
+Observed areas to inspect further:
+
+- Redmine core `MailHandler`
+- `redmine_contacts` rake tasks under `lib/tasks/contacts_email.rake`
+- `redmine_contacts_helpdesk`
+- `helpdesk_mailer` endpoint and related mail handling code
+
+Future event types may include:
+
+```text
+email.received
+email.ignored
+email.created_issue
+email.updated_issue
+helpdesk_ticket.created
+helpdesk_ticket.updated
+```
+
+Custom mail routing should have a safe fallback to current behavior if external
+decision code fails.
+
+## Implementation Phases
+
+### Phase 1: Outbox Skeleton
+
+- Status: implemented and tested on the LAN Redmine copy.
+- Created plugin.
+- Added migration and model.
+- Added helper method for safe event creation.
+- Added issue create/update hooks.
+- Added basic rake task to dump pending rows.
+- Verified issue create/update writes outbox rows in the `Meetings` project.
+
+### Phase 2: Journal Events
+
+- Status: implemented and tested on the LAN Redmine copy.
+- Patched `Journal` with `after_commit`.
+- Records `journal.created`.
+- Includes changed field names from journal details.
+- Avoids full private note content in payload.
+
+### Phase 3: Contact Events
+
+- Status: implemented and tested on the LAN Redmine copy.
+- Patched `Contact` from `redmine_contacts` if available.
+- Records `contact.created`.
+- Records `contact.updated`.
+- Includes contact display name, company, project ids, and relevant user context.
+- Keeps payloads small; full contact detail can be fetched by external workers.
+
+### Phase 4: Worker
+
+- Add locking fields.
+- Process pending rows in batches.
+- Mark `processed_at` only after successful publish.
+- Track attempts and last error.
+- Support dry-run/dump mode.
+
+### Phase 5: External Index Prototype
+
+- Build a small external worker outside Redmine.
+- Read outbox rows.
+- Fetch affected issue/contact/journal details through API or DB.
+- Store in a vector-capable search database.
+- Add a CLI for recent customer timelines.
+
+### Phase 6: Message Bus
+
+- Choose Redis Streams or RabbitMQ based on actual consumer needs.
+- Keep Redmine unchanged; only the worker publisher changes.
+
+## Open Questions
+
+- How long should processed outbox rows be retained?
+- Do we need a Redmine admin page to inspect outbox health, or is a rake task/log enough?
+- What is the current production mail ingestion path?
+- Which vector-capable external search index should be used first?
+- Should contact payloads include normalized primary email/phone, or is that too much inline data?
+
+## Current Checkpoint
+
+Known-good archive:
+
+```text
+dist/redmine_event_outbox-0.0.1-known-good-20260421T143957Z.tar.gz
+```
+
+Manifest:
+
+```text
+dist/redmine_event_outbox-0.0.1-known-good-20260421T143957Z.MANIFEST.md
+```
+
+Tested on LAN Redmine copy:
+
+- Migrated successfully.
+- Passenger restart via `touch tmp/restart.txt`.
+- Verified `issue.created`.
+- Verified `issue.updated`.
+- Verified `journal.created`.
+- Verified `contact.created`.
+- Verified `contact.updated`.
+- Verified `redmine_event_outbox:dump`.
+
+Test artifacts left on LAN copy for traceability:
+
+- Issue `#39858` in project `Meetings`.
+- Contact `#4337` in project `Email Test`.
+- Six outbox rows in `event_outbox_events`.