redmine/docs/event_outbox_spec.md

# Redmine Event Outbox Spec

## Purpose

Add a low-risk event boundary around the legacy Redmine install so external tools
can react to Redmine changes without polling heavily and without making Redmine
dependent on a message bus, search service, or experimental automation code.

The first version should capture issue and journal activity into a local database
outbox table. A separate worker can later publish those rows to Redis, RabbitMQ,
a webhook, or a search/indexing service.

## Goals

- Record issue creation and update events.
- Record issue journal/comment events.
- Record contact creation and update events from `redmine_contacts`.
- Keep Redmine issue saves working even if external messaging/search systems are
  down.
- Make event processing replayable from a durable local table.
- Keep the plugin small and compatible with Redmine 3.4.4-era plugin patterns.
- Support future indexing for queries like "recent events for customer A".

## Non-Goals For V1

- No direct Redis/RabbitMQ publish inside Redmine request callbacks.
- No semantic/vector search inside Redmine.
- No replacement of Redmine email handling in the first version.
- No guarantee that external consumers receive each event exactly once.
- No broad UI in Redmine beyond optional admin/status views later.
- No binary image indexing inside Redmine.

## Safety Rule

Redmine must not fail an issue create/update because a broker, worker, search
index, DNS lookup, network call, or external service failed.

V1 should therefore only write to Redmine's own database from inside Redmine. All
network publishing happens outside Redmine in a worker.

There is one important tradeoff:

- If outbox insert failures are rescued, ticket saves are maximally protected but
  rare event loss is possible.
- If outbox rows are written transactionally and failures are not rescued, event
  durability is stronger but ticket saves can fail because the outbox write
  failed.

For this project, prefer protecting ticket saves. The outbox insert should be
simple and local, and failures should be logged with enough detail to diagnose.

## Proposed Plugin

Plugin name:

```text
redmine_event_outbox
```

Directory:

```text
redmine-copy/plugins/redmine_event_outbox/
```

Initial structure:

```text
init.rb
db/migrate/001_create_event_outbox_events.rb
app/models/event_outbox_event.rb
lib/redmine_event_outbox.rb
lib/redmine_event_outbox/hooks/issues_hook.rb
lib/redmine_event_outbox/patches/contact_patch.rb
lib/redmine_event_outbox/patches/journal_patch.rb
lib/tasks/redmine_event_outbox.rake
```

## Event Table

Table name:

```text
event_outbox_events
```

Columns:

```text
id                integer primary key
event_type        string, required
source_type       string, required
source_id         integer, required
project_id        integer, nullable
issue_id          integer, nullable
journal_id        integer, nullable
user_id           integer, nullable
occurred_at       datetime, required
payload           text/json, required
processed_at      datetime, nullable
attempts          integer, default 0
last_error        text, nullable
locked_at         datetime, nullable
locked_by         string, nullable
created_at        datetime
updated_at        datetime
```

Production Redmine currently uses MySQL. Store `payload` as `text` for Redmine
3.4.4 compatibility and serialize JSON in application code. Do not rely on
native MySQL JSON behavior in V1.

Useful indexes:

```text
index_event_outbox_events_on_processed_at_id
index_event_outbox_events_on_event_type
index_event_outbox_events_on_issue_id
index_event_outbox_events_on_project_id
index_event_outbox_events_on_occurred_at
```

## V1 Event Types

### issue.created

Created after a new issue is saved.

Trigger candidate found in Redmine 3.4.4:

```ruby
controller_issues_new_after_save
```

Payload:

```json
{
  "event_type": "issue.created",
  "issue_id": 123,
  "project_id": 4,
  "tracker_id": 1,
  "status_id": 1,
  "priority_id": 2,
  "author_id": 5,
  "author_name": "Jane User",
  "assigned_to_id": null,
  "assigned_to_name": null,
  "subject": "Example",
  "created_on": "2026-04-21T12:00:00Z",
  "updated_on": "2026-04-21T12:00:00Z"
}
```

### issue.updated

Created after an existing issue update succeeds.

Trigger candidate found in Redmine 3.4.4:

```ruby
controller_issues_edit_after_save
```

Payload should include issue identity plus the current journal id when present:

```json
{
  "event_type": "issue.updated",
  "issue_id": 123,
  "journal_id": 456,
  "project_id": 4,
  "status_id": 2,
  "assigned_to_id": 7,
  "assigned_to_name": "Support User",
  "actor_id": 5,
  "actor_name": "Jane User",
  "subject": "Example",
  "updated_on": "2026-04-21T12:15:00Z"
}
```

### journal.created

Created when a journal row for an issue is committed.

Trigger candidate found in Redmine 3.4.4:

```ruby
Journal.after_commit :on => :create
```

The plugin can patch `Journal` with a separate `after_commit` callback.

Payload:

```json
{
  "event_type": "journal.created",
  "journal_id": 456,
  "issue_id": 123,
  "project_id": 4,
  "user_id": 5,
  "user_name": "Jane User",
  "subject": "Example",
  "private_notes": false,
  "has_notes": true,
  "changed_fields": ["status_id", "assigned_to_id"],
  "created_on": "2026-04-21T12:15:00Z"
}
```

V1 should not put full private note text into the payload. The indexing worker can
fetch detail with appropriate credentials if needed.

### contact.created

Created after a contact is committed.

Trigger candidate:

```ruby
Contact.after_commit :on => :create
```

The plugin can patch `Contact` from `redmine_contacts` when that plugin is
installed.

Payload:

```json
{
  "event_type": "contact.created",
  "contact_id": 321,
  "project_ids": [4],
  "is_company": false,
  "name": "Customer Name",
  "company": "Customer Company",
  "author_id": 5,
  "author_name": "Jane User",
  "assigned_to_id": 7,
  "assigned_to_name": "Support User",
  "created_on": "2026-04-21T12:20:00Z",
  "updated_on": "2026-04-21T12:20:00Z"
}
```

### contact.updated

Created after a contact update is committed.

Trigger candidate:

```ruby
Contact.after_commit :on => :update
```

Payload:

```json
{
  "event_type": "contact.updated",
  "contact_id": 321,
  "project_ids": [4],
  "is_company": false,
  "name": "Customer Name",
  "company": "Customer Company",
  "actor_id": 5,
  "actor_name": "Jane User",
  "assigned_to_id": 7,
  "assigned_to_name": "Support User",
  "updated_on": "2026-04-21T12:25:00Z"
}
```

Contact payloads should include enough human context for downstream consumers to
decide whether to fetch the full contact. Avoid embedding all phone/email/address
data in V1 payloads unless a consumer proves it needs those fields inline.

### helpdesk_ticket.created

Created after a `HelpdeskTicket` row is committed when
`redmine_contacts_helpdesk` is installed.

Payload:

```json
{
  "event_type": "helpdesk_ticket.created",
  "helpdesk_ticket_id": 10,
  "issue_id": 123,
  "project_id": 4,
  "contact_id": 321,
  "message_id": "<message@example>",
  "is_incoming": true,
  "source": 0,
  "from_address": "customer@example.com",
  "to_address": "support@example.com",
  "cc_address": null,
  "subject": "Example",
  "ticket_date": "2026-04-21T12:00:00Z"
}
```

### helpdesk_ticket.updated

Created after an existing `HelpdeskTicket` row is updated. This is useful when
the contact, source, or message metadata is corrected after ticket creation.

### journal_message.created

Created after a `JournalMessage` row is committed. This is the authoritative
per-email metadata layer for helpdesk conversation search.

Payload:

```json
{
  "event_type": "journal_message.created",
  "journal_message_id": 20,
  "journal_id": 456,
  "issue_id": 123,
  "project_id": 4,
  "contact_id": 321,
  "message_id": "<reply@example>",
  "is_incoming": false,
  "source": 0,
  "from_address": "support@example.com",
  "to_address": "customer@example.com",
  "cc_address": null,
  "has_bcc_address": false,
  "private_notes": false,
  "has_notes": true,
  "message_date": "2026-04-21T12:15:00Z"
}
```

The event records whether BCC metadata exists but does not store the BCC address
itself.

### journal_message.updated

Created after an existing `JournalMessage` row is updated. This lets downstream
indexes repair message-level documents when message metadata is corrected.

## Payload Context Policy

Payloads should be lightweight but useful. Include:

- ids needed for follow-up fetches
- event type and timestamp
- issue/contact subject or display name
- user id and user name when known
- project id(s)
- changed field names when available

Avoid:

- full private notes
- large descriptions/background fields
- attachments or binary content
- full email bodies
- BCC addresses
- large custom field dumps

This gives consumers enough information to decide whether they care about an
event while keeping the outbox table compact and lower-risk.

## Delivery Semantics

V1 should provide at-least-once processing from the outbox to downstream systems.

Consumers must be idempotent. Use the outbox row id as the event id:

```json
{
  "event_id": 98765,
  "event_type": "issue.updated"
}
```

Duplicates are acceptable. Silent message loss is not acceptable once an outbox
row exists.

## Worker/Rake Task

Current implemented Redmine-side worker-facing command:

```sh
bundle exec rake redmine_event_outbox:dump RAILS_ENV=production
```

This prints pending rows as JSON and does not mark them processed.

Historical planned Redmine-side publish command:

```sh
bundle exec rake redmine_event_outbox:publish RAILS_ENV=production
```

V1 modes:

```sh
# Print pending rows as JSON without marking processed.
bundle exec rake redmine_event_outbox:dump

# Process pending rows and mark success.
bundle exec rake redmine_event_outbox:publish

# Retry failed/unprocessed rows.
bundle exec rake redmine_event_outbox:publish RETRY=1
```

The current external publisher is `redmine_outbox_worker.py`. Its initial
publisher target is JSONL at `/tmp/redmine-outbox/derived_documents.jsonl`,
which lets us test the Redmine side before choosing Redis Streams or RabbitMQ.

The worker stays small and conservative:

- select pending rows in id order
- lock or claim a bounded batch
- publish each row
- mark `processed_at` only after publish succeeds
- increment `attempts` and write `last_error` on failure
- leave failed rows available for retry
- make duplicate delivery acceptable to consumers

Current external worker commands:

```sh
./redmine_outbox_worker.py --status
./redmine_outbox_worker.py --dry-run --batch-size 10
./redmine_outbox_worker.py --batch-size 20
./redmine_outbox_worker.py --purge-processed-days 30
./redmine_outbox_worker.py --purge-processed-days 30 --apply-purge
```

Processed rows are retained by default. Purge is explicit, age-gated, and
intended for test-instance cleanup, not normal delivery.

Later publisher targets:

- Redis Streams
- RabbitMQ
- webhook HTTP POST
- local search/indexing service

## Search/Indexing Direction

The semantic/fuzzy search system should be external to Redmine.

The first external search index should have strong vector-search support because
future work will use embeddings heavily, including image embeddings. Redmine
should only emit events and identifiers; the external indexer should fetch,
transform, chunk, embed, and store searchable material.

Likely derived entities:

- contacts
- issues
- journals
- helpdesk email messages
- status/assignee changes
- timestamps and project links
- future image/document embeddings

Likely query examples:

```text
recent events for customer A
open issues involving customer A
recent emails from customer A
status changes for customer A this week
```

The outbox does not need to contain all searchable text. It only needs enough
identity and timestamps for a worker to fetch and update the external index.

Candidate vector-capable search/index options to evaluate later:

- PostgreSQL with `pgvector`
- Qdrant
- Weaviate
- OpenSearch/Elasticsearch with vector fields
- LanceDB

Since production Redmine uses MySQL, the search/index database should be treated
as a separate derived system rather than as a feature of the Redmine database.

## Email Handling Direction

Email handling should be handled after the basic issue/journal outbox is stable.

Observed areas to inspect further:

- Redmine core `MailHandler`
- `redmine_contacts` rake tasks under `lib/tasks/contacts_email.rake`
- `redmine_contacts_helpdesk`
- `helpdesk_mailer` endpoint and related mail handling code

Future event types may include:

```text
email.received
email.ignored
email.created_issue
email.updated_issue
helpdesk_ticket.created
helpdesk_ticket.updated
```

Custom mail routing should have a safe fallback to current behavior if external
decision code fails.

## Implementation Phases

### Phase 1: Outbox Skeleton

- Status: implemented and tested on the LAN Redmine copy.
- Created plugin.
- Added migration and model.
- Added helper method for safe event creation.
- Added issue create/update hooks.
- Added basic rake task to dump pending rows.
- Verified issue create/update writes outbox rows in the `Meetings` project.

### Phase 2: Journal Events

- Status: implemented and tested on the LAN Redmine copy.
- Patched `Journal` with `after_commit`.
- Records `journal.created`.
- Includes changed field names from journal details.
- Avoids full private note content in payload.

### Phase 3: Contact Events

- Status: implemented and tested on the LAN Redmine copy.
- Patched `Contact` from `redmine_contacts` if available.
- Records `contact.created`.
- Records `contact.updated`.
- Includes contact display name, company, project ids, and relevant user context.
- Keeps payloads small; full contact detail can be fetched by external workers.

### Phase 4: Worker

- Add locking fields.
- Process pending rows in batches.
- Mark `processed_at` only after successful publish.
- Track attempts and last error.
- Support dry-run/dump mode.

### Phase 5: External Index Prototype

- Build a small external worker outside Redmine.
- Read outbox rows.
- Fetch affected issue/contact/journal details through API or DB.
- Store in a vector-capable search database.
- Add a CLI for recent customer timelines.

### Phase 6: Message Bus

- Choose Redis Streams or RabbitMQ based on actual consumer needs.
- Keep Redmine unchanged; only the worker publisher changes.

## Open Questions

- Do production processed rows need a formal retention window, or should they
  be retained until external index/message-bus replay policy is settled?
- Do we need a Redmine admin page to inspect outbox health, or is a rake task/log enough?
- What is the current production mail ingestion path?
- Which vector-capable external search index should be used first?
- Should contact payloads include normalized primary email/phone, or is that too much inline data?

## Current Checkpoint

Known-good archive:

```text
dist/redmine_event_outbox-0.0.1-known-good-20260421T143957Z.tar.gz
```

Manifest:

```text
dist/redmine_event_outbox-0.0.1-known-good-20260421T143957Z.MANIFEST.md
```

Tested on LAN Redmine copy:

- Migrated successfully.
- Passenger restart via `touch tmp/restart.txt`.
- Verified `issue.created`.
- Verified `issue.updated`.
- Verified `journal.created`.
- Verified `contact.created`.
- Verified `contact.updated`.
- Verified `redmine_event_outbox:dump`.

Test artifacts left on LAN copy for traceability:

- Issue `#39858` in project `Meetings`.
- Contact `#4337` in project `Email Test`.
- Six outbox rows in `event_outbox_events`.