redmine/README.md

# Legacy Redmine Search And Integration Project

This repository exists to make a heavily customized, business-critical Redmine
3.4.4 installation easier to work with without forcing a risky near-term
platform upgrade.

The original production-like copy used for testing is on the LAN at:

```text
http://192.168.50.170/
```

This project started because a large amount of customer/vendor communication is
stored in Redmine through the old RedmineUP CRM and Helpdesk plugins, but the
default UI and documented APIs are not good enough for operational search,
contact lookup, message history review, and future automation.

The goal is not "modernize Redmine" in one jump. The goal is to create safe,
pragmatic tooling around the existing system so the business can search and use
its real communication history.

## Why This Project Exists

The Redmine install has been tightly integrated into day-to-day business
operations. It stores:

- CRM contacts
- helpdesk tickets
- helpdesk email conversations
- issue status and assignment history
- internal journal notes

The old RedmineUP plugin stack is effectively local legacy code now:

- `redmine_contacts` 4.1.2 PRO
- `redmine_contacts_helpdesk` 3.0.9 PRO

Tracked local plugin source lives under:

- [plugins](/home/iadnah/redmine/plugins)

The full `redmine-copy/` tree is an ignored working/reference copy of the legacy
install. Make local plugin changes in `plugins/` first, then deploy or copy them
into the test Redmine instance or `redmine-copy/` as needed.

The Redmine API/MCP wrapper project now lives in:

- [redMCP](/home/iadnah/redmine/redMCP)

That subproject contains the PHP wrapper that composes normal Redmine issue API
responses with local Helpdesk metadata. Its dependencies are managed by Composer;
`redMCP/vendor/`, local env files, and `composer.phar` are ignored.

There is no realistic short-term plan to:

- migrate to a newer RedmineUP package
- upgrade cleanly to Redmine 6
- replace the whole system before extracting value from the data already inside
  it

So this repository treats those plugins as maintainable local forks where needed.

## Core Problem We Are Solving

The main operational need is better search over real customer/vendor
communications.

The crucial discovery was that core Redmine issue data is not sufficient on its
own. In the helpdesk workflow, the issue author can be `Anonymous`, while the
actual customer identity and email metadata live in:

- `helpdesk_tickets`
- `journal_messages`
- `contacts`

That means any serious search/indexing system must treat helpdesk data as
first-class and must not rely only on Redmine issue API fields.

## Architecture Direction

The working architecture is:

1. Redmine records low-risk local outbox events.
2. An external worker reads outbox rows.
3. The worker enriches those rows with read-only MySQL joins against
   helpdesk/contact tables.
4. The worker builds ticket-level and message-level documents.
5. Those documents are indexed externally, eventually with vector search
   support.

The important safety rule is:

```text
External search/indexing failures must not break normal Redmine saves.
```

That is why the event boundary lives in a local DB outbox table rather than in
request-time webhooks, brokers, or direct embedding calls.

## What Has Been Finished

### 1. Contact API Exploration And CLI

The old RedmineUP contacts plugin already exposes useful JSON routes such as:

```text
/projects/customer-service/contacts.json
```

That led to the first standalone helper:

- [redmine_contacts.py](/home/iadnah/redmine/redmine_contacts.py:1)

It currently supports:

- fetching contacts to a local cache
- fuzzy-ish contact search over cached JSON
- light contact updates through the Redmine API
- helpdesk metadata lookup through the local read-only helpdesk API routes

### 2. Event Outbox Plugin

A small plugin was created at:

- [redmine-copy/plugins/redmine_event_outbox](/home/iadnah/redmine/redmine-copy/plugins/redmine_event_outbox)

It records local database events into `event_outbox_events`.

Known-good archive:

- [dist/redmine_event_outbox-0.0.1-known-good-20260421T143957Z.tar.gz](/home/iadnah/redmine/dist/redmine_event_outbox-0.0.1-known-good-20260421T143957Z.tar.gz)
- [manifest](/home/iadnah/redmine/dist/redmine_event_outbox-0.0.1-known-good-20260421T143957Z.MANIFEST.md:1)

Tested event types on the LAN copy:

- `issue.created`
- `issue.updated`
- `journal.created`
- `contact.created`
- `contact.updated`
- `helpdesk_ticket.created`
- `helpdesk_ticket.updated`
- `journal_message.created`

Planned/implemented locally but not yet observed in the controlled LAN
validation workflow:

- `journal_message.updated`

The Helpdesk user workflow itself is now live-smoke-tested, including inbound
Mailpit import, `issueWithHelpdesk()` metadata, default non-email updates, and
explicit customer-visible Helpdesk replies. The repeatable outbox validation
uses that same controlled workflow to verify Helpdesk ticket/message outbox rows
and worker-derived documents without marking rows processed.

### 3. Local Helpdesk Plugin Fork Changes

We made targeted changes to the local fork of `redmine_contacts_helpdesk`:

- added a read-only JSON controller:
  - [helpdesk_search_controller.rb](/home/iadnah/redmine/redmine-copy/plugins/redmine_contacts_helpdesk/app/controllers/helpdesk_search_controller.rb:1)
- added routes for:
  - ticket by issue
  - issues by contact
  - messages by issue
  - contact timeline
- added shorter alias/usage routes to avoid noisy routing errors in logs
- guarded those endpoints with the existing `view_helpdesk_tickets` permission

These changes were deployed to the LAN Redmine copy and route-loaded
successfully.

### 4. Local Fork Hygiene

Before touching the RedmineUP plugin forks, rollback archives were created:

- [redmine_contacts-4.1.2-local-before-helpdesk-search-20260421T215548Z.tar.gz](/home/iadnah/redmine/dist/redmine_contacts-4.1.2-local-before-helpdesk-search-20260421T215548Z.tar.gz)
- [redmine_contacts_helpdesk-3.0.9-local-before-helpdesk-search-20260421T215548Z.tar.gz](/home/iadnah/redmine/dist/redmine_contacts_helpdesk-3.0.9-local-before-helpdesk-search-20260421T215548Z.tar.gz)

Manifests:

- [contacts manifest](/home/iadnah/redmine/dist/redmine_contacts-4.1.2-local-before-helpdesk-search-20260421T215548Z.MANIFEST.md:1)
- [helpdesk manifest](/home/iadnah/redmine/dist/redmine_contacts_helpdesk-3.0.9-local-before-helpdesk-search-20260421T215548Z.MANIFEST.md:1)

Change tracking docs:

- [docs/redmineup_local_fork_changelog.md](/home/iadnah/redmine/docs/redmineup_local_fork_changelog.md:1)
- [redmine-copy/plugins/redmine_contacts_helpdesk/LOCAL_CHANGELOG.md](/home/iadnah/redmine/redmine-copy/plugins/redmine_contacts_helpdesk/LOCAL_CHANGELOG.md:1)

### 5. Read-Only Helpdesk Export/Search CLI

We also built:

- [redmine_helpdesk_search.py](/home/iadnah/redmine/redmine_helpdesk_search.py:1)

Purpose:

- prove that helpdesk/customer/message data can be pulled correctly using
  read-only SQL joins
- produce ticket-level and message-level documents locally
- provide rough CLI search/timeline tooling for debugging and validation

Current state:

- works over SSH to the LAN Redmine host
- reads MySQL credentials from remote `config/database.yml`
- fetches helpdesk ticket and journal message documents into local JSONL cache
- can search/timeline/issues-by-contact over the local cache

Important note:

This script is a diagnostic/export tool, not the final search architecture. We
intentionally stopped short of treating CLI speed optimization as the main goal.

### 6. External Outbox Worker Prototype

The first external worker prototype is:

- [redmine_outbox_worker.py](/home/iadnah/redmine/redmine_outbox_worker.py:1)

It runs outside Redmine and consumes `event_outbox_events` over SSH/MySQL. The
initial output target is deterministic local JSONL rather than a live search
service:

- default output: `/tmp/redmine-outbox/derived_documents.jsonl`

Current behavior:

- dry-runs pending rows without marking them processed
- reports queue counts with `--status`
- claims bounded batches with `locked_at` and `locked_by`
- enriches helpdesk ticket/message/contact-related events with read-only joins
- writes derived event/ticket/message/contact documents as JSONL
- marks rows processed only after a successful local write
- increments `attempts` and writes `last_error` when processing fails
- previews or applies processed-row cleanup with `--purge-processed-days`

The worker processing policy is documented in:

- [docs/outbox_worker_policy.md](/home/iadnah/redmine/docs/outbox_worker_policy.md:1)

### 7. Test Helpdesk Mail Reset

After importing a production database into the LAN test instance, reset all
active projects to use the local Mailpit test mailbox for Helpdesk settings
with:

- [reset_helpdesk_mail_settings.py](/home/iadnah/redmine/reset_helpdesk_mail_settings.py:1)

The complete post-import workflow is documented in:

- [docs/test_instance_post_import.md](/home/iadnah/redmine/docs/test_instance_post_import.md:1)

Use the read-only validator to check the test instance without changing it:

```sh
./validate_test_instance.py
```

Run the Helpdesk/redMCP live smoke test after the post-import checks pass:

```sh
./helpdesk_smoke_test.py
```

That test is documented in:

- [docs/helpdesk_smoke_test.md](/home/iadnah/redmine/docs/helpdesk_smoke_test.md:1)

Run the Helpdesk outbox worker validation when changing outbox hooks, worker
enrichment, or Helpdesk/redMCP behavior:

```sh
./validate_helpdesk_outbox_worker.py
```

That test is documented in:

- [docs/helpdesk_outbox_worker_validation.md](/home/iadnah/redmine/docs/helpdesk_outbox_worker_validation.md:1)

Preview the affected projects and settings:

```sh
./reset_helpdesk_mail_settings.py --dry-run
```

Apply the reset:

```sh
./reset_helpdesk_mail_settings.py
```

Defaults match the current Mailpit test setup:

- incoming POP3: `192.168.1.105:1110`
- outgoing SMTP: `192.168.1.105:1025`
- incoming username/password: `test` / `testpass`
- outgoing SMTP authentication: none
- answer-from pattern: `helpdesk-{identifier}@example.test`

If Mailpit moves, pass the host that Redmine can reach:

```sh
./reset_helpdesk_mail_settings.py --mailpit-host 192.168.50.170
```

### 8. redMCP Helpdesk Semantics

The redMCP wrapper now makes Helpdesk behavior explicit:

- `redMCP/bin/redmcp-server.php` runs as a stdio MCP server.
- `redMCP/bin/redmcp-http-server.php` runs as a bearer-token-protected
  Streamable HTTP MCP server for network client testing, with PID/status/stop
  helpers and optional debug JSONL logging.
- `redMCP/bin/generate-bearer-token.php` generates local MCP bearer tokens.
- `projects()` and `project()` expose Redmine's built-in `/projects.json`
  project list/detail APIs.
- `issues()` and `filterIssues()` expose Redmine's built-in `/issues.json`
  issue filters.
- `search()` and `searchIssues()` expose Redmine's built-in `/search.json`
  text search.
- `issueWithHelpdesk()` composes normal issue data with Helpdesk ticket/message
  metadata.
- `updateIssue()` is safe by default and does not send customer email.
- `updateIssue(..., ['send_helpdesk_email' => true])` and
  `sendHelpdeskIssueResponse()` deliberately use the Helpdesk email path.

The live smoke test verifies both default non-email updates and explicit
customer-visible replies through Mailpit.

## LAN Deployment Progress

The LAN Redmine copy at `192.168.50.170` was inspected and updated via SSH.

Confirmed environment facts:

- SSH user used: `reddev`
- remote Redmine path: `/usr/share/redmine`
- remote Ruby: `2.5.1`
- remote Bundler: `1.17.3`
- remote DB config: MySQL via `config/database.yml`

Remote plugin rollback archives were created on the LAN host under:

```text
/home/reddev/redmine-plugin-backups/
```

The deployed helpdesk search routes were verified with:

- remote `rake routes`
- HTTP requests returning normal login redirects when unauthenticated

## What Is Not Finished Yet

### 1. External Search Index

The worker can now publish bounded JSONL batches and mark successful rows
processed. We have not yet built the actual external index.

Planned direction:

- Qdrant first
- OpenAI embeddings first
- ticket-level docs for "which issue mentioned this?"
- message-level docs for "how did we handle a similar case?"

This should wait until the worker validation pass has documented the derived
document shape that will feed the index.

### 2. Pre-Existing UI/Plugin Bugs

We discovered old plugin issues while working:

- duplicate avatar DOM ids likely causing repeated/wrong thumbnails
- `attachments/contacts_thumbnail` digest warning
- `acts_as_list` Redmine 4 compatibility warning

These are logged in:

- [docs/pre_existing_issues.md](/home/iadnah/redmine/docs/pre_existing_issues.md:1)

They are important context but are not the primary search deliverable.

## Current Repo Files That Matter Most

Project docs:

- [docs/event_outbox_spec.md](/home/iadnah/redmine/docs/event_outbox_spec.md:1)
- [docs/redmineup_local_fork_changelog.md](/home/iadnah/redmine/docs/redmineup_local_fork_changelog.md:1)
- [docs/helpdesk_smoke_test.md](/home/iadnah/redmine/docs/helpdesk_smoke_test.md:1)
- [docs/test_instance_post_import.md](/home/iadnah/redmine/docs/test_instance_post_import.md:1)
- [docs/pre_existing_issues.md](/home/iadnah/redmine/docs/pre_existing_issues.md:1)

Tooling:

- [redmine_contacts.py](/home/iadnah/redmine/redmine_contacts.py:1)
- [redmine_helpdesk_search.py](/home/iadnah/redmine/redmine_helpdesk_search.py:1)

Local plugin work:

- [redmine-copy/plugins/redmine_event_outbox](/home/iadnah/redmine/redmine-copy/plugins/redmine_event_outbox)
- [redmine-copy/plugins/redmine_contacts_helpdesk/app/controllers/helpdesk_search_controller.rb](/home/iadnah/redmine/redmine-copy/plugins/redmine_contacts_helpdesk/app/controllers/helpdesk_search_controller.rb:1)

## Current Recommended Next Steps

If continuing this project, the next best work is:

1. Run the post-import validator and Helpdesk smoke test to establish a clean
   baseline.
2. Generate controlled Helpdesk activity in `fud-helpdesk`.
3. Inspect `event_outbox_events` for the corresponding issue, journal,
   Helpdesk ticket, and journal message rows.
4. Check worker queue state with `redmine_outbox_worker.py --status`.
5. Process a bounded JSONL batch after dry-run output is correct.
6. Choose the first external index target after JSONL publishing is stable.

## Practical Commands

Fetch contacts:

```sh
./redmine_contacts.py fetch
```

Search contacts:

```sh
./redmine_contacts.py search "customer name or phone"
```

Preview a contact update:

```sh
./redmine_contacts.py update 123 --set first_name=Corrected
```

Fetch helpdesk metadata through the Redmine read API:

```sh
./redmine_contacts.py helpdesk-ticket 39858
./redmine_contacts.py helpdesk-issues 4337 --limit 50
./redmine_contacts.py helpdesk-messages 39858 --limit 100
./redmine_contacts.py helpdesk-timeline 4337 --limit 100
```

Fetch read-only helpdesk documents directly over SSH/MySQL:

```sh
./redmine_helpdesk_search.py fetch
```

Preview pending outbox events and their derived documents without marking them
processed:

```sh
./redmine_outbox_worker.py --dry-run --batch-size 10
```

Check queue status:

```sh
./redmine_outbox_worker.py --status
```

Process a bounded outbox batch into JSONL and mark successful rows:

```sh
./redmine_outbox_worker.py --batch-size 20
```

Preview processed-row cleanup:

```sh
./redmine_outbox_worker.py --purge-processed-days 30
```

Search the cached helpdesk documents:

```sh
./redmine_helpdesk_search.py search "inventory not updated" --type message --limit 10
```

Show issues by contact:

```sh
./redmine_helpdesk_search.py issues-by-contact 1299 --limit 20
```

## Summary

This project began as a way to search and use legacy Redmine data without
breaking the system or forcing a full upgrade. The important progress so far is
not cosmetic UI work. It is the architectural clarification that helpdesk data
is the authoritative customer communication layer, plus the first safe event and
export tooling around it.

The repo is now at the point where bounded external worker publishing is proven
against the LAN test instance. The next meaningful leap is choosing and building
the first external search/index target, likely Qdrant with OpenAI embeddings.