redmine/README.md

# Legacy Redmine Search And Integration Project

This repository exists to make a heavily customized, business-critical Redmine
3.4.4 installation easier to work with without forcing a risky near-term
platform upgrade.

The original production-like copy used for testing is on the LAN at:

```text
http://192.168.50.170/
```

This project started because a large amount of customer/vendor communication is
stored in Redmine through the old RedmineUP CRM and Helpdesk plugins, but the
default UI and documented APIs are not good enough for operational search,
contact lookup, message history review, and future automation.

The goal is not "modernize Redmine" in one jump. The goal is to create safe,
pragmatic tooling around the existing system so the business can search and use
its real communication history.

## Why This Project Exists

The Redmine install has been tightly integrated into day-to-day business
operations. It stores:

- CRM contacts
- helpdesk tickets
- helpdesk email conversations
- issue status and assignment history
- internal journal notes

The old RedmineUP plugin stack is effectively local legacy code now:

- `redmine_contacts` 4.1.2 PRO
- `redmine_contacts_helpdesk` 3.0.9 PRO

Tracked local plugin source lives under:

- [plugins](/home/iadnah/redmine/plugins)

The full `redmine-copy/` tree is an ignored working/reference copy of the legacy
install. Make local plugin changes in `plugins/` first, then deploy or copy them
into the test Redmine instance or `redmine-copy/` as needed.

There is no realistic short-term plan to:

- migrate to a newer RedmineUP package
- upgrade cleanly to Redmine 6
- replace the whole system before extracting value from the data already inside
  it

So this repository treats those plugins as maintainable local forks where needed.

## Core Problem We Are Solving

The main operational need is better search over real customer/vendor
communications.

The crucial discovery was that core Redmine issue data is not sufficient on its
own. In the helpdesk workflow, the issue author can be `Anonymous`, while the
actual customer identity and email metadata live in:

- `helpdesk_tickets`
- `journal_messages`
- `contacts`

That means any serious search/indexing system must treat helpdesk data as
first-class and must not rely only on Redmine issue API fields.

## Architecture Direction

The working architecture is:

1. Redmine records low-risk local outbox events.
2. An external worker reads outbox rows.
3. The worker enriches those rows with read-only MySQL joins against
   helpdesk/contact tables.
4. The worker builds ticket-level and message-level documents.
5. Those documents are indexed externally, eventually with vector search
   support.

The important safety rule is:

```text
External search/indexing failures must not break normal Redmine saves.
```

That is why the event boundary lives in a local DB outbox table rather than in
request-time webhooks, brokers, or direct embedding calls.

## What Has Been Finished

### 1. Contact API Exploration And CLI

The old RedmineUP contacts plugin already exposes useful JSON routes such as:

```text
/projects/customer-service/contacts.json
```

That led to the first standalone helper:

- [redmine_contacts.py](/home/iadnah/redmine/redmine_contacts.py:1)

It currently supports:

- fetching contacts to a local cache
- fuzzy-ish contact search over cached JSON
- light contact updates through the Redmine API
- helpdesk metadata lookup through the local read-only helpdesk API routes

### 2. Event Outbox Plugin

A small plugin was created at:

- [redmine-copy/plugins/redmine_event_outbox](/home/iadnah/redmine/redmine-copy/plugins/redmine_event_outbox)

It records local database events into `event_outbox_events`.

Known-good archive:

- [dist/redmine_event_outbox-0.0.1-known-good-20260421T143957Z.tar.gz](/home/iadnah/redmine/dist/redmine_event_outbox-0.0.1-known-good-20260421T143957Z.tar.gz)
- [manifest](/home/iadnah/redmine/dist/redmine_event_outbox-0.0.1-known-good-20260421T143957Z.MANIFEST.md:1)

Tested event types on the LAN copy:

- `issue.created`
- `issue.updated`
- `journal.created`
- `contact.created`
- `contact.updated`

Planned/implemented locally but not fully LAN-validated as a complete workflow:

- `helpdesk_ticket.created`
- `helpdesk_ticket.updated`
- `journal_message.created`
- `journal_message.updated`

### 3. Local Helpdesk Plugin Fork Changes

We made targeted changes to the local fork of `redmine_contacts_helpdesk`:

- added a read-only JSON controller:
  - [helpdesk_search_controller.rb](/home/iadnah/redmine/redmine-copy/plugins/redmine_contacts_helpdesk/app/controllers/helpdesk_search_controller.rb:1)
- added routes for:
  - ticket by issue
  - issues by contact
  - messages by issue
  - contact timeline
- added shorter alias/usage routes to avoid noisy routing errors in logs
- guarded those endpoints with the existing `view_helpdesk_tickets` permission

These changes were deployed to the LAN Redmine copy and route-loaded
successfully.

### 4. Local Fork Hygiene

Before touching the RedmineUP plugin forks, rollback archives were created:

- [redmine_contacts-4.1.2-local-before-helpdesk-search-20260421T215548Z.tar.gz](/home/iadnah/redmine/dist/redmine_contacts-4.1.2-local-before-helpdesk-search-20260421T215548Z.tar.gz)
- [redmine_contacts_helpdesk-3.0.9-local-before-helpdesk-search-20260421T215548Z.tar.gz](/home/iadnah/redmine/dist/redmine_contacts_helpdesk-3.0.9-local-before-helpdesk-search-20260421T215548Z.tar.gz)

Manifests:

- [contacts manifest](/home/iadnah/redmine/dist/redmine_contacts-4.1.2-local-before-helpdesk-search-20260421T215548Z.MANIFEST.md:1)
- [helpdesk manifest](/home/iadnah/redmine/dist/redmine_contacts_helpdesk-3.0.9-local-before-helpdesk-search-20260421T215548Z.MANIFEST.md:1)

Change tracking docs:

- [docs/redmineup_local_fork_changelog.md](/home/iadnah/redmine/docs/redmineup_local_fork_changelog.md:1)
- [redmine-copy/plugins/redmine_contacts_helpdesk/LOCAL_CHANGELOG.md](/home/iadnah/redmine/redmine-copy/plugins/redmine_contacts_helpdesk/LOCAL_CHANGELOG.md:1)

### 5. Read-Only Helpdesk Export/Search CLI

We also built:

- [redmine_helpdesk_search.py](/home/iadnah/redmine/redmine_helpdesk_search.py:1)

Purpose:

- prove that helpdesk/customer/message data can be pulled correctly using
  read-only SQL joins
- produce ticket-level and message-level documents locally
- provide rough CLI search/timeline tooling for debugging and validation

Current state:

- works over SSH to the LAN Redmine host
- reads MySQL credentials from remote `config/database.yml`
- fetches helpdesk ticket and journal message documents into local JSONL cache
- can search/timeline/issues-by-contact over the local cache

Important note:

This script is a diagnostic/export tool, not the final search architecture. We
intentionally stopped short of treating CLI speed optimization as the main goal.

### 6. External Outbox Worker Prototype

The first external worker prototype is:

- [redmine_outbox_worker.py](/home/iadnah/redmine/redmine_outbox_worker.py:1)

It runs outside Redmine and consumes `event_outbox_events` over SSH/MySQL. The
initial output target is deterministic local JSONL rather than a live search
service:

- default output: `.cache/redmine_outbox/derived_documents.jsonl`

Current behavior:

- dry-runs pending rows without marking them processed
- claims bounded batches with `locked_at` and `locked_by`
- enriches helpdesk ticket/message/contact-related events with read-only joins
- writes derived event/ticket/message/contact documents as JSONL
- marks rows processed only after a successful local write
- increments `attempts` and writes `last_error` when processing fails

### 7. Test Helpdesk Mail Reset

After importing a production database into the LAN test instance, reset all
Helpdesk-enabled projects to use the local Mailpit test mailbox with:

- [reset_helpdesk_mail_settings.py](/home/iadnah/redmine/reset_helpdesk_mail_settings.py:1)

Preview the affected projects and settings:

```sh
./reset_helpdesk_mail_settings.py --dry-run
```

Apply the reset:

```sh
./reset_helpdesk_mail_settings.py
```

Defaults match the current Mailpit test setup:

- incoming POP3: `192.168.1.105:1110`
- outgoing SMTP: `192.168.1.105:1025`
- incoming username/password: `test` / `testpass`
- outgoing SMTP authentication: none
- answer-from pattern: `helpdesk-{identifier}@example.test`

If Mailpit moves, pass the host that Redmine can reach:

```sh
./reset_helpdesk_mail_settings.py --mailpit-host 192.168.50.170
```

## LAN Deployment Progress

The LAN Redmine copy at `192.168.50.170` was inspected and updated via SSH.

Confirmed environment facts:

- SSH user used: `reddev`
- remote Redmine path: `/usr/share/redmine`
- remote Ruby: `2.5.1`
- remote Bundler: `1.17.3`
- remote DB config: MySQL via `config/database.yml`

Remote plugin rollback archives were created on the LAN host under:

```text
/home/reddev/redmine-plugin-backups/
```

The deployed helpdesk search routes were verified with:

- remote `rake routes`
- HTTP requests returning normal login redirects when unauthenticated

## What Is Not Finished Yet

### 1. The Real Worker/Indexer

This is the main unfinished piece.

Still needed:

- run and document end-to-end validation of `redmine_outbox_worker.py` against
  the LAN copy
- decide the first real external index target
- map the derived JSONL document shape into that index

### 2. External Search Index

We have not yet built the actual external index.

Planned direction:

- Qdrant first
- OpenAI embeddings first
- ticket-level docs for "which issue mentioned this?"
- message-level docs for "how did we handle a similar case?"

### 3. Full Helpdesk Event Validation

The local code includes helpdesk outbox hooks and read-only helpdesk API
changes, but the complete create/update test matrix for:

- `helpdesk_ticket.*`
- `journal_message.*`

still needs to be run and documented cleanly on the LAN copy.

### 4. Pre-Existing UI/Plugin Bugs

We discovered old plugin issues while working:

- duplicate avatar DOM ids likely causing repeated/wrong thumbnails
- `attachments/contacts_thumbnail` digest warning
- `acts_as_list` Redmine 4 compatibility warning

These are logged in:

- [docs/pre_existing_issues.md](/home/iadnah/redmine/docs/pre_existing_issues.md:1)

They are important context but are not the primary search deliverable.

## Current Repo Files That Matter Most

Project docs:

- [docs/event_outbox_spec.md](/home/iadnah/redmine/docs/event_outbox_spec.md:1)
- [docs/redmineup_local_fork_changelog.md](/home/iadnah/redmine/docs/redmineup_local_fork_changelog.md:1)
- [docs/pre_existing_issues.md](/home/iadnah/redmine/docs/pre_existing_issues.md:1)

Tooling:

- [redmine_contacts.py](/home/iadnah/redmine/redmine_contacts.py:1)
- [redmine_helpdesk_search.py](/home/iadnah/redmine/redmine_helpdesk_search.py:1)

Local plugin work:

- [redmine-copy/plugins/redmine_event_outbox](/home/iadnah/redmine/redmine-copy/plugins/redmine_event_outbox)
- [redmine-copy/plugins/redmine_contacts_helpdesk/app/controllers/helpdesk_search_controller.rb](/home/iadnah/redmine/redmine-copy/plugins/redmine_contacts_helpdesk/app/controllers/helpdesk_search_controller.rb:1)

## Current Recommended Next Steps

If continuing this project, the next best work is:

1. Stop treating the CLI exporter as the end product.
2. Build the external worker that consumes `event_outbox_events`.
3. Start with a simple derived output target:
   - JSONL
   - SQLite
   - or local files
4. Then connect that worker to:
   - Qdrant
   - OpenAI embeddings
5. Validate end-to-end helpdesk search using real historical message data.

## Practical Commands

Fetch contacts:

```sh
./redmine_contacts.py fetch
```

Search contacts:

```sh
./redmine_contacts.py search "customer name or phone"
```

Preview a contact update:

```sh
./redmine_contacts.py update 123 --set first_name=Corrected
```

Fetch helpdesk metadata through the Redmine read API:

```sh
./redmine_contacts.py helpdesk-ticket 39858
./redmine_contacts.py helpdesk-issues 4337 --limit 50
./redmine_contacts.py helpdesk-messages 39858 --limit 100
./redmine_contacts.py helpdesk-timeline 4337 --limit 100
```

Fetch read-only helpdesk documents directly over SSH/MySQL:

```sh
./redmine_helpdesk_search.py fetch
```

Preview pending outbox events and their derived documents without marking them
processed:

```sh
./redmine_outbox_worker.py --dry-run --batch-size 10
```

Process a bounded outbox batch into local JSONL and mark successful rows:

```sh
./redmine_outbox_worker.py --batch-size 20
```

Search the cached helpdesk documents:

```sh
./redmine_helpdesk_search.py search "inventory not updated" --type message --limit 10
```

Show issues by contact:

```sh
./redmine_helpdesk_search.py issues-by-contact 1299 --limit 20
```

## Summary

This project began as a way to search and use legacy Redmine data without
breaking the system or forcing a full upgrade. The important progress so far is
not cosmetic UI work. It is the architectural clarification that helpdesk data
is the authoritative customer communication layer, plus the first safe event and
export tooling around it.

The repo is now at the point where the next meaningful leap is an external
worker/indexer, not more Redmine UI surface.