redmine/AGENT.md

# AGENT.md

This file is a restart brief for future agent sessions on this repository.

## Project Identity

This repository supports a legacy, business-critical Redmine 3.4.4
installation. The real purpose is to make CRM/helpdesk communication data
searchable and automatable without a risky near-term platform migration.

The LAN copy used for testing is:

```text
http://192.168.50.170/
```

This is not a greenfield app. Treat it as a local-fork-and-integration project
around an existing Redmine install and old RedmineUP plugins.

Tracked local plugin source lives in `plugins/`. The full `redmine-copy/` tree
is ignored and should be treated as a working/reference copy, not the source of
truth for local plugin changes.

## Original Motivation

The business stores most external communication inside Redmine through:

- `redmine_contacts`
- `redmine_contacts_helpdesk`

The default UI/API is not good enough for:

- efficient contact search
- light contact maintenance
- helpdesk message search
- customer communication timeline lookup
- future semantic/vector search

The key realization from prior work:

```text
Helpdesk-created issues may have Anonymous issue authors.
The real customer identity lives in helpdesk_tickets, journal_messages, and contacts.
```

Never design the search/index layer as if `issues.author` were enough.

## High-Level Architecture Decision

The intended architecture is:

1. Redmine writes local outbox rows.
2. External worker reads those rows.
3. Worker enriches from read-only MySQL joins.
4. Worker builds ticket/message documents.
5. External index stores/searches those documents.

Safety rule:

```text
External failures must not break Redmine saves.
```

That is why the event boundary is local DB outbox, not request-time webhooks or
embedding calls.

## Redmine And Plugin Baseline

- Redmine version: `3.4.4`
- Old RedmineUP plugin versions in use:
  - `redmine_contacts` 4.1.2 PRO
  - `redmine_contacts_helpdesk` 3.0.9 PRO
- Near-term upgrade to newer Redmine/RedmineUP is not the current goal.
- Treat these plugins as local legacy code when necessary.

## Repository Landmarks

Top-level docs:

- [README.md](README.md)
- [docs/event_outbox_spec.md](docs/event_outbox_spec.md)
- [docs/redmineup_local_fork_changelog.md](docs/redmineup_local_fork_changelog.md)
- [docs/pre_existing_issues.md](docs/pre_existing_issues.md)

Main scripts:

- [redmine_contacts.py](redmine_contacts.py)
- [redmine_helpdesk_search.py](redmine_helpdesk_search.py)
- [redmine_outbox_worker.py](redmine_outbox_worker.py)

Local Redmine copy:

- [redmine-copy](redmine-copy)

Important local plugin paths:

- [redmine-copy/plugins/redmine_event_outbox](redmine-copy/plugins/redmine_event_outbox)
- [redmine-copy/plugins/redmine_contacts_helpdesk](redmine-copy/plugins/redmine_contacts_helpdesk)

## What Has Already Been Done

### Contact CLI

`redmine_contacts.py` exists and supports:

- contact fetch to local cache
- fuzzy-ish contact search
- light contact updates
- helpdesk read API calls:
  - ticket by issue
  - issues by contact
  - messages by issue
  - contact timeline

### Event Outbox Plugin

`redmine_event_outbox` exists and the known-good archive is:

- `dist/redmine_event_outbox-0.0.1-known-good-20260421T143957Z.tar.gz`

Known-good tested LAN events:

- `issue.created`
- `issue.updated`
- `journal.created`
- `contact.created`
- `contact.updated`

Local code also adds optional helpdesk events:

- `helpdesk_ticket.created`
- `helpdesk_ticket.updated`
- `journal_message.created`
- `journal_message.updated`

These helpdesk event paths are implemented locally but should still be treated
as needing fuller end-to-end validation.

### Helpdesk Read API In Local Plugin Fork

The local `redmine_contacts_helpdesk` fork includes:

- `helpdesk_search_controller.rb`
- routes under `/helpdesk_search/*`
- alias/usage routes to avoid noisy routing errors

This was deployed to the LAN copy and route-loaded successfully.

### Helpdesk Export/Search CLI

`redmine_helpdesk_search.py` was created to prove the data model and export
path. It:

- SSHes to the LAN Redmine host
- reads remote MySQL credentials from `config/database.yml`
- runs read-only MySQL queries
- exports ticket/message docs to local JSONL cache
- provides rough local search/timeline/issues-by-contact commands

Important:

```text
This script is a diagnostic/export tool, not the final search architecture.
```

Do not get dragged into treating local CLI search speed as the core mission.

### External Outbox Worker Prototype

`redmine_outbox_worker.py` now exists as the first worker/indexer boundary.
It:

- reads and optionally claims pending `event_outbox_events` over SSH/MySQL
- supports `--dry-run` for non-mutating previews
- uses `locked_at`, `locked_by`, `processed_at`, `attempts`, and `last_error`
- enriches helpdesk ticket/message/contact events with read-only joins
- writes derived JSONL to `/tmp/redmine-outbox/derived_documents.jsonl`
- marks rows processed only after a successful local write

This is still a prototype output target, not the final vector index.

## LAN Host Facts

These were verified previously:

- SSH host/user: `reddev@192.168.50.170`
- SSH key used previously: `/tmp/reddev`
- remote Redmine path: `/usr/share/redmine`
- remote Ruby: `2.5.1`
- remote Bundler: `1.17.3`
- remote DB: MySQL 5.7
- remote plugin rollback archives were stored in:
  - `/home/reddev/redmine-plugin-backups/`

If using those exact credentials again, verify they still exist before relying
on them.

## Remote Plugin Changes Previously Deployed

Changes were copied to the LAN Redmine host for:

- `plugins/redmine_event_outbox/...`
- `plugins/redmine_contacts_helpdesk/...`

Passenger was restarted with:

```text
touch /usr/share/redmine/tmp/restart.txt
```

Routes were verified by remote `rake routes`.

## Fork Hygiene Rules

Before editing RedmineUP plugin code:

1. create a rollback archive in `dist/`
2. record the change in:
   - `docs/redmineup_local_fork_changelog.md`
   - plugin-local changelog if appropriate
3. keep edits scoped
4. prefer read-only APIs/endpoints first
5. validate on the LAN copy before claiming the change is done

Existing rollback archives:

- `dist/redmine_contacts-4.1.2-local-before-helpdesk-search-20260421T215548Z.tar.gz`
- `dist/redmine_contacts_helpdesk-3.0.9-local-before-helpdesk-search-20260421T215548Z.tar.gz`

## Important Pre-Existing Issues

Read:

- [docs/pre_existing_issues.md](docs/pre_existing_issues.md)

Especially remember:

1. duplicate `id="avatar"` is likely a real long-standing UI bug
2. `attachments/contacts_thumbnail` digest warning is probably log noise
3. `acts_as_list` warning is an upgrade-compatibility note, not a current
   blocker

Do not confuse those with the main search deliverable unless they directly block
the work in front of you.

## Current Strategic State

The project is past the "can we get data out?" phase.

We now know:

- contact JSON access exists
- helpdesk/customer identity can be extracted
- outbox events can be recorded safely
- helpdesk read API routes can be added locally
- full ticket/message export is feasible

The most useful next step is:

```text
Build the external worker/indexer pipeline.
```

Not:

- polishing a local CLI into a full search product
- building a big Redmine admin UI
- chasing unrelated legacy plugin cleanup

## Recommended Next Steps For A Future Session

1. Re-read:
   - `README.md`
   - `docs/event_outbox_spec.md`
   - `docs/redmineup_local_fork_changelog.md`
   - `docs/pre_existing_issues.md`
2. Verify local and remote state:
   - plugin files present
   - LAN host reachable
   - remote routes still load
3. Validate `redmine_outbox_worker.py` end to end against the LAN copy
4. Refine read-only joins that enrich:
   - `helpdesk_tickets`
   - `journal_messages`
   - `contacts`
   - `issues`
   - `journals`
5. Confirm deterministic ticket/message/contact JSONL output
6. Only after that, connect Qdrant and embeddings

## Query/Data Model Notes

Authoritative helpdesk context lives in:

- `helpdesk_tickets`
  - `issue_id`
  - `contact_id`
  - `from_address`
  - `to_address`
  - `cc_address`
  - `message_id`
  - `ticket_date`
  - `source`
- `journal_messages`
  - `journal_id`
  - `contact_id`
  - `from_address`
  - `to_address`
  - `cc_address`
  - `bcc_address`
  - `message_id`
  - `is_incoming`
  - `message_date`
- `contacts`
- `issues`
- `journals`

The worker/indexer should prefer direct DB joins as the source of truth for this
layer.

## What To Be Careful About

- Do not assume Redmine core issue API data is enough for helpdesk search.
- Do not treat old RedmineUP code as untouchable vendor code.
- Do not claim a plugin change is validated just because `ruby -c` passes.
- Do not burn time optimizing local CLI search if the real goal is external
  indexing.
- Do not revert unrelated repo changes; the Redmine tree may be dirty.

## If You Need A One-Paragraph Summary

This project exists to safely extract and search real CRM/helpdesk communication
history from a legacy Redmine 3.4.4 install. We already have a contact CLI, a
local event outbox plugin, local helpdesk read API routes, and a rough
helpdesk-export CLI. The next meaningful milestone is an external worker that
consumes outbox rows, enriches from helpdesk/contact MySQL tables, and builds
deterministic ticket/message documents for a future Qdrant/OpenAI-backed search
index.