diff --git a/HANDOFF.md b/HANDOFF.md new file mode 100644 index 0000000..1d618b0 --- /dev/null +++ b/HANDOFF.md @@ -0,0 +1,77 @@ +# redmine handoff + +## Where to find things +- Project rules: `/home/jason/projects/redmine/PROJECT.md` +- Active checklist: `/home/jason/projects/redmine/PLAN.md` +- Main script touched: `/home/jason/projects/redmine/redmine_contacts.py` +- Ticket sync script: `/home/jason/projects/redmine/redmine_tickets.py` +- Tests added: `/home/jason/projects/redmine/tests/test_redmine_contacts.py` +- Ticket sync tests: `/home/jason/projects/redmine/tests/test_redmine_tickets.py` +- Ticket spam cleanup handoff: `/home/jason/projects/redmine/docs/ticket_spam_cleanup_handoff_2026-07-09.md` +- Known-spam fast close plan: `/home/jason/projects/redmine/docs/known_spam_fast_close_plan.md` +- Rebot legacy spam rules: `/home/jason/projects/redmine/docs/rebot_legacy_spam_rules.md` +- Known-spam profile seed: `/home/jason/projects/redmine/profiles/known-spam/customer-service.json` +- Ticket decomposition plan: `/home/jason/projects/redmine/docs/ticket_decomposition_knowledge_ingestion_plan.md` +- Obsidian process note: `/home/jason/lilaBuild/vaults/obsidian-ldr/Processes/Redmine/ticket-decomposition-and-knowledge-ingestion.md` +- Decomposition pilot contact proposals: `/home/jason/projects/redmine/docs/decomposition_pilot_contact_update_proposals_2026-07-09.json` +- KarpathyWiki plugin inspection: `/home/jason/projects/redmine/docs/karpathywiki_plugin_inspection.md` + +## Last completed milestones +1. Applied the shared PROJECT template into this repo with redmine-specific requirements and continuity-doc expectations. +2. Implemented and tested `redmine_contacts.py` upgrades: + - date/tag filtering helpers used by `fetch` + `search` + - `show` command (cache/api source) + - `issue-full` command (issue + helpdesk context) + - `update` tag and custom-field ergonomic options (`--add-tag`, `--remove-tag`, `--set-custom-field`) +3. Added unit tests for new filter/tag/custom-field behavior and verified: + - `python3 -m unittest discover -s tests -p 'test_redmine_contacts.py' -v` + - `python3 -m py_compile redmine_contacts.py` +4. Completed LAN integration validation on `192.168.50.170`: + - validated `show --source api` and `issue-full` + - executed safe `update --apply` on contact `14` with temporary tag/custom-field, then rolled it back to original state + - locked fetch behavior: default `fetch` caches full results; filtered cache requires `--cache-mode filtered` +5. Implemented and verified a dry-run-first `batch-update` command: + - supports date/tag filters + `--max-contacts` + - supports shared update payload options (`--set`, `--add-tag`, `--remove-tag`, `--set-custom-field`) + - defaults to dry-run output and only writes when `--apply` is passed +6. Implemented deterministic batch ordering + UX upgrades: + - added `batch-update --order` with choices: + - `as-is` + - `oldest-updated-first` / `newest-updated-first` + - `oldest-created-first` / `newest-created-first` + - `id-asc` / `id-desc` + - added tests for oldest-updated-first ordering and missing-timestamp behavior + - expanded argparse `--help` descriptions/options across commands + - added Bash completion script at `/home/jason/projects/redmine/redmine_contacts_completion.sh` and verified with `bash -n` +7. Captured Jason's broader contact triage loop vision in: + - `/home/jason/projects/redmine/docs/contact_triage_loop_workflow.md` + - includes workflow explanation, scoring rubric examples, queue-tag architecture, and a mermaid loop diagram + - linked the document from `/home/jason/projects/redmine/README.md` for discoverability +8. Adjusted `fetch` filter behavior to match expected UX: + - `fetch --cache-mode auto` (default) now caches filtered results when any filter flag is supplied + - `--cache-mode full` remains available for canonical full-snapshot caching + - added tests for `resolve_fetch_cache_mode` and verified against LAN command example +9. Implemented and verified ticket-centric sync/search foundation: + - new `/home/jason/projects/redmine/redmine_tickets.py` + - new `/home/jason/projects/redmine/tests/test_redmine_tickets.py` + - `sync` command caches enriched issue payloads using standard `/issues` API with `include=journals,helpdesk` + - per-project cache under `.cache/redmine_tickets_lan/customer-service/*.json` + - `manifest.json` tracks `updated_on`, local cache paths, and per-ticket action idempotency markers + - `search` command filters cached issues without API calls + - `spam-check` action scores cached tickets for promotional/spam signals and can record local idempotency with `--apply` +10. LAN validation completed on `192.168.50.170` / `customer-service`: + - dry-run: `seen=10 fetched=10 skipped_cached=0 written=0` + - cache write: `seen=10 fetched=10 skipped_cached=0 written=10` + - immediate second sync: `seen=10 fetched=0 skipped_cached=10 written=0` + - cached search showed newest open tickets matching UI, including `#41788` / `#41789` “Up to 20% Off Select Fans & Blowers” + - `spam-check` flagged `#41788` and `#41789` as `SPAM` (`score=2`) and left the other 8 as not-spam/low-score + - `spam-check --apply` recorded local manifest action markers; a second `spam-check` checked 0 tickets + - `close-if-spam` dry-run planned closures for `#41788` and `#41789` with payload `{status_id: 5, notes: ...}` + - `close-if-spam` fetched/reviewed sender `contact_id=621` history and found `4/10` recent tickets scoring as spam, so `likely_spammer=True` + - no Redmine ticket close mutation has been applied yet; `close-if-spam --apply` is pending explicit approval + - `close-if-spam --apply` defaults write impersonation to `--impersonate-user rebot`; reads/dry-runs do not impersonate + - tests verified: `python3 -m unittest discover -s tests -p 'test_redmine_tickets.py' -v` (`23` tests) + - compile verified: `python3 -m py_compile redmine_tickets.py tests/test_redmine_tickets.py redmine_contacts.py` + +## Next step and why +Jason identified the Obsidian KarpathyWiki plugin as close to the desired decomposition behavior. Inspection found it has no standalone CLI and depends on Obsidian `App/Vault/TFile` APIs, but its prompts/schema are strong references for a headless mirror. Compatibility rule: the plugin owns `obsidian-ldr/wiki/` (`wiki/entities`, `wiki/concepts`, `wiki/sources`, `wiki/index.md`, `wiki/log.md`), so the Python decomposer must not write there by default; use LDR-owned paths like `Timeline/`, `entities/organizations/`, and `Processes/Redmine/`, with explicit wikilinks to plugin pages when useful. Current implementation: `redmine_tickets.py decompose --issue-id/--input-json --format json|markdown` produces dry-run plans and includes a KarpathyWiki-style structured extraction layer (`summary`, `entities`, `concepts`, `key_points`, `contradictions`, `clean_text`) for the three pilot tickets. Tests pass (`23` tests). Dry-run outputs live under `.cache/decomposition-pilot-20260709/dry-run-output/`. Next: add prompt-driven/LLM JSON extraction for broader ticket types, then compare output quality against hand-written pilot notes before adding vault-write mode. \ No newline at end of file diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..1800ecd --- /dev/null +++ b/PLAN.md @@ -0,0 +1,49 @@ +# redmine contacts + project-standardization plan + +Scope: standardize this repository around the shared `PROJECT.md` workflow and improve `redmine_contacts.py` so contact review/update work can be done in controlled batches with less API/UI friction. + +Target stopping point for this phase: deliver tested CLI upgrades for filtering/show/update ergonomics, plus research documentation for duplicate-contact merge feasibility. + +## Checklist + +### Contact tooling foundation + +- [x] Apply shared project template to `/home/jason/projects/redmine/PROJECT.md` with redmine-specific rules. +- [x] Create continuity docs (`PLAN.md`, `HANDOFF.md`) for ongoing milestone handoff. +- [x] Add tests for new `redmine_contacts.py` contact-filter + tag/custom-field patch behavior. +- [x] Implement date/tag contact filtering utilities used by `fetch` and `search`. +- [x] Add `show` command with cache/api source control. +- [x] Add `issue-full` command that composes full Redmine issue data with helpdesk ticket/messages. +- [x] Add `update` enhancements: + - [x] `--add-tag` + - [x] `--remove-tag` + - [x] `--set-custom-field =` +- [x] Verify new unit tests pass and script compiles. +- [x] Document duplicate-contact merge feasibility (API support vs scripted browser-equivalent merge path). +- [x] Add config-vault project index entry for redmine workstream. +- [x] Decide and implement default fetch cache behavior: `fetch` now caches full fetched contacts by default; `--cache-mode filtered` opt-in stores only filtered subset. +- [x] Add live integration validation against LAN Redmine copy (`192.168.50.170`) for new commands requiring API responses. +- [x] Add dry-run-first `batch-update` command for "N contacts matching filters" workflows. +- [x] Add controlled ordering options for batch runs (`--order` including oldest-updated-first) to support systematic sweep-through campaigns. +- [x] Expand argparse `--help` output with clearer descriptions/option help across commands. +- [x] Add Bash completion script: `/home/jason/projects/redmine/redmine_contacts_completion.sh`. +- [x] Capture the broader contact triage loop workflow in project docs (including scoring concept + diagram). +- [x] Make `fetch` intuitive: with filters and default `--cache-mode auto`, cache filtered results by default (override with `--cache-mode full`). +- [x] Implement Phase 1 contact loop tracer: per-contact evidence pull (3-5 recent tickets/messages) + scoring output in dry-run mode. + +### Ticket-centric spam cleanup substrate + +- [x] Plan ticket-centric sync/search/action architecture in `docs/ticket_sync_tool_plan.md`. +- [x] Implement `redmine_tickets.py` incremental sync using standard Redmine issue API with `include=journals,helpdesk`. +- [x] Implement per-ticket JSON cache plus state-store abstraction (`ManifestTicketStateStore` now, SQLite-capable seam later). +- [x] Implement cached `search` command. +- [x] Implement idempotent `spam-check` action. +- [x] Implement dry-run-first `close-if-spam` action with recent sender/contact lookback. +- [x] Ensure Redmine write calls can impersonate users via `X-Redmine-Switch-User`. +- [x] Default spam cleanup writes to `--impersonate-user rebot` (Harry Botter), while reads/dry-runs do not impersonate. +- [x] Verify LAN dry-run against `192.168.50.170` / `customer-service`: `#41788` and `#41789` flagged and closure-planned. +- [x] Document complete handoff in `docs/ticket_spam_cleanup_handoff_2026-07-09.md`. +- [ ] Confirm LAN issue `status_id=5` is the desired closed status. +- [ ] Run `close-if-spam --apply` on LAN for the currently planned spam tickets only. +- [ ] Re-sync/inspect LAN tickets to verify status closed, activity author `rebot`, and second apply is idempotent. +- [ ] After LAN apply verification, expand cache/sync batch size beyond the first 10 tickets. diff --git a/PROJECT.md b/PROJECT.md new file mode 100644 index 0000000..941f49e --- /dev/null +++ b/PROJECT.md @@ -0,0 +1,171 @@ +--- +tags: + - projects + - redmine + - contacts + - helpdesk +description: Legacy Redmine tooling and local plugin forks for reliable contact/search/helpdesk operations without requiring a risky platform-wide upgrade. +title: redmine +--- +## Project rules +The following rules must be strictly adhered to when working on this project. Deviations from these rules are permitted only when the user explicitly requests or authorizes it. + +### Understand the expectation of the user and what they are doing while you work +The user is expecting you to work continuously for long blocks of time, such as 20+ minutes minimum, during which time they plan to be away from the computer. They only intend to be highly-available at the outset of the work (during planning / clarifications), at the beginning/end of major milestones, or where, prior to starting a particular segment of work, you warn them you expect their involvement will be needed more often for some reason. + +Keep this in mind when planning your work, and try to structure your work to meet the user's expectations about this. + +#### For commands you need the user to run/approve, consider shell scripts +At the outset of work or the beginning of each phase, carefully think through what commands or file access you may need to perform, which is likely to require approval from the user. Also, think about any commands or file edits you will likely require, but would not be able to do yourself for some reason (e.g. you'd be tempted to instruct the user what to do, asking them to confirm results, instead of doing it yourself). + +#### Best practices for handling these requirements + +1. If a command will need to be run often, particularly with only slight variations, consider creating a temporary (project-specific) shell script or similar, which the user can approve once, and then you can use it repeatedly without needing more approvals. +2. If you will need to do something that requires limited root or sudo access, such as restarting a service or making edits to a root-owned file, do not plan on asking the user for approval each time. Instead, suggest the following: + 1. Ask for limited/scoped sudo access which doesn't require a password, and is limited to exactly the commands you'll need to run. + 2. Ask for a temporary change in file permissions so you can edit the file(s) without root access until the work is finished. +3. If working with remote hosts and ssh is available, do not plan on asking the user to shell into the remote host themselves and run commands on your behalf. Instead, suggest the creation and setup of a temporary ssh public key you can use that won't require a password each time you use it. Then, if the user agrees to that, verify you'll be able to connect via ssh without user assistance or approval for what you anticipate you'll need to do. +4. **Browser or similar tool isn't listening/running as expected**: If you need to use a tool like a web browser (like camofox or chrome with CDP) which seems like it should already be running and available to you, but when you try to use it, it doesn't work... try to solve that yourself without immediately running to the user for help. Valid solutions or troubleshooting for this consists of or should include: + 1. Try starting the thing yourself (like camofox or a browser) + 2. Check if an alternative method is available, and use it instead (e.g. you want to use Firecrawl, but the API connection doesn't work. For what you're trying to do, if camofox would work okay and it's available... just use that instead). + +#### Avoid stopping mid-segment to ask for preferred implementation methods +During the course of work, you might run into situations where the project requirements could be implemented or understood more than one way. If that happens, your normal behavior tends to be stopping to ask the user which they would prefer, and presenting pros/cons. If you can, try to keep questions like that isolated to the outset of the work or the beginning/end of major milestones. The user should not be left with the impression you will busy yourself with work for 20+ minutes, so they leave and come back a half hour later, only to find that you stopped after 5 minutes to ask a question like this, especially not if you could have reasonably anticipated the question. + +If an implementation detail like this arises unexpectedly, you should do the following if you can: + +1. Choose the implementation path that is the lowest effort to _undo_ if it's not what the user wanted, and which has the lowest chance of turning into a "rabbit hole". + +### Avoid "rabbit holes" or "wild goose chases" +Do not send the user on a quest of running various commands or actions to help you debug or test something, without first _thoroughly_ researching the likely causes of whatever problem has been encountered. + +As much as possible, do not tell the user what commands to run or what files to edit. Instead, run the commands or edit the files yourself, unless there is a prevailing reason you prefer to have the user run the commands or make the edits. Valid reasons to ask the user to make edits or run commands on your behalf include: + +- File permissions or access controls prevent you from doing it yourself +- The user specified they want to be involved / run things on your behalf +- Taking action yourself would involve opening/reading files that are very likely to contain passwords, API keys, or other secrets. + +#### What is meant by a "rabbit hole" or "wild goose chase" +During the course of the plan, you encounter an error message or maybe a command doesn't work the way you expected. You proceed to ask the user to troubleshoot the problem with you, by directing them to do things like run commands, reboot things, or change files. It becomes a back-and-forth with the user, where instead of progressing on the overall goals of the project, we become stuck on troubleshooting one thing for an extended period of time. + +**Example of what a rabbit hole can look like:** A project being worked on includes 15+ different elements to the plan, one of which is setting up apache2 to properly act as a reverse proxy to serve opencode. You encounter an error that indicates opencode isn't working correctly, and then you start troubleshooting it by asking the user to run various commands, copy-pasting the results to you, making edits to various files, etc. The user isn't in a position to understand or estimate how much of this will actually be needed to get the problem fixed, so they continue working with you in this manner, when they had originally planned to go work on something unrelated for an hour, while you worked on this project. So, they keep going back and forth with you to troubleshoot this opencode problem, hoping with each turn, you'll finally be done asking them to run commands. Before the user realizes it, an hour and half have transpired, and the project really hasn't moved forward at all because we've been fixated on fussing with that one bullet point about opencode, instead of the wider project. + +A chain of events similar to the above is what is meant by a "rabbit hole" or "wild goose chase". + +### Maintain a PLAN.md and HANDOFF.md +The PLAN.md should begin with a short statement that explains the scope of the project we're working on, including _why_ we're working on it, and the expected timeframe it needs to be at a stopping point. The rest of the document should read like a checklist of project goals or milestones. As you work, you should update the PLAN.md to check off things as they are implemented. + +If you realize a particular checklist item in the plan is likely to become a "rabbit hole", is overly broad, or the expected implementation didn't actually resolve it, do the following: + +1. Edit the PLAN.md with one or more sub-points added, breaking the goal/task down into smaller steps or goals that are more achievable. + 1. If a decision is needed which the user isn't available for or has asked you to save for later, add making that decision as a task/goal of it's own. + 2. Create a separate Markdown file containing the context for what decision needs to be made and what the options are, and explicitly reference that file in the sub-point. +2. Move on to the next part of the plan you can actually complete. +3. To indicate statuses with the "[ ]" fields of the markdown, use the following: + 1. "x" : fully completed + 2. "~": partially completed + 3. "?": blocked until a decision is made or issue troubleshooted + +#### HANDOFF.md +Each time you reach a major milestone with the project, update HANDOFF.md so that it reflects enough information that a different agent or worker could pick up where you left off. It should briefly explain: + +1. Where to locate _this_ file. +2. Where to find the PLAN.md +3. The last 2-3 things you finished with the project. +4. The thing you were planning on doing next, and why. + +### Use and update "global" docs and knowledge bases + +Two different shared obsidian vaults exist, each with a different function. You should make liberal usage of them. The user may also have a private obsidian vault, named obsidian-private (but they might not). + +- **obsidian-ldr**: Company knowledge-base about processes and standards, business practices, workplace rules or policies, employees or other contacts, etc. + - Business policies or processes, service/product definitions or promotional materials + - Workplace rules, best practices, SOPs, etc. + - Records of key business decisions, clients, employees, vendors, etc. +- **configs**: Technical docs about systems, infrastructure, tools, or software the company uses or is planning. May include troubleshooting info, coding standards, etc. + - Network configs, info about services available, etc. + - List of projects being worked on and where to find them + - Preferences about tooling, configs, fixtures, etc. + - **When to add things to this vault:** + - You changed a shared configuration on a company network or server + - Setup or installed a new server/service + - Started a new project + - Learned a new fact about company systems or infrastructure not already documented within the vault +- *obsidian-private*: Notes and info that's personal or private to the user. It should/may contain information that's not useful to other people, is private to the user, or moderately sensitive. + - **If it exists, typically used for:** + - User's _personal_ preferences or opinions + - Personal projects other people don't use + - Personalized configs for various tools/systems + - Personal schedules, todo lists, or journals + - **Typical location:** + - `~/obsidian-private` + +#### Leverage the knowledge bases: +When something comes up you're not familiar with, and which plausibly seems to be related to company-specific jargon, methods, people, or similar, check the relevant vault before asking the user to explain what it is, and before you act on assumptions. Then, ask clarifying questions if needed. + +If in the course of doing your work or asking the user questions, you learn things which seem like a future person/agent working on this or another project might also have questions about, _add what you learned to the appropriate vault_. + +##### If unsure what to add or whether to add it, do the following: +Within the most appropriate vault's `INBOX` folder, create a markdown file (or using obsidian's tooling for this) named like YYYY-MM-projectName if it doesn't already exist. Then, append your input or observations to that file. + +### Make liberal, appropriate use of delegation and background processes +You should have the ability to delegate work to sub-agents or even to opencode. Make intelligent use of these capabilities. +#### Here are some suggested uses for that: +- **Information Lookup**: When information might exist in multiple places, considering doing things like delegating one agent to check Obsidian and another to search the internet, then compare/contrast results to make a decision. + +## If unclear where the project should live, make the home as follows +You might be provided instructions within a chat interface, or you might be pointed to a copy of this file somewhere on disk that isn't really a project folder. For instance, you might be pointed to this file within the INBOX section of an obsidian vault like `obsidian-ldr` or `configs`. In a case like this, you should do the following: + +1. First, confirm with the user what this "project" is called, and whether it actually does have a "home" directory it should live within. If they say that it does, just use that. Otherwise, proceed with this list. +2. If the user is available, suggest 3-5 potential codenames for this project. Ask them to choose a name. Try to provide: + 1. 1-2 functional, boring, but descriptive names + 2. 1-3 creative, mythical, or narrative names +3. If the user isn't available, just pick a descriptive name for the project. +4. **Create and setup project directory:** + 1. Convert the selected name to kebab-case or camel-case. + 2. Within the user's `~/projects/` directory, create a directory for the project using the converted project name. + 3. Within that directory: + 1. Place a copy of this file inside of that project, and name it PROJECT.md. + 2. Create a README.md within that directory, and add a succinct description to the top which summarizes the scope/purpose of the project. + 3. Initialize the project directory as a git repository. + 4. **If `goto` is available, add the project to it** + 1. Check if the shell command `goto` is available. If it is, add the project to goto. +5. **Add it to the project index in `configs`**: + 1. Within the `configs` Obsidian vault, check inside of `projects/` for a markdown article about this project. If one does not exist, create it. + 2. Within that file, include a short section that explains where the actual project directory is, and a basic gist (very short) explaining what the project is (scope, problem it solves, etc). + +### Troubleshooting + +#### Obsidian vault not found or not working +If the vault isn't found, try the following: +- If `goto` is available, run `goto list` to see if it locates it +- Check these paths: + - `~/projects/` : for instance, `~/projects/obsidian-ldr` or `~/projects/configs` + +If the vault isn't "working", consider how you're trying to access it: + +- If you're using an MCP tool, that tool might be connected to the wrong vault, or it my rely on the obsidian GUI to be running. +- Check to see if one of the following methods will work, and if so, use one of these. Raise the issue about the vault not working correctly _after_ you reach the next project milestone. + - obsidian cli + - Usage of `fd`, `rg`, and similar shell commands + +*** +# Actual project instructions + +The above information and rules govern how you should approach the following project. Below this section are the redmine-specific requirements. + +## redmine project-specific requirements + +- Project name is **redmine**. +- Project home is **`/home/jason/projects/redmine/`** (already exists; do not relocate). +- Primary objective: safe operational tooling around legacy Redmine CRM/Helpdesk data so support, triage, and automation workflows can move faster without a platform rewrite. +- Prioritize `redmine_contacts.py` ergonomics for data-use outcomes: + - filtering contacts by date/tag combinations for batch cleanup workflows, + - full issue retrieval with helpdesk context, + - precise, non-destructive contact updates (tags/custom fields). +- Maintain continuity docs in-repo: + - `PLAN.md` (active checklist using `[x]`, `[~]`, `[?]`) + - `HANDOFF.md` (last milestones + next step) +- Keep plugin source-of-truth in tracked `plugins/` paths. Do not treat ignored `redmine-copy/` as canonical source. +- Prefer dry-run/read-only validation where possible before mutating live-like test data. +- When documenting research or API feasibility, store findings in `docs/` with concrete route, method, auth, and risk notes. diff --git a/README.md b/README.md index 625ac9b..cb6d0fe 100644 --- a/README.md +++ b/README.md @@ -395,12 +395,26 @@ Project docs: - [docs/helpdesk_smoke_test.md](docs/helpdesk_smoke_test.md) - [docs/test_instance_post_import.md](docs/test_instance_post_import.md) - [docs/pre_existing_issues.md](docs/pre_existing_issues.md) +- [docs/contact_triage_loop_workflow.md](docs/contact_triage_loop_workflow.md) +- [docs/contact_triage_loop_raw_notes_2026-07-04.md](docs/contact_triage_loop_raw_notes_2026-07-04.md) Tooling: - [redmine_contacts.py](redmine_contacts.py) +- [redmine_tickets.py](redmine_tickets.py) — local power-user client (sync, property search, view, walk) - [redmine_helpdesk_search.py](redmine_helpdesk_search.py) +Power-user client (sync + CLI): + +- [docs/cli_power_user_client.md](docs/cli_power_user_client.md) — **usage for humans/agents** +- [docs/raw-notes/features-for-redmine_tickets-sync.md](docs/raw-notes/features-for-redmine_tickets-sync.md) — roadmap index +- [docs/2026-07-14_localsearch_before_sanity.md](docs/2026-07-14_localsearch_before_sanity.md) — old thin-search baseline + +Ticket-centric spam cleanup docs: + +- [docs/ticket_sync_tool_plan.md](docs/ticket_sync_tool_plan.md) +- [docs/ticket_spam_cleanup_handoff_2026-07-09.md](docs/ticket_spam_cleanup_handoff_2026-07-09.md) + Local plugin work: - [redmine-copy/plugins/redmine_event_outbox](redmine-copy/plugins/redmine_event_outbox) @@ -408,7 +422,39 @@ Local plugin work: ## Current Recommended Next Steps -If continuing this project, the next best work is: +### Current priority: power-user local client + automations + +Durable CS cache + CLI for search/view/walk (see [docs/cli_power_user_client.md](docs/cli_power_user_client.md)). + +```bash +cd /home/jason/projects/redmine +CACHE=/home/jason/projects/redmine/cache/redmine_tickets +./redmine_tickets.py --project customer-service --cache-dir "$CACHE" \ + search --where 'assigned_to IS NULL' --max-tickets 10 +./redmine_tickets.py --project customer-service --cache-dir "$CACHE" \ + view --print +``` + +Cron: `cron-scripts/sync-live-customer-service.sh` (still `30d` window — tighten later). + +Spam cleanup remains available on LAN before prod apply — see handoff doc below. + +### Ticket-centric spam cleanup (secondary) + +- [redmine_tickets.py](redmine_tickets.py) +- [docs/ticket_sync_tool_plan.md](docs/ticket_sync_tool_plan.md) +- [docs/ticket_spam_cleanup_handoff_2026-07-09.md](docs/ticket_spam_cleanup_handoff_2026-07-09.md) + +Next safe sequence for spam apply: + +1. Confirm LAN issue `status_id=5` is the intended closed status. +2. Run `redmine_tickets.py close-if-spam --apply` on the LAN copy only, using default `--impersonate-user rebot`. +3. Re-sync/inspect affected LAN tickets. +4. Verify status closed, activity author `rebot` / Harry Botter, and idempotency on a second apply. + +### Other Redmine indexing / outbox workstream + +If continuing the older outbox/search-index project, the next best work is: 1. Run the post-import validator and Helpdesk smoke test to establish a clean baseline. @@ -427,10 +473,12 @@ Fetch contacts: ./redmine_contacts.py fetch ``` -Search contacts: +Search contacts. Short examples: `./redmine_contacts.py search --help`. What the CLI can do: [docs/redmine_contacts_search.md](docs/redmine_contacts_search.md). Situations it cannot do yet: [docs/redmine_contacts_search_proposals.md](docs/redmine_contacts_search_proposals.md). ```sh -./redmine_contacts.py search "customer name or phone" +./redmine_contacts.py --base-url https://rm.ldrprep.com search "Sahar Daod" +./redmine_contacts.py --base-url https://rm.ldrprep.com \ + search promotional-spam --with-tag promotional-spam --refresh ``` Preview a contact update: diff --git a/Rebuild or something.md b/Rebuild or something.md new file mode 100644 index 0000000..74fe2ce --- /dev/null +++ b/Rebuild or something.md @@ -0,0 +1 @@ +I had a _really powerful_ program that had been `redmine_tickets.py`, `bin/rtl`, and the other tools it connected with. The curses-based TUI was amazingly fast and powerful. Then, for \ No newline at end of file diff --git a/bin/checkin_truth.md b/bin/checkin_truth.md new file mode 100644 index 0000000..bb5a849 --- /dev/null +++ b/bin/checkin_truth.md @@ -0,0 +1,16 @@ +reliable "change sku" API endpoint that corrects the SKU assigned to one or more individual_product entries owned by the same user. + +## What usually causes the problem: + +**Not confident to choose/verify SKU**: While receiving and checking in packages from someone like Moses, there we receive one or more items we can't confidently match to one of his products. Then, either: + +a) We choose an "AUTO-" sku or "add unexpected item", and type a description of what we have. +b) We make a guess or a mistake, and and it turns out to be incorrect. + +On the employee-facing side of the portal, we have a tool for fixing this from the [[Refurbishment Panel]] and also [[Item Manager]]. Customers like Moses who want to correct the SKU chosen for an item seem to lack an intended, easy-to-use way to accomplish that. + +## Benefits and Tradeoffss about AUTO SKUs and our receiving procedures + +### Our intention should be maximal truth within a predictable timespan and price, pertaining to all handling of inventory. That includes confirming and documenting we received somethinmg addressed to the customer, even if it wasn't expected by anyone. + +AUTO skus are usually left in a "non-final condition" like [[Customer Service Needed]] until somebody does something about it. In practice, we have a persistent rumor that setting "CSN" notifies customers, most customers decide they don't want the items, and Danny is the person who deals with that. \ No newline at end of file diff --git a/bin/cli-sync.sh b/bin/cli-sync.sh new file mode 100755 index 0000000..85013a5 --- /dev/null +++ b/bin/cli-sync.sh @@ -0,0 +1,17 @@ +#!/bin/bash + +cd /home/jason/projects/redmine/ + +BURL="https://rm.ldrprep.com/" +TCACHE="./cache/redmine_tickets" + +./redmine_contacts.py --base-url $BURL --project customer-service --cache ./cache/redmine_contacts/ fetch + +./redmine_tickets.py --base-url $BURL --project customer-service --cache-dir "$TCACHE" sync \ + --remote-status-id '*' --remote-updated-since 48h --run-log --run-log-keep 20 + +./redmine_tickets.py --base-url $BURL --project quality-tracker --cache-dir "$TCACHE" sync \ + --remote-status-id '*' --remote-updated-since 48h --run-log --run-log-keep 20 + + ./redmine_tickets.py --base-url $BURL --project meetings --cache-dir "$TCACHE" sync \ + --remote-status-id '*' --remote-updated-since 48h --run-log --run-log-keep 20 \ No newline at end of file diff --git a/bin/qaa b/bin/qaa new file mode 100755 index 0000000..21a6472 --- /dev/null +++ b/bin/qaa @@ -0,0 +1,5 @@ +#!/usr/bin/env bash +# Short alias example (qaa) → same as rtl (prod CS). Rename/copy for more namespaces later. +set -euo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +exec "$ROOT/bin/rtl" "$@" diff --git a/bin/rtd b/bin/rtd new file mode 100755 index 0000000..06178f7 --- /dev/null +++ b/bin/rtd @@ -0,0 +1,6 @@ +#!/usr/bin/env bash +# Muscle-memory wrapper: LAN/dev customer-service cache + base URL. +# Usage: rtd note --issue-id 39305 --body 'test' --apply +set -euo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +exec "$ROOT/redmine_tickets.py" --config "$ROOT/configs/dev.json" "$@" diff --git a/bin/rtl b/bin/rtl new file mode 100755 index 0000000..2e2eaf1 --- /dev/null +++ b/bin/rtl @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +# Muscle-memory wrapper: prod customer-service cache + base URL. +# Usage: rtl search --where 'assigned_to IS NULL' --max-tickets 10 +set -euo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +# rtl is the production Redmine wrapper. Keep an explicitly supplied mode, but +# never silently default its NMI briefs to the dev profile. +export NMI_MODE="${NMI_MODE:-production}" +exec "$ROOT/redmine_tickets.py" --config "$ROOT/configs/live.json" "$@" diff --git a/contact_enrichment.py b/contact_enrichment.py new file mode 100644 index 0000000..5af4e53 --- /dev/null +++ b/contact_enrichment.py @@ -0,0 +1,441 @@ +#!/usr/bin/env python3 +"""Contact enrichment: map Redmine CRM contacts to a target CRM schema. + +This module is intentionally decoupled from ``redmine_contacts.py`` so the sync +module can import it without pulling in CLI / cache logic. All enrichment is +deterministic and side-effect free; external lookups are accepted via a +configurable provider so callers can inject rate-limited clients. +""" + +from __future__ import annotations + +import json +import re +from dataclasses import dataclass, field +from datetime import datetime, timezone +from typing import Any, Callable + + +EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}") +PHONE_RE = re.compile(r"[\d\s\-().+]{7,}") +URL_RE = re.compile(r"https?://[^\s<>\"']+|www\.[^\s<>\"']+") + + +def _utc_now() -> datetime: + return datetime.now(timezone.utc) + + +@dataclass +class EnrichmentResult: + """Output of enriching a single Redmine contact.""" + + contact: dict[str, Any] + enriched: dict[str, Any] + changed_fields: list[str] + warnings: list[str] = field(default_factory=list) + skipped: bool = False + reason: str = "" + + +@dataclass +class EnrichmentConfig: + """Configuration for the enrichment pipeline.""" + + source: str = "redmine" + confidence_threshold: float = 0.7 + default_country_code: str = "US" + tags_from_redmine: list[str] = field(default_factory=lambda: ["from-redmine"]) + external_lookup: Callable[[str], dict[str, Any]] | None = None + domain_to_company: dict[str, str] = field(default_factory=dict) + + +class EnrichmentError(RuntimeError): + pass + + +class RateLimitError(EnrichmentError): + """Raised when an external lookup exhausts retries or hits a hard limit.""" + + pass + + +def enrich_contact( + redmine_contact: dict[str, Any], + config: EnrichmentConfig | None = None, +) -> EnrichmentResult: + """Map a Redmine contact to the target CRM schema and fill missing fields.""" + config = config or EnrichmentConfig() + contact_id = redmine_contact.get("id") + if contact_id is None: + return EnrichmentResult( + contact=redmine_contact, + enriched={}, + changed_fields=[], + skipped=True, + reason="Missing Redmine contact id", + ) + + warnings: list[str] = [] + + emails = _extract_emails(redmine_contact, warnings) + phones = _extract_phones(redmine_contact, config.default_country_code, warnings) + company = _enrich_company(redmine_contact, emails, config, warnings) + name_parts = _name_parts(redmine_contact) + address = _extract_address(redmine_contact) + notes = _build_notes(redmine_contact) + tags = _build_tags(redmine_contact, config) + + enriched: dict[str, Any] = { + "crm_id": None, + "redmine_contact_id": contact_id, + "full_name": name_parts["full_name"], + "first_name": name_parts["first_name"], + "middle_name": name_parts.get("middle_name"), + "last_name": name_parts["last_name"], + "company": company, + "title": _clean_str(redmine_contact.get("job_title")), + "emails": emails, + "phones": phones, + "website": _clean_str(redmine_contact.get("website")), + "birthday": _format_birthday(redmine_contact.get("birthday")), + "address": address, + "notes": notes, + "tags": tags, + "source": config.source, + "synced_at": _utc_now().isoformat(), + "enriched_at": _utc_now().isoformat(), + "last_redmine_update": _format_timestamp(redmine_contact.get("updated_at", redmine_contact.get("updated_on"))), + } + + changed_fields = [key for key, value in enriched.items() if value not in (None, "", [], {})] + + if not emails and not company: + warnings.append("No email or company; record may be unmatchable in CRM") + + return EnrichmentResult( + contact=redmine_contact, + enriched=enriched, + changed_fields=changed_fields, + warnings=warnings, + ) + + +def _clean_str(value: Any) -> str | None: + if value is None: + return None + cleaned = str(value).strip() + return cleaned or None + + +def _extract_emails(redmine_contact: dict[str, Any], warnings: list[str]) -> list[dict[str, Any]]: + emails: list[dict[str, Any]] = [] + primary = _clean_str(redmine_contact.get("email")) + if primary: + emails.append({"value": primary.lower(), "type": "work"}) + + # Search background and custom fields for additional emails when primary missing. + if not emails: + haystack = _contact_text_haystack(redmine_contact) + for match in EMAIL_RE.findall(haystack): + candidate = match.lower() + if not any(existing["value"] == candidate for existing in emails): + emails.append({"value": candidate, "type": "work"}) + if not emails: + warnings.append("No email found") + return emails + + +def _extract_phones( + redmine_contact: dict[str, Any], + default_country_code: str, + warnings: list[str], +) -> list[dict[str, Any]]: + phones: list[dict[str, Any]] = [] + raw_phone = _clean_str(redmine_contact.get("phone")) + if raw_phone: + normalized = _normalize_phone(raw_phone, default_country_code) + if normalized: + phones.append({"value": normalized, "type": _classify_phone_type(raw_phone)}) + + # Do not mine background for phones unless the explicit field is empty. + if not phones: + haystack = _contact_text_haystack(redmine_contact) + for match in PHONE_RE.findall(haystack): + normalized = _normalize_phone(match, default_country_code) + if normalized and not any(existing["value"] == normalized for existing in phones): + phones.append({"value": normalized, "type": _classify_phone_type(match)}) + if len(phones) >= 3: + break + if not phones: + warnings.append("No phone found") + return phones + + +def _normalize_phone(raw: str, default_country_code: str) -> str | None: + digits = re.sub(r"\D", "", raw) + if len(digits) < 7: + return None + if len(digits) == 10 and default_country_code == "US": + digits = "1" + digits + if digits.startswith("1") and len(digits) == 11: + return f"+{digits}" + return f"+{digits}" + + +def _classify_phone_type(raw: str) -> str: + lowered = raw.lower() + if "mobile" in lowered or "cell" in lowered or "m:" in lowered: + return "mobile" + if "home" in lowered or "h:" in lowered: + return "home" + return "work" + + +def _enrich_company( + redmine_contact: dict[str, Any], + emails: list[dict[str, Any]], + config: EnrichmentConfig, + warnings: list[str], +) -> str | None: + company = _clean_str(redmine_contact.get("company")) + if company: + return company + + if config.external_lookup is not None: + try: + lookup = config.external_lookup(str(redmine_contact.get("id"))) + company = _clean_str(lookup.get("company")) + if company: + return company + except RateLimitError: + warnings.append("External company lookup hit rate limit") + except Exception as exc: # noqa: BLE001 + warnings.append(f"External company lookup failed: {exc}") + + if emails: + domain = emails[0]["value"].split("@", 1)[-1] + company = config.domain_to_company.get(domain) + if company: + return company + company = _company_from_domain(domain) + if company: + return company + + warnings.append("Company could not be enriched") + return None + + +def _company_from_domain(domain: str) -> str | None: + # Simple deterministic heuristic; keeps component dependency-free. + if not domain or "." not in domain: + return None + name = domain.rsplit(".", 1)[0] + name = re.sub(r"[^a-zA-Z0-9]+", " ", name).strip() + if not name: + return None + return name.capitalize() + " Inc" + + +def _name_parts(redmine_contact: dict[str, Any]) -> dict[str, Any]: + first = _clean_str(redmine_contact.get("first_name")) + middle = _clean_str(redmine_contact.get("middle_name")) + last = _clean_str(redmine_contact.get("last_name")) + + if redmine_contact.get("is_company"): + company = _clean_str(redmine_contact.get("company")) or first or "" + return {"full_name": company, "first_name": company, "middle_name": None, "last_name": None} + + parts = [part for part in (first, middle, last) if part] + full_name = " ".join(parts) + + # If no structured name, try to infer from display-style strings. + if not full_name: + full_name = _clean_str(redmine_contact.get("name")) or "" + if full_name: + split = full_name.split() + first = split[0] if split else None + last = " ".join(split[1:]) if len(split) > 1 else None + + return {"full_name": full_name or None, "first_name": first, "middle_name": middle, "last_name": last} + + +def _extract_address(redmine_contact: dict[str, Any]) -> dict[str, Any]: + attrs = redmine_contact.get("address_attributes") or {} + if not attrs: + return {} + return { + "street": _clean_str(attrs.get("street")), + "city": _clean_str(attrs.get("city")), + "region": _clean_str(attrs.get("state")), + "country": _clean_str(attrs.get("country")), + "postal_code": _clean_str(attrs.get("zip")), + } + + +def _build_notes(redmine_contact: dict[str, Any]) -> str: + parts: list[str] = [] + background = _clean_str(redmine_contact.get("background")) + if background: + parts.append(background) + + custom_fields = redmine_contact.get("custom_fields") or [] + for field in custom_fields: + name = field.get("name") + value = field.get("value") + if name and value: + parts.append(f"{name}: {value}") + + contact_id = redmine_contact.get("id") + if contact_id is not None: + parts.append(f"Redmine contact: {contact_id}") + + return "\n".join(parts) + + +def _build_tags(redmine_contact: dict[str, Any], config: EnrichmentConfig) -> list[str]: + raw_tags = redmine_contact.get("tag_list") or [] + if isinstance(raw_tags, str): + tags = {part.strip().lower() for part in raw_tags.split(",") if part.strip()} + elif isinstance(raw_tags, list): + tags = {str(tag).strip().lower() for tag in raw_tags if str(tag).strip()} + else: + tags = set() + + for tag in config.tags_from_redmine: + tags.add(tag.lower()) + + project_id = redmine_contact.get("project_id") + if project_id is not None: + tags.add(f"project-{project_id}") + + return sorted(tags) + + +def _contact_text_haystack(redmine_contact: dict[str, Any]) -> str: + pieces: list[str] = [] + background = _clean_str(redmine_contact.get("background")) + if background: + pieces.append(background) + for field in redmine_contact.get("custom_fields") or []: + value = field.get("value") + if value: + pieces.append(str(value)) + return "\n".join(pieces) + + +def _format_birthday(value: Any) -> str | None: + cleaned = _clean_str(value) + if not cleaned: + return None + for fmt in ("%Y-%m-%d", "%Y/%m/%d", "%m/%d/%Y", "%d/%m/%Y"): + try: + return datetime.strptime(cleaned, fmt).date().isoformat() + except ValueError: + continue + return None + + +def _format_timestamp(value: Any) -> str | None: + if value is None: + return None + raw = str(value).strip() + if not raw: + return None + normalized = raw.replace("Z", "+00:00") + try: + dt = datetime.fromisoformat(normalized) + except ValueError: + return None + if dt.tzinfo is None: + dt = dt.replace(tzinfo=timezone.utc) + return dt.astimezone(timezone.utc).isoformat() + + +def build_sync_payload( + redmine_contacts_iter: list[dict[str, Any]], + config: EnrichmentConfig | None = None, +) -> dict[str, Any]: + """Enrich a batch and return a summary suitable for a sync module.""" + config = config or EnrichmentConfig() + results = [enrich_contact(contact, config) for contact in redmine_contacts_iter] + return { + "enriched_at": _utc_now().isoformat(), + "count": len([r for r in results if not r.skipped]), + "records": [result.enriched for result in results if not result.skipped], + "skipped": [ + {"redmine_contact_id": result.contact.get("id"), "reason": result.reason} + for result in results + if result.skipped + ], + "warnings": [ + {"redmine_contact_id": result.contact.get("id"), "warning": warning} + for result in results + for warning in result.warnings + ], + } + + +def with_rate_limit( + func: Callable[..., Any], + *, + max_calls: int = 10, + per_seconds: float = 1.0, + max_retries: int = 5, + backoff_base: float = 1.0, + backoff_cap: float = 30.0, +) -> Callable[..., Any]: + """Wrap a callable with a simple token-bucket rate limit and retry policy. + + The wrapper is stateful; create one per external provider session. It sleeps + using ``time.sleep`` so it is safe for sync scripts but should not be used + inside async loops. + """ + import time + + tokens = float(max_calls) + last_refill = time.monotonic() + lock_retried = 0 + + def _refill() -> None: + nonlocal tokens, last_refill + now = time.monotonic() + elapsed = now - last_refill + tokens = min(float(max_calls), tokens + elapsed * (max_calls / per_seconds)) + last_refill = now + + def wrapper(*args: Any, **kwargs: Any) -> Any: + nonlocal tokens, lock_retried + for attempt in range(max_retries + 1): + _refill() + if tokens < 1: + sleep_time = (1 - tokens) * (per_seconds / max_calls) + time.sleep(max(0.0, sleep_time)) + _refill() + tokens -= 1 + try: + return func(*args, **kwargs) + except RateLimitError: + raise + except Exception as exc: # noqa: BLE001 + if attempt == max_retries: + raise + wait = min(backoff_cap, backoff_base * (2 ** attempt)) + time.sleep(wait) + raise EnrichmentError("Rate-limited wrapper exhausted retries") + + return wrapper + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser(description="Enrich Redmine CRM contacts for sync.") + parser.add_argument("contacts", type=json.loads, help="JSON-encoded Redmine contact or list of contacts.") + parser.add_argument("--source", default="redmine") + parser.add_argument("--confidence-threshold", type=float, default=0.7) + args = parser.parse_args() + + raw = args.contacts + contacts = raw if isinstance(raw, list) else [raw] + config = EnrichmentConfig(source=args.source, confidence_threshold=args.confidence_threshold) + print(json.dumps(build_sync_payload(contacts, config), indent=2, sort_keys=True)) diff --git a/cron-scripts/sync-dev-customer-service.sh b/cron-scripts/sync-dev-customer-service.sh new file mode 100755 index 0000000..b8d9d2b --- /dev/null +++ b/cron-scripts/sync-dev-customer-service.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# Dev/LAN CS ticket sync → cache-dev, then rematerialize queues. +set -euo pipefail +cd /home/jason/projects/redmine +export REDMINE_API_KEY="${REDMINE_API_KEY:?REDMINE_API_KEY required}" +CACHE=/home/jason/projects/redmine/cache-dev/redmine_tickets +./redmine_tickets.py \ + --base-url http://192.168.50.170 \ + --project customer-service \ + --cache-dir "$CACHE" \ + sync \ + --remote-status-id '*' \ + --remote-updated-since 6h \ + --run-log \ + --run-log-keep 20 \ + --quiet + +./redmine_tickets.py \ + --base-url http://192.168.50.170 \ + --project customer-service \ + --cache-dir "$CACHE" \ + materialize-queues \ + --config queues.example.json \ + >/dev/null diff --git a/cron-scripts/sync-live-contacts.sh b/cron-scripts/sync-live-contacts.sh new file mode 100755 index 0000000..7b19139 --- /dev/null +++ b/cron-scripts/sync-live-contacts.sh @@ -0,0 +1,6 @@ +#!/usr/bin/env bash +# Optional slower contacts mirror refresh (prod cache). Run less often than ticket sync. +set -euo pipefail +cd /home/jason/projects/redmine +#export REDMINE_API_KEY="${REDMINE_API_KEY:?REDMINE_API_KEY required}" +./bin/rtl contacts-sync --quiet diff --git a/cron-scripts/sync-live-customer-service.sh b/cron-scripts/sync-live-customer-service.sh new file mode 100755 index 0000000..1d6bdb3 --- /dev/null +++ b/cron-scripts/sync-live-customer-service.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +# Prod CS ticket sync → durable cache, then rematerialize local queues. +# Window: 6h after warm cache (bump to 30d only for catch-up). +set -euo pipefail +cd /home/jason/projects/redmine +#CACHE=/home/jason/projects/redmine/cache/redmine_tickets +./bin/rtl \ + sync \ + --remote-status-id '*' \ + --remote-updated-since 1d \ + --run-log \ + --run-log-keep 20 \ + --quiet + +# Refresh symlink queues for external agents/scripts (local-only). +./bin/rtl \ + materialize-queues \ +# --config queues.example.json \ + >/dev/null diff --git a/dispatch-dry-run-2026-09-19.md b/dispatch-dry-run-2026-09-19.md new file mode 100644 index 0000000..6cf61f8 --- /dev/null +++ b/dispatch-dry-run-2026-09-19.md @@ -0,0 +1,142 @@ +# Dispatch dry-run report: customer-service New/unassigned (updated since 2026-09-18) + +Generated: 2026-09-19 19:12 EDT | Tickets examined: 17 | Cache: /home/jason/projects/redmine/cache/redmine_tickets + +## Summary + +- Obvious spam/promotional: 6 +- Automated operational notifications (Amazon/AM Shipping receipts, invoices): 9 +- Inbound business inquiries needing review: 0 +- Tickets with extractable entities (tracking/order/item): 9 +- High attention score (>=40): 17 + +## Per-ticket details + +### #43690 — Product Session: Moses / Esire — 90-day ticket themes (from #43412) +- Status: New | Category: Product Session | Assignee: — | Updated: 2026-09-19T11:52 +- Spam: `automated-operational` score=0 — automated operational notification +- Triage score: **84.0** (status:new; unassigned+open; kw:complaint; kw:missing; kw:wrong; has-entities) +- Entities: item_or_sku:corrections(medium); item_or_sku:still(medium); item_or_sku:verification(medium); item_or_sku:match(medium); item_or_sku:backlog(medium); item_or_sku:reference(medium); item_or_sku:sampling(medium); item_or_sku:stills(medium); item_no:200230889(medium); item_no:200222683(medium); item_no:200226501(medium); item_no:200243595(medium); account_ref:structure(medium); account_ref:request(medium) +- Suggested action: Auto-categorize/route to Receipts/Bill Needs Paid and resolve if no action needed.; High attention: needs Dispatch containment review now.; Run `entities nmi` to attach private NMI brief note. + +### #43711 — Don’t miss out on your work perks +- Status: New | Category: — | Assignee: — | Updated: 2026-09-19T10:28 +- Spam: `spam-promotional` score=3 — sender has promotional/automated hint; body/journal contains promotional language; ticket text contains promotional sender hint +- Triage score: **57.0** (stale>8h; status:new; unassigned+open) +- Entities: none detected +- Suggested action: Close as spam (spam-check → close-if-spam).; High attention: needs Dispatch containment review now. + +### #43694 — On Hand Now: GOLDENROD 2-Packs at $16 +- Status: New | Category: — | Assignee: — | Updated: 2026-09-18T20:55 +- Spam: `spam-promotional` score=2 — body/journal contains promotional language; ticket text contains promotional sender hint +- Triage score: **57.0** (stale>22h; status:new; unassigned+open) +- Entities: none detected +- Suggested action: Close as spam (spam-check → close-if-spam).; High attention: needs Dispatch containment review now. + +### #43693 — Nature2 3-Packs: $40 + FREE Poly Prep +- Status: New | Category: — | Assignee: — | Updated: 2026-09-18T20:54 +- Spam: `spam-promotional` score=3 — subject contains promotional language; ticket text contains promotional sender hint +- Triage score: **57.0** (stale>22h; status:new; unassigned+open) +- Entities: none detected +- Suggested action: Close as spam (spam-check → close-if-spam).; High attention: needs Dispatch containment review now. + +### #43692 — Re: #25643: New message from our team may require attention +- Status: New | Category: — | Assignee: — | Updated: 2026-09-18T20:40 +- Spam: `unknown-review` score=0 — no signals +- Triage score: **57.0** (stale>22h; status:new; unassigned+open) +- Entities: none detected +- Suggested action: High attention: needs Dispatch containment review now. + +### #43683 — Looking for 3 Prep Centers +- Status: New | Category: — | Assignee: — | Updated: 2026-09-18T14:04 +- Spam: `spam-promotional` score=2 — body/journal contains promotional language; ticket text contains promotional sender hint +- Triage score: **70.0** (stale>29h; status:new; unassigned+open) +- Entities: none detected +- Suggested action: Close as spam (spam-check → close-if-spam).; High attention: needs Dispatch containment review now. + +### #43682 — One customer generated $9,176.07 in 7 days +- Status: New | Category: — | Assignee: — | Updated: 2026-09-18T14:04 +- Spam: `spam-promotional` score=2 — body/journal contains promotional language; ticket text contains promotional sender hint +- Triage score: **70.0** (stale>29h; status:new; unassigned+open) +- Entities: none detected +- Suggested action: Close as spam (spam-check → close-if-spam).; High attention: needs Dispatch containment review now. + +### #43681 — Your call transcription from 937-632-8358 to 513-930-2899 +- Status: New | Category: — | Assignee: — | Updated: 2026-09-18T13:47 +- Spam: `automated-operational` score=0 — automated operational notification +- Triage score: **70.0** (stale>29h; status:new; unassigned+open) +- Entities: none detected +- Suggested action: Auto-categorize/route to Receipts/Bill Needs Paid and resolve if no action needed.; High attention: needs Dispatch containment review now. + +### #43676 — Special Low MOQ Friday Offer! +- Status: New | Category: — | Assignee: — | Updated: 2026-09-18T12:45 +- Spam: `spam-promotional` score=4 — sender has promotional/automated hint; subject contains promotional language; ticket text contains promotional sender hint +- Triage score: **70.0** (stale>30h; status:new; unassigned+open) +- Entities: none detected +- Suggested action: Close as spam (spam-check → close-if-spam).; High attention: needs Dispatch containment review now. + +### #43675 — Order ID: 1091195 - an inquiry about a parcel ref. tracking # 1ZY228K59096949586 (UPS) +- Status: New | Category: — | Assignee: — | Updated: 2026-09-18T12:34 +- Spam: `unknown-review` score=0 — no signals +- Triage score: **78.0** (stale>30h; status:new; unassigned+open; has-entities) +- Entities: tracking_ups:1ZY228K59096949586; order_id:1091195; order_id:PW9yG(medium) +- Suggested action: High attention: needs Dispatch containment review now.; Run `entities nmi` to attach private NMI brief note. + +### #43550 — Invoice 341570 from AM Shipping Supplies Co., Inc. +- Status: New | Category: Bill Needs Paid | Assignee: — | Updated: 2026-09-18T12:22 +- Spam: `automated-operational` score=0 — automated operational notification +- Triage score: **78.0** (stale>30h; status:new; unassigned+open; has-entities) +- Entities: account_ref:provided(medium) +- Suggested action: Auto-categorize/route to Receipts/Bill Needs Paid and resolve if no action needed.; High attention: needs Dispatch containment review now.; Run `entities nmi` to attach private NMI brief note. + +### #43542 — Ordered: "1000 Pack Secure Seal White..." +- Status: New | Category: Receipts | Assignee: — | Updated: 2026-09-18T12:20 +- Spam: `automated-operational` score=0 — automated operational notification +- Triage score: **78.0** (stale>30h; status:new; unassigned+open; has-entities) +- Entities: order_amazon:111-6425162-2592263; item_no:202029700(medium) +- Suggested action: Auto-categorize/route to Receipts/Bill Needs Paid and resolve if no action needed.; High attention: needs Dispatch containment review now.; Run `entities nmi` to attach private NMI brief note. + +### #43552 — Invoice 341571 from AM Shipping Supplies Co., Inc. +- Status: New | Category: Receipts | Assignee: — | Updated: 2026-09-18T12:19 +- Spam: `automated-operational` score=0 — automated operational notification +- Triage score: **78.0** (stale>30h; status:new; unassigned+open; has-entities) +- Entities: account_ref:provided(medium) +- Suggested action: Auto-categorize/route to Receipts/Bill Needs Paid and resolve if no action needed.; High attention: needs Dispatch containment review now.; Run `entities nmi` to attach private NMI brief note. + +### #43565 — Ordered: 2 "Ribbon Cutting Machine,..." and 6 more items +- Status: New | Category: Receipts | Assignee: — | Updated: 2026-09-18T12:14 +- Spam: `automated-operational` score=0 — automated operational notification +- Triage score: **78.0** (stale>30h; status:new; unassigned+open; has-entities) +- Entities: order_amazon:114-5077128-2677066; order_amazon:114-5944516-3478625; order_id:43564; item_no:202029700(medium) +- Suggested action: Auto-categorize/route to Receipts/Bill Needs Paid and resolve if no action needed.; High attention: needs Dispatch containment review now.; Run `entities nmi` to attach private NMI brief note. + +### #43569 — Shipped: "12" X 15" (500 Count) Self..." and 1 more item +- Status: New | Category: Receipts | Assignee: — | Updated: 2026-09-18T12:14 +- Spam: `automated-operational` score=0 — automated operational notification +- Triage score: **78.0** (stale>30h; status:new; unassigned+open; has-entities) +- Entities: order_amazon:114-5944516-3478625; item_no:200962600(medium) +- Suggested action: Auto-categorize/route to Receipts/Bill Needs Paid and resolve if no action needed.; High attention: needs Dispatch containment review now.; Run `entities nmi` to attach private NMI brief note. + +### #43572 — Re: Invoice 341571 from AM Shipping Supplies Co., Inc. +- Status: New | Category: Bill Needs Paid | Assignee: — | Updated: 2026-09-18T12:13 +- Spam: `automated-operational` score=0 — automated operational notification +- Triage score: **78.0** (stale>31h; status:new; unassigned+open; has-entities) +- Entities: account_ref:specialists(medium); account_ref:provided(medium) +- Suggested action: Auto-categorize/route to Receipts/Bill Needs Paid and resolve if no action needed.; High attention: needs Dispatch containment review now.; Run `entities nmi` to attach private NMI brief note. + +### #43586 — Shipped: 2 "Ribbon Cutting Machine,..." and 4 more items +- Status: New | Category: Receipts | Assignee: — | Updated: 2026-09-18T12:02 +- Spam: `automated-operational` score=0 — automated operational notification +- Triage score: **78.0** (stale>31h; status:new; unassigned+open; has-entities) +- Entities: order_amazon:114-5077128-2677066; item_no:200962600(medium) +- Suggested action: Auto-categorize/route to Receipts/Bill Needs Paid and resolve if no action needed.; High attention: needs Dispatch containment review now.; Run `entities nmi` to attach private NMI brief note. + +## Recommended next steps + +1. Run `spam-check --apply` then `close-if-spam --apply` on the cached set to clear obvious promotional spam (#43676). +2. For automated operational tickets already categorized (Receipts/Bill Needs Paid), consider a rule to auto-set status to Resolved with an internal note. +3. For #43675 (Cyberstorm tracking inquiry), queue an NMI private note via `entities nmi --issue-id 43675 --queue-private-note`. +4. For high-score tickets (e.g., #43690, #43711), perform containment pass: check escalation vectors, set priority/dates, create subtask if specific person needed. +5. Extend the same search to `quality-tracker`, `attendance-schedule`, and `meetings` once configs are added. + +_Report generated by maintainer using existing redmine_tickets.py / redmine_triage.py / redmine_ticket_entities.py (read-only, no Redmine writes)._ \ No newline at end of file diff --git a/docs/2026-07-14_localsearch_before_sanity.md b/docs/2026-07-14_localsearch_before_sanity.md new file mode 100644 index 0000000..005ad62 --- /dev/null +++ b/docs/2026-07-14_localsearch_before_sanity.md @@ -0,0 +1,218 @@ +# Local `search` in `redmine_tickets.py` (pre-sanity snapshot) + +**Date:** 2026-07-14 +**Status:** documentation of current behavior *before* any search sanity/cleanup pass +**Script:** `/home/jason/projects/redmine/redmine_tickets.py` +**Typical cache:** `/home/jason/projects/redmine/cache/redmine_tickets` + +This note captures what local `search` does and does **not** do as of 2026-07-14, after remote sync filters and run-log work landed. It is intentionally frank: docs had been thin; this is the baseline. + +--- + +## Mental model + +```text +sync = talk to Redmine → update local JSON cache +search = filter/list what is already cached +``` + +`search` is a **cache browser / structured filterer**, not: + +- a live Redmine issue query tool +- full-text corpus search +- the semantic Redmine search path (that lives elsewhere) + +If a ticket is not already under the cache directory for the selected project, **`search` cannot see it.** + +--- + +## Where data comes from + +Cached issues live as one JSON file per ticket: + +```text +//.json +``` + +Example for the durable customer-service cache: + +```text +/home/jason/projects/redmine/cache/redmine_tickets/customer-service/*.json +/home/jason/projects/redmine/cache/redmine_tickets/manifest.json +``` + +`search` loads via `TicketCache.iter_cached_issues()` (reads every `*.json` in the project subdir). It does **not** call Redmine and does **not** require `REDMINE_API_KEY`. + +--- + +## CLI surface (current) + +Parent options still apply: + +- `--base-url` (unused by search, but accepted on the parent parser) +- `--project` +- `--cache-dir` + +Subcommand: + +```bash +./redmine_tickets.py \ + --project customer-service \ + --cache-dir /home/jason/projects/redmine/cache/redmine_tickets \ + search [filters...] [output...] +``` + +### Filters (all local, AND’d) + +| Flag | Matches on | Notes | +|------|------------|--------| +| `--updated-since` | issue `updated_on` ≥ value | via `parse_dt` (absolute date/datetime) | +| `--updated-before` | issue `updated_on` ≤ value | same | +| `--status` | status **name** | case-insensitive **exact** match (`Closed`, `New`, …) — **not** `status_id` | +| `--category` | category **name** | case-insensitive **exact** match | + +Implementation: `filter_issues()` in `redmine_tickets.py`. + +### Order + limit + +| Flag | Behavior | +|------|----------| +| `--order newest-updated-first` | **default** | +| `--order oldest-updated-first` | | +| `--order id-asc` / `id-desc` | | +| `--order as-is` | filesystem glob order from `iter_cached_issues` | +| `--max-tickets N` | after filter + sort, keep first N | + +Implementation: `select_issues()`. + +### Output + +| Mode | What you get | +|------|----------------| +| default table | `#id`, `updated_on`, status name, category name, subject (via `print_issue_rows`) | +| `--json` | full cached issue payloads as one JSON array | +| `--jsonl` | one full issue JSON object per line (good for piping) | + +Human table is summary-only. Full subject/body/journals/helpdesk are only in `--json` / `--jsonl`. + +--- + +## What it allows (summary) + +1. Scope one project + one cache root. +2. Filter cached issues by `updated_on` window, status **name**, category **name**. +3. Sort and cap results. +4. Emit table, JSON, or JSONL. +5. Run offline against cache (no Redmine, no API key). + +--- + +## What it does **not** allow (current gaps) + +This is the important half of the snapshot: + +- **No free-text search** (subject / description / journals / from-address) +- **No sender / email / contact filters** +- **No tracker, assignee, priority, tags** +- **No `status_id` / `open` / `closed` shorthand** on search (only status **name**) +- **No remote/API filters** on `search` + (`--remote-updated-on`, `--remote-updated-since`, `--remote-status-id` are **sync-only**) +- **No relative time helpers** like `30d` / `2 days ago` on local search flags + (those helpers exist for **remote sync** convenience flags; local `--updated-since` wants a real parseable date/datetime) +- **No boolean logic** (AND/OR/NOT nesting) +- **No query language** comparable to Redmine UI filters or `rg` +- **No ranking / relevance** +- **No “search then sync missing ones”** +- **No pagination UI** beyond `--max-tickets` +- **No writes** (search never mutates cache or Redmine) + +### Practical content search today + +Pipe full payloads and use external tools: + +```bash +./redmine_tickets.py \ + --project customer-service \ + --cache-dir /home/jason/projects/redmine/cache/redmine_tickets \ + search --jsonl | rg -i 'gaylord|rate sheet' +``` + +Or inspect per-ticket files under the project cache dir directly. + +--- + +## Examples that work + +```bash +cd /home/jason/projects/redmine + +# Everything currently cached for customer-service (table) +./redmine_tickets.py \ + --project customer-service \ + --cache-dir /home/jason/projects/redmine/cache/redmine_tickets \ + search + +# Cached tickets updated since a date, newest first, cap 50 +./redmine_tickets.py \ + --project customer-service \ + --cache-dir /home/jason/projects/redmine/cache/redmine_tickets \ + search \ + --updated-since 2026-06-14 \ + --max-tickets 50 + +# Only closed, as full JSONL for downstream tools +./redmine_tickets.py \ + --project customer-service \ + --cache-dir /home/jason/projects/redmine/cache/redmine_tickets \ + search \ + --status Closed \ + --jsonl +``` + +--- + +## Related commands people confuse with “search” + +| Command | Relation to local search | +|---------|--------------------------| +| `sync` | fills the cache that `search` reads; supports **server-side** `status_id` + `updated_on` | +| `spam-check` | walks **cache**, scores spam; not general search | +| `known-spam-scan` | profile match over **cache** | +| `decompose` | one-ticket analysis, cache-first | +| semantic index / external search | separate stack; not this subcommand | + +--- + +## Remote vs local filter naming (context for sanity later) + +As of the 2026-07-14 sync work: + +| Side | Flags (examples) | Used by | +|------|------------------|---------| +| **Remote (API)** | `--status-id` / `--remote-status-id`, `--remote-updated-on`, `--remote-updated-since`, `--remote-updated-before` | `sync` list call only | +| **Local (post)** | `--updated-since`, `--updated-before`, `--status`, `--category` | `sync` post-filter **and** `search` | + +Help text for the shared local flags still says things like “applied after API list,” which is accurate for `sync` but slightly misleading when those same flags appear under `search` (where there is no API list). A future sanity pass may want search-specific wording. + +--- + +## Implementation pointers + +| Piece | Location | +|-------|----------| +| Filter logic | `filter_issues()` | +| Sort / max | `select_issues()` | +| Cache read | `TicketCache.iter_cached_issues()` | +| Table output | `print_issue_rows()` | +| CLI wiring | `build_parser()` → `search` subparser; `main()` branch `args.command == "search"` | +| Shared filter args | `add_common_filters()` | + +--- + +## Bottom line (baseline for sanity) + +**Does:** local structured filter/list over already-synced ticket JSON — date window, status name, category name, sort, limit, table/json/jsonl. + +**Doesn’t:** live Redmine query, free-text, contact/sender, remote filters, relative-date helpers on local flags, or advanced query syntax. + +Any “make search sane” work after this file should treat the above as the pre-change contract and call out intentional expansions (e.g. `--from`, `--subject-contains`, `--text`, relative dates on local filters) separately. diff --git a/docs/cli_power_user_client.md b/docs/cli_power_user_client.md new file mode 100644 index 0000000..8d9f1ea --- /dev/null +++ b/docs/cli_power_user_client.md @@ -0,0 +1,685 @@ +# Power-user Redmine client — usage (`redmine_tickets.py`) + +Scriptable local client for **humans and agents**: durable ticket cache, property search, Markdown view, and interactive walk. + +| Env | Redmine | Cache | +|-----|---------|--------| +| **Prod** | `https://rm.ldrprep.com` | `/home/jason/projects/redmine/cache/redmine_tickets` | +| **Dev / mutation testing** | LAN (e.g. `192.168.50.170` / `retired.int.ldrprep.com`) | `/home/jason/projects/redmine/cache-dev/redmine_tickets` | + +**Safety:** treat prod cache as read/sync. Test any **writes** on LAN first. Mutations impersonate `rebot` when implemented. + +Related: + +- Roadmap index: [raw-notes/features-for-redmine_tickets-sync.md](raw-notes/features-for-redmine_tickets-sync.md) +- Pre-MVP search snapshot: [2026-07-14_localsearch_before_sanity.md](2026-07-14_localsearch_before_sanity.md) +- Sync plan: [ticket_sync_tool_plan.md](ticket_sync_tool_plan.md) +- Cron scripts: `cron-scripts/sync-live-customer-service.sh`, `cron-scripts/sync-dev-customer-service.sh` + +--- + +## Setup + +```bash +cd /home/jason/projects/redmine +export REDMINE_API_KEY=… # required for sync / live API only +CACHE=/home/jason/projects/redmine/cache/redmine_tickets +PROJECT=customer-service +``` + +Shorthand used below: + +```bash +RT="./redmine_tickets.py --project $PROJECT --cache-dir $CACHE" +``` + +--- + +## Sync (fill / refresh cache) + +Remote filters hit Redmine; local cache is updated only for tickets whose `updated_on` changed. + +```bash +# Initial / catch-up window +$RT sync --remote-status-id '*' --remote-updated-since 30d --run-log + +# Frequent refresh (preferred after warm cache) +$RT sync --remote-status-id '*' --remote-updated-since 2h --run-log --quiet +``` + +| Flag | Meaning | +|------|---------| +| `--remote-status-id` / `--status-id` | Server: `*`, `open`, `closed`, or numeric | +| `--remote-updated-since` | Server: `30d`, `2h`, `YYYY-MM-DD`, … → `updated_on >= …` | +| `--remote-updated-on` | Server: raw Redmine operator string | +| `--run-log` | Ring-buffer stats → `$CACHE/run-stats.json` | +| `--run-log-keep N` | Keep last N runs (default 3; cron uses 20) | +| `--quiet` | Less progress noise | + +**Do not** use local `--updated-since` as a substitute for remote filters on large projects (local only post-filters stubs already listed). + +Cron note: live scripts currently still use `30d` on a short interval (~3–4s after warm-up). **Tighten later** to `2h`/`6h`. + +--- + +## Search (local cache only) + +No API key required. Only sees tickets already on disk. + +## Search table columns + +Default table (tab-separated): + +```text +#id updated_on status assignee contact category control_tags subject +``` + +Contact comes from `helpdesk_ticket.contact.email` (or name/company/from_address). + +### Classic filters + +```bash +$RT search --status New --max-tickets 50 +$RT search --updated-since 2026-07-01 --order newest-updated-first +``` + +### Property queries (`--where`) + +Engine: `redmine_ticket_query.py`. Repeatable `--where` = **AND**. + +```bash +$RT search --where 'subject.includes("account")' --max-tickets 20 + +$RT search --where 'assigned_to IS NULL' --max-tickets 10 + +$RT search \ + --where 'helpdesk_ticket.contact.email == helpdesk_ticket.contact.company' \ + --ids-only + +$RT search \ + --where '(subject.includes("verification code") OR subject.includes("magic link")) AND assigned_to IS NULL' +``` + +| Construct | Example | +|-----------|---------| +| Equality | `status.name == "New"` | +| Inequality | `assigned_to != "Jason T."` | +| Null | `assigned_to IS NULL` / `IS NOT NULL` | +| Substring | `subject.includes("rate sheet")` | +| Relative time | `updated_on.since("5 days")` | +| Path == path | `helpdesk_ticket.contact.email == helpdesk_ticket.contact.company` | +| Boolean | `AND` `OR` `&&` `||` + `()` | + +Nested dicts compared to strings use `.name` (or email/id) when present. + +### Output modes + +| Flag | Output | +|------|--------| +| (default) | Table: id, updated_on, status, category, subject | +| `--json` | Full issue JSON array | +| `--jsonl` | One full issue JSON per line | +| `--ids-only` | Issue ids only (best for scripts) | + +Match count goes to **stderr** on table mode so stdout stays pipe-clean for `--ids-only` / jsonl. + +### Automation pattern + +```bash +$RT search --where 'status.name == "New"' --where 'assigned_to IS NULL' --ids-only \ + | while read -r id; do + $RT view "$id" --print > "/tmp/ticket-$id.md" + done +``` + +--- + +## Saved filters (`saved-filters/`) + +Put one expression per file (comments with `#` allowed): + +```text +saved-filters/unassigned.expr +saved-filters/helpdeskEmailMatchCompany.expr +saved-filters/triage-needed-5d.expr +``` + +Reference them from CLI: + +```bash +$RT search --where '@unassigned' --max-tickets 20 +$RT search --where 'alias.helpdeskEmailMatchCompany' --ids-only +$RT walk --where '@triage-needed-5d' --max-tickets 10 --pager cat +``` + +Bare names that match a file also work: `--where unassigned`. + +### Short configs / wrappers + +```bash +./bin/rtl search --where 'assigned_to IS NULL' --max-tickets 10 # prod CS +./bin/rtd note --issue-id 39305 --body 'LAN' --apply # dev CS +./redmine_tickets.py --config configs/live.json ... +``` + +`--allow-prod-writes` is a **parent** flag (before the subcommand): + +```bash +./bin/rtl --allow-prod-writes draft apply +``` + +--- + +## Named queue folders (`materialize-queues`) + +Config: [queues.example.json](../queues.example.json) + +```bash +$RT materialize-queues --config queues.example.json +# writes: $CACHE/queues//*.json → symlinks into $CACHE/$PROJECT/.json +``` + +Agents that only understand files can list `$CACHE/queues/unassigned/`. + +Dry-run: + +```bash +$RT materialize-queues --config queues.example.json --dry-run +``` + +Refresh after cron sync if you want queues kept current (optional second cron step). + +--- + +## View (single ticket → Markdown) + +Cache-first. Does not call Redmine unless you add that later. + +```bash +$RT view 41934 --print +$RT view --issue-id 41934 --pager auto # glow → bat → batcat → cat +$RT view 41934 --write-md --pager cat +``` + +| Flag | Meaning | +|------|---------| +| `--print` | Markdown to stdout (no pager) | +| `--pager auto\|glow\|bat\|batcat\|cat` | How to display when not `--print` | +| `--write-md` | Force write under markdown cache | +| `--force-md` | Regen even if `updated_on` stamp matches | + +Markdown cache path: + +```text +$CACHE/$PROJECT/markdown/.md +$CACHE/$PROJECT/markdown/.md.updated_on +``` + +Regenerates when ticket `updated_on` changes (stamp file). + +--- + +## Walk (interactive feed) + +```bash +# From property search +$RT walk --where 'assigned_to IS NULL' --max-tickets 10 --pager cat + +# Explicit ids +$RT walk --ids 41934,41899 --pager cat +``` + +Controls: `n` / Enter = next, `p` = previous, `q` = quit, `r` = redraw. + +Same filter flags as search (`--where`, `--status`, `--max-tickets`, …). `--ids` overrides search selection. + +--- + +## Internal note vs customer email (critical) + +These are **different Redmine/Helpdesk operations**. The CLI uses **separate commands** so they are hard to confuse. + +| | `note` | `customer-email` | +|--|--------|------------------| +| **What it is** | Internal journal note on the issue | Helpdesk “send note” / customer reply | +| **Emails customer?** | **No** | **Yes** (when applied) | +| **API** | `PUT /issues/:id.json` with `notes` (+ optional `private_notes`) | `POST /helpdesk/email_note.json` with `message.content` | +| **Private?** | Optional `--private` | **Never** (private cannot email) | +| **Confirm flag** | None needed | **Required:** `--i-understand-this-emails-the-customer` | +| **Default** | Queue local draft | Queue local draft | +| **Apply now** | `--apply` | `--apply` (+ confirm flag already required) | +| **Prod writes** | Need `--allow-prod-writes` | Need `--allow-prod-writes` | +| **Impersonation** | `--impersonate-user rebot` (default) on apply | same | + +### Queue an internal note (safe default — no email) + +```bash +$RT note --issue-id 41934 --body 'Staff-only follow-up' --private +# later: +$RT draft apply # still does NOT email +``` + +### Apply internal note immediately (LAN/test preferred) + +```bash +$RT --base-url http://192.168.50.170 note --issue-id 123 \ + --body 'LAN test note' --apply +``` + +### Queue a customer email (explicit, scary name + confirm flag) + +```bash +$RT customer-email --issue-id 41934 \ + --body 'Thanks for writing — we received your request.' \ + --i-understand-this-emails-the-customer +# later: +$RT draft apply --i-understand-this-emails-the-customer +``` + +### Send customer email immediately + +```bash +$RT --base-url http://192.168.50.170 customer-email --issue-id 123 \ + --body 'LAN test customer reply' \ + --i-understand-this-emails-the-customer \ + --apply +``` + +### Production writes + +Writes to hosts outside the LAN allow-list (e.g. `https://rm.ldrprep.com`) are **refused** unless you pass `--allow-prod-writes`. Prefer testing on LAN first. + +### Draft list + +```bash +$RT draft list +# columns include EMAILS_CUSTOMER vs no-email +``` + +**Do not** use `note` when you mean to email a customer. **Do not** use `customer-email` for staff chatter. + +--- + +## set-fields (status / assignee / priority / category / due date) + +Does **not** email the customer. Default queues a draft; `--apply` writes (LAN-safe by default). + +```bash +# By numeric id +$RT set-fields --issue-id 41934 --status-id 5 --assigned-to-id 0 + +# By human name (uses meta cache; run meta-refresh once) +$RT meta-refresh --force +$RT set-fields --issue-id 41934 --status Closed --assigned-to unassigned \ + --notes 'Unassign + close' --private +``` + +| Flag | Meaning | +|------|---------| +| `--status` / `--status-id` | name or numeric status | +| `--assigned-to` / `--assigned-to-id` | name/login or id; `unassigned` / `0` clears | +| `--priority` / `--priority-id` | name or id | +| `--category` / `--category-id` | name or id; `none` / `0` clears | +| `--due-date` | `YYYY-MM-DD` | +| `--subject` | replace subject | +| `--notes` / `--private` | optional internal journal note with the update | +| `--refresh-meta` | force name→id cache refresh | + +Meta cache: `$CACHE/meta//` + +--- + +## Contacts (local mirror + light mutations) + +Separate from full `redmine_contacts.py` CRM CLI on purpose (different cache + surface), but **workflows can hand off**. + +```bash +# Mirror / search / show +$RT contacts-sync --max-contacts 100 +$RT contacts-search --query bavaria --max-contacts 20 +$RT contact-show --contact-id 1890 + +# Ticket → contact bridge (starts on ticket, ends on contact tools) +$RT ticket-contact --issue-id 41934 +$RT ticket-contact --issue-id 41934 --show --fetch + +# Contact mutation (draft/apply, same safety as tickets; does NOT email) +$RT contact-set --contact-id 4515 --add-tag maybe-customer +$RT --allow-prod-writes draft apply # if draft was for prod +# LAN apply: +$RT --base-url http://192.168.50.170 contact-set --contact-id 2284 \ + --add-tag maybe-customer --apply --refresh-cache +``` + +| Piece | Path | +|-------|------| +| Local mirror | `$CACHE/contacts/.json` | +| Full CRM (batch/loops/custom fields) | `redmine_contacts.py` | +| Ticket workbench contacts | `contacts-*`, `contact-set`, `ticket-contact` | + +Store: `$CACHE/contacts/.json` + `manifest.json`. + +--- + +## Local control tags (not Redmine tags) + +```bash +$RT control-tag add --tag bavaria --issue-id 41934 --property subject --annotation '…' +$RT control-tag list +$RT search --control-tag bavaria +$RT control-tag suggest --tag account --where 'subject.includes("account")' --max-tickets 20 +$RT control-tag suggest --tag account --where 'subject.includes("account")' --apply +$RT control-tag wipe --i-understand-this-wipes-local-control-tags +``` + +Store: `$CACHE/control-tags/tags.json` + +--- + +## Attachments + +```bash +$RT attachments list --issue-id 39305 --refresh +$RT attachments download --issue-id 39305 --all +$RT attachments upload --issue-id 39305 --file ./doc.pdf # dry-run +$RT --base-url http://192.168.50.170 attachments upload \ + --issue-id 39305 --file ./doc.pdf --apply --notes 'attached scan' +``` + +Default download dir: `$CACHE/$PROJECT/attachments//`. +Does **not** email the customer. Prod write needs `--allow-prod-writes`. + +--- + +## Entities (tracking / order patterns) + +```bash +$RT entities show --issue-id 41394 +$RT entities scan --where 'subject.includes("1Z")' --max-tickets 20 +$RT entities scan --type tracking_ups --ids-only +``` + +Heuristic only — not NMI-backed. Also appears in `view` Markdown. + +--- + +## Browse (interactive loop: list → pager → actions) + +```bash +# Default: fzf (preview pane) → glow/bat pager → action menu → back to fzf +./bin/rtl browse --where 'assigned_to IS NULL' --max-tickets 80 --open-only + +# Explicit pager / no preview +./bin/rtl browse --where '@unassigned' --pager glow +./bin/rtl browse --status New --no-preview + +# One-shot (pick once, page, exit — no menu/loop) +./bin/rtl browse --where 'subject.includes("invoice")' --once +``` + +### Loop keys + +| Step | What happens | +|------|----------------| +| **List** | fzf list (Esc quits). Header lists actions. Preview shows ticket snippet. | +| **Enter** | Open ticket Markdown in glow/bat/batcat (`--pager`) | +| **After pager** | Action menu for that ticket | + +### Action menu + +| Key | Action | +|-----|--------| +| Enter / `v` | View again | +| `n` | Queue **internal note** (no email) | +| `p` | Queue **private** internal note | +| `f` | Queue **field update** (status/assignee by name) | +| `t` | Add **local control-tag** | +| `c` | Show ticket→contact handoff | +| `e` | Extract entities (patterns) | +| `m` | **NMI brief** → confirm → queue as **private** note draft | +| `i` | Print issue id | +| `l` | Back to list | +| `q` | Quit browse | + +Mutations from browse **queue drafts only** (loud `draft_id=` banner). Apply later with `draft apply`. +Customer email is **not** in the menu (use CLI deliberately). + +```bash +# Notes from file (+ optional Textile via pandoc) +./bin/rtl note --issue-id 41934 --body-file ./reply.md --private +./bin/rtl note --issue-id 41934 --body-file ./reply.md --to-textile --private + +# Open tickets only (search/browse) +./bin/rtl search --open-only --where 'assigned_to IS NULL' --max-tickets 20 +``` + +Non-interactive (no TTY): prints list lines and exits. + +--- + +## OCR rename (scanned attachments → better subject) + +```bash +# One ticket +./bin/rtl ocr-rename --issue-id 12345 + +# Scan for generic "Scanned document…" tickets +./bin/rtl ocr-rename --where '@scanned-document' --only-generic-subjects --max-tickets 15 + +# Apply subject on LAN immediately (note still queued private by default) +./bin/rtd ocr-rename --issue-id 12345 --apply-subject +``` + +Uses `pdftotext`, else `pdftoppm`+`tesseract`. Queues subject `fields` draft + private OCR note (disable with `--no-queue-subject` / `--no-queue-note`). + +--- + +## Categories + +```bash +./bin/rtl categories list +./bin/rtl categories list --refresh +``` + +--- + +## Entities + NMI + +```bash +./bin/rtl entities show --issue-id 41394 +./bin/rtl entities nmi --issue-id 41394 +# Env: NMI_CLI, NMI_ENV_FILE, NMI_MODE — see docs/raw-notes/features/nmi-bridge.md +# Browse action `m` queues NMI brief as private note draft +``` + +--- + +## Local drafts (queue + apply) + +```bash +$RT draft list +$RT draft cancel +$RT draft cancel --all-pending +$RT draft apply # internal notes +$RT draft apply --i-understand-this-emails-the-customer # customer_email only +``` + +Stored under `$CACHE/drafts/*.json`. + +--- + +## Other subcommands (ops) + +| Command | Role | +|---------|------| +| `spam-check` / `close-if-spam` | Deterministic spam tooling (LAN for apply) | +| `known-spam-scan` / `known-spam-close` | Profile-based spam | +| `tag-maybe-customer` | Contact tagging from inquiry evidence | +| `decompose` | Dry-run vault-oriented plan for one ticket | + +--- + +## Modules + +| File | Role | +|------|------| +| `redmine_tickets.py` | CLI entry: sync, search, view, walk, spam, … | +| `redmine_ticket_query.py` | Property expression compiler | +| `redmine_ticket_view.py` | Markdown render + pager | +| `redmine_contacts.py` | Shared Redmine client / contacts | + +Tests: `tests/test_redmine_ticket_query_and_view.py`, `tests/test_redmine_tickets_remote_filters.py`. + +--- + +## What is **not** here yet + +- Queued notes / helpdesk replies / field mutations (P2 remaining) +- Free-text semantic search (Qdrant / semantic_index — separate track) +- SQLite projection layer (planned architecture, not shipped) + +**Shipped for queues/filters:** saved-filters + `materialize-queues` (see above). + +--- + +## Quick “is the client alive?” check + +```bash +$RT search --max-tickets 3 +$RT search --where 'assigned_to IS NULL' --max-tickets 3 --ids-only +ID=$($RT search --max-tickets 1 --ids-only | head -1) +$RT view "$ID" --print | head -30 +``` + +--- + +## TUI (curses keyboard-driven batch processor) [NEW] + +Fast single-screen TUI for the exact workflow you described: queue a template of changes, scroll/cycle tickets, paint the template onto many tickets with one key, then one button to write everything. + +```bash +./bin/rtl tui --where 'assigned_to IS NULL' --max-tickets 60 --open-only +./bin/rtl tui --control-tag triage-needed --max-tickets 30 +``` + +**Workflow in the TUI:** +1. **Set template once** (optional fields; menus or type+filter): + - `c` : Category picker (from meta cache; type to filter) + - `a` : Assignee picker (unassigned=0) + - `s` : Status picker + - `p` : Priority (type a number, e.g. 2; hotkeys 1/2/3 also work) + - `P` : Project ID (for moving the ticket) + - `t` : Local control tag (string) + - `C` : Add contact tag (repeatable; creates if new) + - `n` : Private note (applies to painted tickets) + - `e` / `m` : Toggle entity-extraction / NMI-brief flags (become control tags + notes) + +2. **Navigate the list**: + - `j` / `k` / arrows : move + - `/` : filter the visible list (live) + - `r` : redraw + +3. **Paint / apply**: + - `SPACE` or `Enter` on a ticket: **apply the current template** to it (records per-ticket pending: fields, note, tags, toggles). Shows marker [FNTEM] etc. + +4. **Commit**: + - `A` : Queue drafts for all painted tickets (safe default). + - (If you passed `--apply` and have `--allow-prod-writes` on LAN/prod, `S` for direct apply — use with care.) + +Other keys: `x` clear pending on current, `R` force meta refresh, `q` quit (prompts if pending), `?` help. + +**What gets written on commit:** +- Field updates (status/assignee/priority/category/project) via KIND_FIELDS draft (or direct). +- Private note (per ticket or from template). +- Local control tags (via control-tag add). +- Contact tags (via contact_fields drafts for the helpdesk contact). +- Special toggles → control tags ("entity-extract") + placeholder NMI note if needed. + +All the safety rules (drafts vs apply, prod gate, impersonate rebot) still apply. The TUI just removes the per-ticket menu tax. + +**Tips:** +- Run `meta-refresh --force` once for full picker lists. +- Combine with saved filters: `./bin/rtl tui --where '@unassigned'` +- The TUI loads from the same cache/search logic as browse/walk. + + +## TUI Commit behavior update (auto-apply on A) + +As of latest change: + +- Pressing **A** in the TUI now: + 1. Shows a **single** "Are you SURE?" confirmation for the whole batch. + 2. Queues the drafts for every painted ticket. + 3. Immediately applies those drafts (using the normal apply_draft_to_redmine + mark_applied path). + +This removes the need to manually run `draft apply ` for each ticket. + +`S` remains available for the old direct-apply (bypass draft) path if you ever want it. + +All normal safety (LAN/prod gate, --allow-prod-writes, impersonate rebot, no customer emails from this path) remains in force. + +See the module docstring in redmine_ticket_tui.py or run the TUI and press `?` for current keys. + + +## NMI Enrichment Improvements (entities nmi) + +The `entities nmi` command (and the NMI path used by browse 'm' and TUI) now: + +- Automatically generates rich deep links when it sees item numbers, tracking, request ids, order ids, accounts: + - Item Manager + Item Statement for item_no + - Box History (boxFinder) for tracking numbers + - Order summary and package/request history pages + - User Info for account numbers + (See also the reference note: [[INBOX/WIP/Linking to Entities Within NMI]] ) + +- When an item_no is resolved, it calls the new `getItemHistory` API endpoint via `nmi item history --item-no=... --snapshot`: + - Returns merged timeline of product_history (moves, condition changes), service_fees (billings), and comments. + - The private note / cs-brief output now includes a "=== NMI Deep Links ===" section and a "=== Item History (getItemHistory) ===" section (truncated for notes; full in --format json). + +Usage: +``` +rtl entities nmi --issue-id 123456 --format note +rtl entities nmi --issue-id 123456 --format json # full history payload +rtl entities nmi --issue-id 123456 --no-history +``` + +The same enrichment logic feeds the TUI and browse NMI quick action. + + +--- + +## Post-apply cache refresh (anti-staleness) + +After any mutation that hits live Redmine (TUI `A`, `draft apply`, `set-fields --apply`, direct notes, etc.) the affected ticket's JSON is immediately re-fetched and overwritten in the local cache. + +This means: +- Next `rtl browse`, `rtl tui`, or search sees the new status / assignee / journals / updated_on right away. +- No need to wait for the next full sync for the tickets you just touched. + +In the TUI: +- `r` → local redraw (fast) +- `R` → force live refresh of the **currently highlighted** ticket (pulls fresh from Redmine + updates cache + view) + +This directly supports the rule: "update the existing cached file when the ticket has changed" and "agents update the ticket when their work is done." + + +### TUI "S" key (direct apply) + +`S` is the **direct/bypass** path (still requires write permission). + +- It skips the draft queue for field changes and does an immediate `update_issue`. +- We now also call the same live cache refresh that `A` uses, so the TUI and later `browse` see the change. +- Behavior difference from `A`: + - No "are you sure?" batch confirmation. + - No draft records created for the main fields. + - Still gated by `--allow-prod-writes` (or `--apply`). + +Current recommended path for most batch work remains **A** (one confirm, goes through drafts for auditability, then auto-applies). + +### TUI safety / UX improvements (2026-07) + +- If you start without `--allow-prod-writes`, a clear **READ-ONLY** banner appears near the top. +- Bottom status/help line is now wrapped to avoid `_curses.error` on narrow/short terminals (common when ssh'ing from Windows Terminal, Tabby, or split panes). +- After confirming `A`, the screen immediately shows "COMMIT + APPLY IN PROGRESS" feedback so it doesn't look hung while the writes happen. +- `S` now shows "fields updated DIRECTLY" + cache sync messages when it actually performs a live write. + diff --git a/docs/contact_loop_filter_contract.md b/docs/contact_loop_filter_contract.md new file mode 100644 index 0000000..357664c --- /dev/null +++ b/docs/contact_loop_filter_contract.md @@ -0,0 +1,183 @@ +# Contact Loop Filter Contract + +## Purpose + +This document defines the extensible contact-tagging loop contract for `redmine_contacts.py loop-run`. + +The loop runner is intentionally boring: it selects contacts, fetches recent Redmine helpdesk context, calls filters, and handles generic queue-state tags. Rich business logic belongs in FILTER-2 scripts/programs. + +## Architecture + +```mermaid +flowchart TD + A[loop-run profile + CLI filters] --> B[Select contacts] + B --> C{For each contact} + C --> D[Run FILTER-1] + D -->|false| E[Runner adds DTAG] + D -->|integer X| F[Runner fetches X recent tickets] + F --> G[Optional junk/text sanitizer] + G --> H[Run FILTER-2 command] + H -->|false / no-action| E + H -->|true / handled| I[Runner removes RTAGS] + E --> J[Next contact] + I --> J +``` + +## Runtime profile + +Profiles are JSON files. The initial version deliberately supports a small surface that can grow without forcing all policy logic into the main CLI. + +Example: + +```json +{ + "name": "noop-ai-checked", + "default_tag": "ai-checked", + "remove_tags": ["needs-review"], + "filter_1": { + "type": "builtin", + "name": "fixed-ticket-limit", + "limit": 5 + }, + "ticket_fetch": { + "fetch_full_issues": false, + "message_limit": 25, + "sanitize_text": true + }, + "filter_2": { + "type": "command", + "argv": ["python3", "filters/noop_filter2.py"], + "timeout_seconds": 30, + "allow_mutation": false + } +} +``` + +Required fields: + +- `default_tag`: tag added by the runner when FILTER-1 or FILTER-2 returns false. +- `filter_1`: contact-level gate. +- `filter_2`: ticket/contact worker. + +Optional fields: + +- `name`: human label for reports. +- `remove_tags`: tags removed by the runner when FILTER-2 returns true. +- `ticket_fetch.default_limit`: fallback ticket count if FILTER-1 omits a limit. +- `ticket_fetch.fetch_full_issues`: if true, fetch `issue-full` payloads for each recent issue. +- `ticket_fetch.message_limit`: message limit for full issue payloads. +- `ticket_fetch.sanitize_text`: compact obvious junk/noisy text before passing to FILTER-2. + +## FILTER-1 contract + +FILTER-1 receives a contact object and decides whether to continue. + +Supported v0 modes: + +- builtin `fixed-ticket-limit`: returns a configured integer. +- command filters can be added using the same stdin/stdout JSON pattern as FILTER-2. + +Result meanings: + +```json +{"result": false, "reason": "skip this contact"} +``` + +Runner behavior: add `default_tag`, do not fetch tickets, move to next contact. + +```json +{"result": 5, "reason": "inspect recent tickets"} +``` + +Runner behavior: fetch 5 recent helpdesk issues for this contact and call FILTER-2. + +## FILTER-2 contract + +FILTER-2 is the extensibility point for meaningful business workflows. It may classify contacts, update tickets, update contacts, create/link external records, or do nothing. + +The runner passes one JSON bundle on stdin: + +```json +{ + "profile": { + "name": "noop-ai-checked", + "default_tag": "ai-checked", + "remove_tags": ["needs-review"] + }, + "contact": {"id": 123}, + "issues": [ + { + "issue_summary": {"issue_id": 456, "subject": "Example"}, + "full_issue": null + } + ], + "runtime": { + "base_url": "http://192.168.50.170", + "project": "customer-service", + "dry_run": true, + "helper_command": "/home/jason/projects/redmine/redmine_contacts.py" + } +} +``` + +FILTER-2 writes JSON to stdout: + +```json +{"result": false, "summary": "not enough evidence"} +``` + +Runner behavior: add `default_tag`. + +```json +{ + "result": true, + "summary": "classified as customer", + "mutations": ["contact:123 add customer, needs-enriched, needs-nmi-sync"] +} +``` + +Runner behavior: remove `remove_tags` from the contact. FILTER-2 is responsible for any domain-specific mutations it claims to perform. + +## Redmine access for FILTER-2 + +FILTER-2 programs get standard environment variables: + +- `REDMINE_BASE_URL` +- `REDMINE_PROJECT` +- `REDMINE_API_KEY` (inherited only if present; never printed by the runner) +- `REDMINE_CONTACTS_HELPER` +- `REDMINE_LOOP_DRY_RUN` (`1` unless `loop-run --apply` is used) + +Command FILTER paths are executed with the profile directory as the working directory unless `filter_2.cwd` overrides it. Keep profile-local scripts under `filters/` when possible. + +Recommended mutation path for v0 workers is to call the helper CLI, for example: + +```bash +$REDMINE_CONTACTS_HELPER update 123 --add-tag customer --apply +``` + +In dry-run mode, FILTER-2 must not mutate Redmine. The runner sets `REDMINE_LOOP_DRY_RUN=1` so scripts can enforce this consistently. + +## Runner-owned semantics + +The runner always owns generic queue state: + +- FILTER-1 false -> add `default_tag`. +- FILTER-2 false -> add `default_tag`. +- FILTER-2 true -> remove `remove_tags`. +- FILTER crash/non-JSON -> fail the run; do not mutate that contact as a fallback. + +This keeps specialized FILTER-2 programs powerful without making failure handling spooky. + +## Initial tracer goal + +The first implementation should prove the conveyor belt, not intelligence: + +- select a filtered contact batch, +- run builtin FILTER-1 with a fixed ticket limit, +- fetch recent helpdesk issues, +- call a no-op FILTER-2 command, +- dry-run the default tag update, +- optionally apply it with `--apply`. + +Once that is reliable, richer FILTER-2 programs can classify customers, automailers, vendors, enrichment targets, and company/person relationships. diff --git a/docs/contact_triage_loop_draft_2026-07-09.md b/docs/contact_triage_loop_draft_2026-07-09.md new file mode 100644 index 0000000..d62309f --- /dev/null +++ b/docs/contact_triage_loop_draft_2026-07-09.md @@ -0,0 +1,57 @@ +# Contact Triage Loop – Draft-Stage Contact-First Approach (2026-07-09) + +**Status**: Draft / exploratory. Intended as a lightweight, iterative starting point that can be tested manually in small batches before any automation is written. + +**Context**: This note captures the refined direction discussed on 2026-07-09 for cleaning the customer-service project (currently >1700 open tickets). The previous keyword-on-subject approach is deprioritized. The stronger signal is **contact-first review** using recent activity history. + +## Core Principle (Contact-First) +- Group decisions by **contact** (author/email) rather than individual tickets. +- For each contact, examine their **most recent 3–5 tickets/messages**. +- Decide whether that contact (and most of their open tickets) can be safely closed as spam, stale, promotional, or low-value. +- This directly targets the observed pattern that many spam/low-value tickets come from the same small set of repeat contacts. + +## Draft Workflow (Repeatable in Small Batches) + +1. **Discover candidate contacts** + Find authors with multiple open tickets whose most recent activity is old or low-value. + Starting signals: + - Authors with 5+ open tickets + - Authors whose most recent ticket is >60–90 days old + - Contacts whose tickets are mostly short/generic with zero follow-up + +2. **Per-contact review (the decision step)** + For each contact, review their last 3–5 tickets and ask: + - Is this contact mainly sending spam, sales pitches, or unrelated noise? + - Has there been any meaningful reply or resolution path in the last 60–90 days? + - Are the tickets duplicates, test messages, or one-off low-value imports? + - Do they match existing `spam` or `promotional` tag patterns? + +3. **Close decision rules** + Close most or all of a contact’s open tickets if any of the following are true: + - Recent pattern is clearly spam/stale/no-value + - No meaningful activity or resolution path in 90+ days + - Tickets are duplicates or low-value imports + + **Do not close** if the contact shows any recent legitimate customer issue or money-related matter. + +4. **Execution & logging** + - Use existing Redmine MCP tools or the semantic search CLI on mcpness for discovery. + - Apply `spam`/`promotional` tags where appropriate for future filtering. + - Record a short note on the contact or in a running log (which contacts were reviewed and outcome). + +5. **Guardrails** + - Never close tickets involving money, formal complaints, or recent customer replies. + - When in doubt, skip the contact. + - After bulk cleanup, maintain a monthly review of “Open tickets from contacts with no activity in 60+ days.” + +## Why This Stays Draft-Stage for Now +- Allows manual testing on the first 10–20 worst contacts to see what the real data looks like. +- Validates the contact-history signal before investing in looping scripts or automation. +- Directly supports shrinking the 1700+ open ticket count with low risk. + +## Next Natural Steps (for the other thread) +- Run the first discovery pass on the worst offending contacts using current tools. +- Refine the decision questions or tagging rules based on what the contacts actually show. +- Decide whether to move any part of this into a lightweight script once the manual pattern is validated. + +This note is a clean hand-off for the active thread working on the contact triage loop project. \ No newline at end of file diff --git a/docs/contact_triage_loop_raw_notes_2026-07-04.md b/docs/contact_triage_loop_raw_notes_2026-07-04.md new file mode 100644 index 0000000..a723c3b --- /dev/null +++ b/docs/contact_triage_loop_raw_notes_2026-07-04.md @@ -0,0 +1,154 @@ +eh, I think you misunderstood me about the filters. I see this: + +` batch-update Apply one payload to a filtered contact batch.` + +Unless I'm misunderstanding how to use that. it sounds like it's not what I wanted/ + +I was hoping to be able to do this: + +1. In batches of perhaps 10-20, pull a list of contacts that have not been updated for a while, and who either: + - also have specific tags + - also lack specific tags + +2. For each of those contacts, do the following: + - check if there's anything about the contact itself that informs me I shouldn't need to look at anything else about it, and I can quickly tag them as something like "spam". I expect most of the time, I _won't_ be able to do that. + + - Fetch the 3-5 most recent tickets from the contact, and look at them, using some framework to score any tags that should apply + - Examples: + - **customer**: + - somebody on our side emailed back +1 + - seems like they're talking about packages or items in our warehouse +1 + + - **promo** + **spam**: contact seems like a business sending us advertisements + - from address is something like "no-reply", "noreply", "deals", or "promotions" +1 + - it mentions a deal, discount, or coupon +1 + + - If the scores for any particular set set of tags >= a threshhold, use add_tag to add those tags to the contact + - otherwise, add a tag like "ai-checked" + - The last step caused the updated date of the contact to increase + +3. Due to the very last step above (which bumps the updated date), if I wait and loop back through Step 1 again a few minutes or hours later, I won't retrieve the same contacts I already tagged. + - In this way, a looping/repeating task will eventually cycle through all contacts. + - Eventually, the ones it has already tagged will have been updated long enough ago, that they will get pulled again + +## To be clear: + +I am describing a broad pattern of workflow here, not specifics. I am just explaining this because: + + a) The way you implemented the changes seems counterintuitive for how I want to use this, but I could of course be misunderstanding. + + b) I think the basic concept of this loop should be added to the project documentation, maybe even with a diagram + +## Further down the road: + +- [ ] I will want to develop tags that indicates a specific kind of evaluation needs performed with a contact, just as summarizing their most recent communications with us. + - [ ] When looping through contacts, I can add that tag to contacts whose comms need to be summarized + - [ ] Then, another agent or script can fetch only contacts tagged to indicate they need summarized + - [ ] That script/agent fetches their recent communications, does it's job, and removes the tag, indicating that job is finished +- [ ] Other likely tags I will eventually implement: + - [ ] basic info needs corrected: + - Typical auto-import problems exist in the contact data, such as: + - company field contains their email address + - username portion of email address is in first_name, but their actual name is in the footer of the email or in the "from" part + - basic contact info is actually in the ticket, but not in the correct contact fields: + - first name, last name + - additional email addresses + - phone number(s) + - job title + - name of their business + - We know the name of the company they work for, but no contact.is_company=true exists with that name + - Create it + - Tag it as needing enrichment + - Update the contact(s) who work for that company, setting company = that company + - [ ] **NMI mapping**: + - [ ] Contact seems to be a customer, but their "Account Number" field is blank, so we're not sure which NMI account is their's + - [ ] Try to figure it out by comparing stuff like: + - [ ] email address + - [ ] name of company + - [ ] tracking numbers or order ids mentioned + - [ ] ^^ above, but for other contacts who work at the same company + - [ ] Update linkages all around: + - [ ] Update that company (and it's employees) contacts in redmine so "Account Number" matches `user_account.user_account_id` in NMI + - [ ] Set the redmine.contact_id's into a mapping table in NMI that link which contacts are for which account + - [ ] Update obsidian profile about the contact/person to include both pieces of info + - [ ] **entity extraction and enrichment**: + - members of our team responded to somebody, but we don't have (much) info about them in either redmine contacts or in obsidian + - This tells us the person is probably an active employee, so we: + - search for 5-10 other tickets they worked on lately, and use that to infer a profile of what their job is or fill in context about what they've worked on lately + - **phone number linkage to accounts**: + - [ ] transcribe/ingest tickets that are call transcripts, adding summary to contact file automatically + - [ ] add "note/call" to contact automatically + - [ ] identify if follow-up action is needed; if it is, create ticket, assigned to relevant person + + - [ ] **bobcat caller id**: recognized contact calls us... + - [ ] **route call based on context, not just the number**: + - [ ] we're super busy, short staffed, and person likes to talk alot: "sorry, we're not available" + - [ ] Bob usually talks to this person, but Bob is marked "do not disturb": "voice mail / not available" + - [ ] **show on-screen bobcat overlay caller-id** + - [ ] **Needing enrichment** (likely more than one tag to represent the kind of enrichment): + - [ ] **social media**: + - [ ] try to identify their social media profiles (linkedin, facebook, twitter, etc) using name, email, etc. + - [ ] possible search known social media groups for people with same/similar name + - [ ] identify product website(s)= + - [ ] **potential suppliers**: + - [ ] search places like alibaba for products we've seen them sell, and try to narrow down who their suppliers are + - [ ] pull checkin pictures of shipping labels from them, and cross-reference with other info to locate possible suppliers + - [ ] **special handling**: + - [ ] **SLA/VIP**: this contact needs very fast or special responses for some reason + - [ ] possibly tied to dedicated agent who does something special with their emails, like: + - [ ] **trace package**: pull data we'd need to trace the workflows about it: + - [ ] if it looks like it has a tracking number, kick off the workflow to pull that up in NMI, get pictures, check who worked on it, tracking status, etc. + - [ ] Possibly fetch surveillance from synology autmatically: + - [ ] For each major timestamp involved ( box.checkin_date, box.presort_date, individual_product.checkin_date ) pull 2-3 minutes of footage before/after from cameras known to overlook the 2-3 nearest locations in the warehouse + - [ ] split the video feed into a series of still images 30-seconds apart + - [ ] have a VLM explain what it sees in the images, even checking for specific things: + - [ ] presence of truck at loading dock + - [ ] person seen walking away from area carrying similar locking box/item + - [ ] **import order**: email is an invoice, PO, or similar listing what a person is sending to us, perhaps the reason, and when it should arrive. + - [ ] _they're a vendor_: log that supplies were being delivered and what we spent on them + - [ ] _prep customer_: make a prep inbound order if it doesn't exist already + - [ ] **vendor receipt or invoice**: + - [ ] automatically log/input that into the expense tracker if it's not there already + - [ ] check if there is a purchase order for it. + - [ ] if there is, do something like mark it en-route, received, etc. + - [ ] create ticket, assigning somebody to put it away + - [ ] assign an AI-INFO-BLOCK to the manager, asking them to find out who ordered it, what for, and why they didn't make a PO + - [ ] **employee and it's about scheduling** + - [ ] check the attendance-schedule project to see if they already reported it in there + - [ ] if not, add it + - [ ] assign an AI-INFO-BLOCK asking them to explain why they could not do that themselves + - [ ] **check for time-off blockers**: + - [ ] somebody else already has time off within 1-2 days + - [ ] customer has complained about delays in the past week + - [ ] same person asked off on the same weekday/pattern in the past 3-4 weeks, and no good reason + - [ ] overall billings/productivity have been low for over a week + - [ ] **if blocked**: warn them they can't have the day off unless they can prove it's manditory/unavoidable + - [ ] if they say it is, AI-INFO-BLOCK until they provide proof or admit they don't have it + - [ ] **very simple question** + - [ ] **sales 20-questions**: they sent an email that amounts to playing 20-questions about capacity (wanting to start project) + - [ ] **AI-agent uses obsidian and other data**: + - [ ] generate really slick looking brochure quickly, using obsidian-backed data + - [ ] loop human in to approve it + - [ ] update/expand based on human input, then send it over to client + - [ ] save it for later, so next time, it can just be sent right away + - [ ] **spammer-be-gone**: + - [ ] Identify habitual or serious spammers, time wasters, and promoters emailing us + - [ ] use gmail MCP to add filters to contact@ldrprep.com so that person's mail doesn't get imported into redmine anymore + - [ ] **high-risk contact**: + - [ ] there is some special risk or liability related to talking to that contact. For examples: + - [ ] litigious person or debt collector + - [ ] disgruntled former employee + - [ ] stalker spouse of current employee + - [ ] handle ticket(s) in a manner people who shouldn't talk to them can't see it or can't respond to it + - [ ] phone calls from them do not ring to the general line anyone can answer + - [ ] **lead follow-up**: + - [ ] contacts marked as wanting to potentially buy something from us or start services, but we've heard crickets for 7+ days + - [ ] encourage somebody to call/email them, maybe even suggest what to say/send + +## Please ensure this info is saved with the project and easy to locate + +For right now, much of this will not be immediately practical to implement, but I think storing it here is both important context and saves time. + + + + diff --git a/docs/contact_triage_loop_workflow.md b/docs/contact_triage_loop_workflow.md new file mode 100644 index 0000000..3d81e4e --- /dev/null +++ b/docs/contact_triage_loop_workflow.md @@ -0,0 +1,211 @@ +# Contact Triage Loop Workflow (Vision + Operating Pattern) + +## Why this exists + +This document captures Jason's intended **repeatable contact-review loop** so work can proceed in deterministic batches without reprocessing the same contacts immediately. + +Captured from uploaded project note on **2026-07-04** (saved into repo context to avoid losing the design intent). + +Raw source capture: +- `/home/jason/projects/redmine/docs/contact_triage_loop_raw_notes_2026-07-04.md` + +--- + +## Core loop pattern + +This is a **multi-pass routing system**, not a final one-shot classifier. + +1. Select a small batch (10-20 contacts) where `updated_on` is old enough and optional tag constraints match. +2. Sort **newest stale first** (recently stale / last-updated descending), so the loop works through the freshest cooled-down contacts first. +3. For each contact: + - quick contact-only triage + - inspect recent helpdesk tickets/messages when the current pass requires it + - route by adding/removing tags that represent either classifications or workflow state +4. Apply tag updates: + - if the pass identifies a route, add that route/classification tag + - otherwise add a status marker like `ai-checked` +5. Because contact metadata was updated, `updated_on` advances. +6. On future runs, recently processed contacts fall out of the immediate stale batch; later passes can pick them up again when stale or when a designated tag routes them to another worker. +7. Later filters/workers may deliberately remove status tags such as `ai-checked` after consuming them, allowing the contact to re-enter another pass of the conveyor belt. + +This forms a self-advancing conveyor belt for incremental cleanup/enrichment. Tags are not all semantic classifications: some are routing/status markers. For example, `ai-checked` means this pass touched the contact; it does not mean the contact's final identity is "AI checked." Likewise, `spam`/`promotions` can be step 1 in a longer handling pipeline, not the final disposition. + +Operational objective: reducing clutter in the customer-service/sales channel is valid even when a sender is technically a vendor, billing system, automailer, or promotional sender, because Redmine customer-service is not the only channel available to them. + +--- + +## Existing tooling alignment (current) + +`redmine_contacts.py` already supports key building blocks: + +- batch candidate selection: + - `batch-update --max-contacts` + - date/tag filters (`--updated-before`, `--with-tag`, `--without-tag`, `--no-tags`) + - deterministic ordering (`--order newest-updated-first` etc) +- safe mutation workflow: + - dry-run by default + - write only with `--apply` + +What is still missing for full loop automation: + +- richer FILTER-2 workers that classify contacts and mutate tickets/contacts when needed +- scoring engines that map evidence to tags with thresholds +- downstream queue workers for enrichment/NMI/company/person linking + +Implemented tracer pieces: + +- `redmine_contacts.py loop-run` +- profile contract in `docs/contact_loop_filter_contract.md` +- sample tracer profile: `profiles/contact-loop/noop-ai-checked.json` +- sample no-op FILTER-2 worker: `profiles/contact-loop/filters/noop_filter2.py` + +The intended architecture is now: **runner owns conveyor-belt mechanics; FILTER-2 owns domain-specific ticket/contact mutation logic.** + +--- + +## Suggested scoring framework (v0) + +### customer +Signals: +- internal/team response exists (`+1`) +- discussion indicates warehouse packages/items (`+1`) + +### promo + spam +Signals: +- sender hints (`no-reply`, `noreply`, `deals`, `promotions`) (`+1`) +- message mentions discount/coupon/deal language (`+1`) + +Threshold model (initial): +- tag set applied when score >= `2` +- if no tag set crosses threshold: add `ai-checked` + +--- + +## Diagrams + +### Diagram 1 — Conveyor-belt loop + +```mermaid +flowchart TD + A[Candidate pool\nRedmine contacts] --> B[Select stale contacts\nupdated_before + tag filters\nmax 10-20] + B --> C[Sort newest stale first\nupdated_on descending] + C --> D{For each contact} + D --> E[Run current pass\nFILTER-1 / worker gate] + E -->|pass can route immediately| F[Add route/status/classification tags] + E -->|needs evidence| G[Fetch recent tickets/messages] + G --> H[Manual review or FILTER-2 worker] + H --> I{Route decided?} + I -->|yes| F + I -->|no| J[Add status marker\nai-checked] + F --> K[Update contact/tickets as allowed] + J --> K + K --> L[updated_on advances\ncontact leaves immediate batch] + L --> M[Later passes use staleness + tags\nto route next work] + M --> N[Later worker may remove status tag\ne.g. ai-checked] + N --> B +``` + +### Diagram 2 — Tag/status re-entry lifecycle + +```mermaid +stateDiagram-v2 + [*] --> Eligible: updated_on older than cutoff\nand tag filters match + Eligible --> InCurrentPass: newest-stale-first selection + InCurrentPass --> Routed: worker adds route tag\nspam/promotions/etc. + InCurrentPass --> Checked: no route yet\nadd ai-checked + Routed --> CoolingDown: tag update bumps updated_on + Checked --> CoolingDown: status tag update bumps updated_on + CoolingDown --> HeldOut: too fresh for current cutoff + HeldOut --> Eligible: time passes / becomes stale again + Checked --> DownstreamQueued: later worker selects ai-checked\nor another status tag + DownstreamQueued --> ReEntered: worker removes ai-checked\nafter consuming status marker + ReEntered --> Eligible: tag filters match next pass +``` + +--- + +## Future queue/tag orchestration pattern + +The long-term architecture should treat tags as explicit work queues, for example: + +- `needs-summary` -> summary worker processes recent communications, then removes tag +- `needs-enrichment:*` -> enrichment workers (social/entity/profile data) +- `needs-nmi-mapping` -> account-linkage worker for NMI mapping tasks +- `special-handling:*` -> SLA/VIP/high-risk handling workflows + +This lets one loop classify and route, while specialized workers execute downstream tasks. + +--- + +## Proposed implementation phases + +### Phase 1 (implemented tracer) +- Add `loop-run` command that performs a **per-contact review loop** in dry-run mode: + - select candidates + - run FILTER-1 + - fetch recent tickets/messages + - call a profile-defined FILTER-2 command + - print proposed runner-owned tag updates + +### Phase 2 +- Replace no-op FILTER-2 with a classifier/mutator worker. +- Use FILTER-2 to update contacts/tickets directly through helper CLI calls. +- Keep runner-owned fallback `ai-checked` and RTAG removal semantics. + +### Phase 3 +- Add queue tags + worker handoff contracts (`needs-summary`, `needs-nmi-mapping`, etc.). + +### Phase 4 +- Integrate external systems (NMI mapping tables, obsidian profile updates, comm summaries, etc.) with strict scoped guards. + +--- + +## Practical run pattern (current CLI) + +Dry-run stale-first loop tracer: + +```bash +./redmine_contacts.py \ + --base-url http://192.168.50.170 \ + --project customer-service \ + loop-run \ + --profile profiles/contact-loop/noop-ai-checked.json \ + --source cache \ + --updated-before 2026-06-01 \ + --without-tag ai-checked \ + --max-contacts 10 \ + --order newest-updated-first +``` + +Apply runner-owned fallback/default tags after reviewing output: + +```bash +./redmine_contacts.py \ + --base-url http://192.168.50.170 \ + --project customer-service \ + loop-run \ + --profile profiles/contact-loop/noop-ai-checked.json \ + --source cache \ + --updated-before 2026-06-01 \ + --without-tag ai-checked \ + --max-contacts 10 \ + --order newest-updated-first \ + --apply +``` + +Older dry-run batch-update skeleton: + +```bash +./redmine_contacts.py \ + --base-url http://192.168.50.170 \ + --project customer-service \ + batch-update \ + --source cache \ + --updated-before 2026-06-01 \ + --with-tag customer \ + --max-contacts 20 \ + --order newest-updated-first \ + --set background=ai-checked +``` + +> Replace payload fields/tags only after reviewing dry-run output. diff --git a/docs/decomposition_pilot_contact_update_proposals_2026-07-09.json b/docs/decomposition_pilot_contact_update_proposals_2026-07-09.json new file mode 100644 index 0000000..e62587e --- /dev/null +++ b/docs/decomposition_pilot_contact_update_proposals_2026-07-09.json @@ -0,0 +1,53 @@ +{ + "generated_at": "2026-07-09", + "status": "proposal-only-not-applied", + "notes": "Generated by ticket decomposition pilot. Do not apply blindly; first decide Redmine custom field/background convention for Onyx article links.", + "proposals": [ + { + "redmine_issue_id": 41424, + "contact_id": 4470, + "email": "support@repackify.com", + "entity_note": "/home/jason/lilaBuild/vaults/obsidian-ldr/entities/organizations/repackify.md", + "onyx_url": "https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md", + "add_tags": [ + "maybe-customer", + "partnership-opportunity", + "buy-from-us", + "gaylords" + ], + "confidence": "medium", + "rationale": "Inbound request to buy used Gaylord boxes from LDR; not generic spam, but not confirmed customer." + }, + { + "redmine_issue_id": 41448, + "contact_id": 3919, + "email": "sales@u2wear.com", + "entity_note": "/home/jason/lilaBuild/vaults/obsidian-ldr/entities/organizations/u2wear.md", + "onyx_url": "https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/u2wear.md", + "add_tags": [ + "Customer", + "returns-management", + "fulfillment", + "elora", + "u2wear" + ], + "confidence": "high", + "rationale": "Jason confirmed sales/info@u2wear.com are customers; ticket references a previously discussed fulfillment and returns workflow." + }, + { + "redmine_issue_id": 41504, + "contact_id": 4476, + "email": "justine.micks@snugglepuppy.com", + "entity_note": "/home/jason/lilaBuild/vaults/obsidian-ldr/entities/organizations/snuggle-puppy.md", + "onyx_url": "https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/snuggle-puppy.md", + "add_tags": [ + "maybe-customer", + "returns-management", + "fba-returns", + "pricing" + ], + "confidence": "high", + "rationale": "Prospect/customer inquiry about warehousing and FBA returns; exposed returns-pricing/rate-sheet collateral gap." + } + ] +} diff --git a/docs/karpathywiki_plugin_inspection.md b/docs/karpathywiki_plugin_inspection.md new file mode 100644 index 0000000..4c482be --- /dev/null +++ b/docs/karpathywiki_plugin_inspection.md @@ -0,0 +1,290 @@ +# KarpathyWiki Obsidian Plugin Inspection + +**Date:** 2026-07-09 +**Plugin:** `karpathywiki` / `green-dalii/obsidian-llm-wiki` +**Repo inspected:** `https://github.com/green-dalii/obsidian-llm-wiki` +**Fresh clone verification:** commit `0979d50` +**License:** Apache-2.0 +**Version observed:** 1.24.0 + +## Why inspected + +Jason observed that the Obsidian Karpathy LLM Wiki plugin is doing much of what the Redmine ticket decomposition process should do: entity/concept extraction, knowledge graph page generation, links, indexes, and conversational query. The question was whether we can trigger it headlessly/CLI or mirror its logic. + +## Repository / structure findings + +The plugin is a compact TypeScript Obsidian plugin. + +Important files: + +```text +src/main.ts +src/wiki/wiki-engine.ts +src/wiki/source-analyzer.ts +src/wiki/page-factory.ts +src/wiki/prompts/ingestion.ts +src/wiki/prompts/generation.ts +src/wiki/prompts/merge.ts +src/wiki/prompts/lint.ts +src/wiki/query-engine/ +src/core/ +``` + +Package metadata: + +```json +{ + "name": "karpathywiki", + "version": "1.24.0", + "main": "main.js" +} +``` + +Fresh clone verification confirmed `package.json` has no `bin` entry: + +```text +PACKAGE {'name': 'karpathywiki', 'version': '1.24.0', 'main': 'main.js', 'bin': None, 'license': 'Apache-2.0'} +``` + +There is no package `bin` entry and no obvious standalone CLI exposed by the project. + +## Obsidian command entry points + +`src/main.ts` registers Obsidian commands: + +```text +ingest-source +ingest-folder +ingest-multiple-files +query-wiki +lint-wiki +regenerate-index +cancel-ingestion +ingest-active-file +view-ingestion-history +recreate-welcome-note +``` + +These are Obsidian command palette commands, not shell commands. + +If Obsidian is running, it may be possible to trigger commands through Obsidian/Advanced URI style automation, but that would still depend on Obsidian being open and the plugin being loaded. It is not a true headless pipeline. + +## Core engine dependency + +The core class is `WikiEngine` in `src/wiki/wiki-engine.ts`. + +It is separate from the plugin UI, but its constructor requires Obsidian types and APIs: + +```ts +constructor( + app: App, + settings: LLMWikiSettings, + getLLMClient: () => LLMClient | null, + schemaManager: SchemaManager, + onFileWrite?: (path: string) => void, + onProgress?: (message: string) => void, + onDone?: (report: IngestReport) => void +) +``` + +It uses: + +- `app.vault.read` +- `app.vault.getMarkdownFiles` +- `app.vault.process` +- `app.vault.create` +- `app.vault.getAbstractFileByPath` +- `app.metadataCache.getFileCache` +- `TFile`, `TFolder`, `Notice`, `normalizePath` + +Therefore, the engine cannot be called directly from Node/Python CLI without either: + +1. running inside Obsidian, or +2. writing an adapter/shim that emulates enough of Obsidian's `App/Vault/metadataCache/TFile` behavior. + +## Portable logic worth mirroring + +The useful logic is not locked to Obsidian conceptually. It can be mirrored or adapted: + +### Source analysis + +`src/wiki/source-analyzer.ts`: + +- reads source content +- checks blank/duplicate content +- prompts LLM for structured JSON +- extracts entities and concepts +- includes aliases +- collects verbatim quotes / provenance +- identifies contradictions, related pages, key points +- normalizes irregular LLM JSON + +The core ingestion prompt is in `src/wiki/prompts/ingestion.ts`. + +Notable extraction output shape: + +```json +{ + "source_title": "...", + "summary": "...", + "entities": [ + { + "name": "...", + "type": "person|organization|project|product|event|place|other", + "aliases": [], + "summary": "...", + "mentions_in_source": [], + "mentions_with_provenance": [], + "related_entities": [], + "related_concepts": [] + } + ], + "concepts": [], + "contradictions": [], + "related_pages": [], + "key_points": [] +} +``` + +### Page generation + +`src/wiki/page-factory.ts`: + +- slugifies entities/concepts +- checks existing pages +- resolves duplicates via slug/alias/LLM semantic match +- generates entity/concept pages +- merges into existing pages +- protects reviewed pages +- corrects polluted wikilinks +- injects provenance/mentions + +### Prompts + +`src/wiki/prompts/generation.ts` includes high-value generation prompts for: + +- entity pages +- concept pages +- source summary pages +- preserving reviewed pages + +### Query / graph retrieval + +The README claims zero-embedding graph retrieval via Personalized PageRank over the wiki link graph. Source references in `WikiEngine` show: + +```ts +buildGraphFromContent(...) +getOrBuildGraph(...) +``` + +and query pipeline files exist under: + +```text +src/wiki/query-engine/pipeline/ +``` + +This is useful design inspiration, but not required for the first Redmine decomposition tool because gbrain already supplies search/indexing and the vault graph itself supplies wikilinks. + +## Recommendation + +Do **not** depend on Obsidian running for Redmine ticket decomposition. + +Instead: + +1. Mirror the plugin's source-analysis and page-generation logic in our own CLI-oriented tool. +2. Reuse its prompt structure and output schema ideas. +3. Write directly to the filesystem vault, but outside the plugin-owned `wiki/` tree by default. +4. Use gbrain for indexing/search after files are written. +5. Keep Redmine contact updates as a separate explicit apply phase. +6. Preserve compatibility with Obsidian/KarpathyWiki by using normal Markdown, wikilinks, and frontmatter fields the plugin understands where practical. + +Compatibility rule: the plugin owns `wiki/entities`, `wiki/concepts`, `wiki/sources`, `wiki/index.md`, and `wiki/log.md`. The headless decomposer should default to LDR-owned paths like `Timeline/`, `entities/organizations/`, and `Processes/Redmine/`, linking to `wiki/...` pages when useful but not editing them. + +This gives us the same broad effect without requiring a GUI Obsidian session and without competing with the existing plugin state. + +## Possible implementation paths + +### Path A — Thin TypeScript headless port + +Build a Node/TypeScript CLI in `/home/jason/projects/redmine` or a sibling project that imports/copied logic from the plugin and supplies a fake Obsidian adapter. + +Pros: + +- Closer to upstream code. +- Easier to vendor future prompt improvements. +- Can reuse zod/AI SDK provider setup. + +Cons: + +- Need to emulate enough of Obsidian `Vault`, `TFile`, metadata cache, and settings. +- Adds Node/TypeScript toolchain complexity to Redmine workflow. + +### Path B — Python mirror using existing Redmine tooling + +Implement the same conceptual pipeline in Python alongside `redmine_tickets.py`: + +- source normalization +- LLM JSON extraction +- entity/concept dedup via filesystem search + slug rules +- page generation templates +- timeline/entity writers +- gbrain sync hooks + +Pros: + +- Fits current Redmine tooling. +- Easier to integrate with ticket cache and contact updates. +- No Obsidian runtime needed. + +Cons: + +- We do not directly reuse TypeScript implementation. +- Need to maintain our own prompt/schema logic. + +### Path C — Obsidian-driven automation + +Use Obsidian command automation / Advanced URI to trigger plugin commands. + +Pros: + +- Minimal code if it works. + +Cons: + +- Requires Obsidian running. +- Harder to make deterministic/headless/cron-safe. +- Harder to feed Redmine tickets and capture structured contact-update proposals. + +## Best fit for LDR Redmine decomposition + +Path B is the best first implementation path. + +Reasons: + +- Redmine tickets are already fetched/cached in Python. +- We need Redmine contact updates and safety gates. +- We need Onyx URLs and gbrain sync hooks. +- We need headless/cron/agent execution. +- We already proved manual note generation format in the decomposition pilot. + +Use the plugin as a reference for: + +- `src/wiki/prompts/ingestion.ts` — source analysis prompt and entity/concept output shape +- `src/wiki/prompts/generation.ts` — entity/concept/source page generation prompts +- `src/wiki/prompts/merge.ts` — classify-then-route merge strategy for updating existing pages +- `src/wiki/prompts/lint.ts` — wiki quality checks and contradiction review ideas +- `src/wiki/source-analyzer.ts` — JSON normalization, duplicate/blank source handling, provenance mentions +- `src/wiki/page-factory.ts` — slug/alias resolution, cross-type collision handling, merge-vs-append routing +- `src/wiki/query-engine/` — graph retrieval ideas for later, not first pass + +Do not try to invoke it via Obsidian as the primary production pipeline; use the source as a design reference for a headless Python mirror. + +## Immediate next action + +Update `ticket_decomposition_knowledge_ingestion_plan.md` so Phase 2 says: + +- use KarpathyWiki plugin prompt/schema as reference +- implement a Python headless mirror first +- consider TypeScript adapter later only if Python mirror becomes too costly + +Then implement a `decompose --dry-run` engine using the pilot tickets as fixtures. diff --git a/docs/known_spam_fast_close_plan.md b/docs/known_spam_fast_close_plan.md new file mode 100644 index 0000000..208648e --- /dev/null +++ b/docs/known_spam_fast_close_plan.md @@ -0,0 +1,623 @@ +# Known-Spam Fast Close Implementation Plan + +**Date:** 2026-07-09 +**Project:** `/home/jason/projects/redmine` +**Primary tool:** `redmine_tickets.py` +**Target Redmine project:** `customer-service` +**Production base URL:** `https://rm.ldrprep.com` +**Agent write attribution:** `rebot` / Harry Botter + +## Goal + +Build a reliable two-prong production spam-cleanup workflow: + +1. **Slow/reviewed path** — the current classifier-driven `sync → spam-check → close-if-spam` flow, used for discovering and validating new spam patterns. +2. **Fast known-profile path** — a verified-profile bulk closer that quickly locates all open `customer-service` tickets matching known spam sender/profile rules and closes them in one bounded, idempotent production operation. + +The immediate business goal is to close known promotional/spam clutter quickly without risking customer tickets. + +--- + +## Current Reality + +The slow path already works in production: + +- `redmine_tickets.py sync` caches enriched tickets using `include=journals,helpdesk`. +- `spam-check` categorizes tickets into classes such as: + - `spam-promotional` + - `automated-operational` + - `unknown-review` +- `close-if-spam` closes reviewed `spam-promotional` tickets using `--impersonate-user rebot`. +- Writes are verified by API as `status=Closed` and close-note author `Harry B.`. +- Idempotency is recorded via local action markers. +- Customer-contact guard is active: before spam-close, the tool fetches the Helpdesk contact and skips if the contact has tag `Customer`. + +Validated production cleanup so far: + +- First production pass closed 20 reviewed promotional-junk tickets. +- Current 200-ticket cache has a reviewed 27-ticket candidate set pending apply. +- Customer pre-tagging has been performed for confirmed customers including Cyberstorm, Esire-family contacts, Shinkiya, Bohoflow, BavariaTrachten, Solride, and u2wear. + +Known safety rule: + +> Absence of `Customer` does **not** imply spam. Presence of `Customer` blocks automated spam closure. + +--- + +## Desired Two-Prong Architecture + +```text + ┌────────────────────────────┐ + │ redmine_tickets.py substrate│ + │ sync/cache/helpdesk/actions │ + └──────────────┬─────────────┘ + │ + ┌───────────────────────────┴───────────────────────────┐ + │ │ + Slow / reviewed path Fast / known profile path + discover new patterns close already-verified spam + │ │ + sync → spam-check → review → close-if-spam known-spam-profile scan → dry-run → apply + │ │ + produces new safe profiles uses frozen allowlisted rules +``` + +### Slow path responsibilities + +Use for: + +- New sender types. +- Ambiguous business/vendor/customer cases. +- Classifier development. +- False-positive discovery. +- Contact/customer enrichment. + +### Fast path responsibilities + +Use for: + +- Previously reviewed spam senders/domains/campaigns. +- High-volume obvious junk such as known promo newsletters and sales blasts. +- Rapid production cleanup once a profile has been approved. + +The fast path must not rediscover intent. It should only act on a curated profile registry. + +--- + +## Fast Known-Spam Profile Design + +### Risk Model Update — Closed Is Recoverable + +Jason clarified an important Redmine/Helpdesk behavior: + +- Closed tickets are not deleted. +- If the other party sends a new message to a closed ticket, Redmine reopens it and status changes to `New Message`. + +That means accidental closure is usually recoverable if the other party still cares enough to reply. The larger risk is **re-closing a ticket after the sender already reopened it**, because that means the system is ignoring renewed human intent. + +Policy consequence: + +1. If a ticket is clearly `spam-promotional` and stale, especially `updated_on` older than 30 days, it is reasonable to close automatically once other safety gates pass. +2. If a ticket has evidence that it was previously closed and later reopened by a new Helpdesk message, do **not** close it automatically, even if it matches spam. +3. Instead, escalate it for human attention: bump priority one level if possible and optionally set a due date 1–3 days out. + +### Terminology + +A **known spam profile** is a conservative, reviewed rule bundle that identifies tickets we are comfortable closing automatically, subject to safety gates. + +Example profiles: + +```yaml +profiles: + globalindustrial-promos: + senders: + - reply@globalindustrial.com + subject_patterns: + - "(?i)\\b(clearance|sale|save|% off|bestsellers|fans|blowers|cooling|storage|power tools|kee safety)\\b" + category: spam-promotional + confidence: high + notes: "Repeated promotional vendor emails; reviewed in LAN + prod broom passes." + + vistaprint-promos: + senders: + - noreply@mail.vistaprint.com + subject_patterns: + - "(?i)\\b(sale|save|% off|email exclusive|ends|qualifying orders|favorite sale|customer faves)\\b" + category: spam-promotional + confidence: high + + qnap-newsletters: + senders: + - enews-noreply@qnap.com + subject_patterns: + - "(?i)\\b(qnap|photodirector|nas|security advisory|product of the year|jc-star|file backup)\\b" + category: spam-promotional + confidence: high + + business-journals-promos: + senders: + - replycincinnati@engaged.bizjournals.com + - replycincinnati@partner.bizjournals.com + - acollett@bizjournals.com + subject_patterns: + - "(?i)\\b(events|nominations|business summit|brand next to key business lists|chambercomp)\\b" + category: spam-promotional + confidence: high +``` + +### Safety gates always applied before close + +Fast path must still apply these universal gates: + +1. Ticket must be open. +2. Ticket must be in `customer-service` unless explicitly overridden. +3. Ticket Helpdesk contact must be fetched. +4. Contact with tag `Customer` must be skipped. +5. Optional: contact with tag `maybe-customer` should be skipped by default unless `--allow-maybe-customer` is explicitly passed. +6. Ticket must not already have local `known-spam-close` / `close-if-spam` marker. +7. If the ticket was previously closed and then reopened by a new Helpdesk message, skip spam-close and escalate instead. +8. Apply writes must impersonate `rebot`. +9. Apply must produce a verification report. + +### Reopened-ticket escalation behavior + +Add a helper such as `ticket_was_reopened_after_close(issue)` that inspects journals/details for evidence of a transition from Closed/Resolved into `New Message` or another open status after a Helpdesk reply. + +When true, do not close as spam. Instead, planned behavior should be: + +```json +{ + "priority_id": "one level higher if known/mappable", + "due_date": "today + 1 to 3 days", + "notes": "Matched spam profile, but ticket appears to have been reopened after closure. Escalating for human review instead of auto-closing." +} +``` + +This should be its own idempotent action marker, e.g. `spam-reopen-escalation`, so the ticket is not bumped repeatedly. + +--- + +## Files to Modify + +- Modify: `/home/jason/projects/redmine/redmine_tickets.py` +- Modify: `/home/jason/projects/redmine/tests/test_redmine_tickets.py` +- Create: `/home/jason/projects/redmine/profiles/known-spam/customer-service.json` +- Create: `/home/jason/projects/redmine/docs/rebot_legacy_spam_rules.md` +- Create/Update: `/home/jason/projects/redmine/docs/known_spam_fast_close_plan.md` +- Update: `/home/jason/projects/redmine/HANDOFF.md` +- Update: `/home/jason/projects/redmine/docs/ticket_spam_cleanup_handoff_2026-07-09.md` +- Optional later: `/home/jason/projects/redmine/docs/known_spam_profile_registry.md` + +--- + +## Proposed CLI + +### Dry-run known-spam scan + +```bash +cd /home/jason/projects/redmine + +./redmine_tickets.py \ + --base-url https://rm.ldrprep.com \ + --project customer-service \ + --cache-dir .cache/redmine_tickets_prod_known_spam_$(date +%Y%m%dT%H%M%SZ) \ + known-spam-scan \ + --profile profiles/known-spam/customer-service.yaml \ + --status-id open \ + --max-tickets 2000 +``` + +Expected output: + +```text +Known-spam scan complete: seen=1732 matched=214 skipped_customer=3 skipped_maybe_customer=5 planned=206 apply=False +profile=globalindustrial-promos issue=... sender=... subject=... +... +Dry run only. Re-run known-spam-close --apply using this cache/profile after review. +``` + +### Apply known-spam close + +```bash +./redmine_tickets.py \ + --base-url https://rm.ldrprep.com \ + --project customer-service \ + --cache-dir .cache/redmine_tickets_prod_known_spam_ \ + known-spam-close \ + --profile profiles/known-spam/customer-service.yaml \ + --close-status-id 5 \ + --impersonate-user rebot \ + --apply +``` + +Expected output: + +```text +Known-spam close complete: planned=206 closed=206 skipped_customer=0 skipped_existing_marker=0 apply=True +``` + +### Verify apply + +```bash +./redmine_tickets.py \ + --base-url https://rm.ldrprep.com \ + --project customer-service \ + --cache-dir .cache/redmine_tickets_prod_known_spam_ \ + verify-known-spam-close +``` + +Expected output: + +```text +Verify known-spam close: checked=206 closed=206 author_harry_b=206 failures=0 +``` + +--- + +## Implementation Tasks + +### Task 1: Add profile loader + +**Objective:** Load known-spam YAML profiles safely. + +**Files:** + +- Modify: `redmine_tickets.py` +- Test: `tests/test_redmine_tickets.py` +- Create: `profiles/known-spam/customer-service.yaml` + +**Implementation notes:** + +- Use stdlib if PyYAML is not guaranteed? Existing environment may support YAML via project deps, but avoid installing packages. +- Safer first version: store profile as JSON instead of YAML if dependency uncertainty matters. +- Recommended file if avoiding deps: + +```text +profiles/known-spam/customer-service.json +``` + +**Test cases:** + +- valid profile loads +- missing required fields fails +- invalid regex fails with helpful error + +**Acceptance:** + +```bash +python3 -m unittest discover -s tests -p 'test_redmine_tickets.py' -v +python3 -m py_compile redmine_tickets.py tests/test_redmine_tickets.py redmine_contacts.py +``` + +--- + +### Task 2: Implement known-profile matcher + +**Objective:** Determine whether a cached ticket matches a known spam profile. + +**Files:** + +- Modify: `redmine_tickets.py` +- Test: `tests/test_redmine_tickets.py` + +**Rules:** + +A ticket matches when: + +- sender matches one of profile senders/domains, and +- subject/body pattern matches if provided, and +- category is `spam-promotional`, and +- it does not match profile exclusions. + +**Test examples:** + +- `reply@globalindustrial.com` + `Clearance Sale: Top Steel Storage And Power Tools` matches `globalindustrial-promos`. +- `reply@globalindustrial.com` + a real invoice/order/customer thread does **not** match if exclusion pattern present. +- `enews-noreply@qnap.com` + `QNAP Security Advisory` matches only if we decide QNAP security advisories are clutter; this profile should be explicit so no one later acts surprised. + +--- + +### Task 3: Add customer/maybe-customer universal skip gates + +**Objective:** Ensure known-spam fast path has the same safety guard as slow path. + +**Files:** + +- Modify: `redmine_tickets.py` +- Test: `tests/test_redmine_tickets.py` + +**Tests:** + +- contact tagged `Customer` is skipped even if ticket matches known profile. +- contact tagged `maybe-customer` is skipped by default. +- `--allow-maybe-customer` permits maybe-customer if explicitly requested. +- absence of tags does not block known spam profile matching. + +**Acceptance:** + +Fast path must never bypass customer safety. + +--- + +### Task 3B: Detect reopened-after-close tickets and escalate instead of closing + +**Objective:** Prevent the cleaner from repeatedly closing a ticket after the other party has reopened it with a new message. + +**Files:** + +- Modify: `redmine_tickets.py` +- Test: `tests/test_redmine_tickets.py` + +**Implementation notes:** + +- Add `ticket_was_reopened_after_close(issue)`. +- Inspect `issue['journals']` and their `details` for status changes. +- Treat evidence of Closed/Resolved → New Message/open status as reopened-after-close. +- If exact status IDs are easier than names, map known status names/IDs from existing Redmine data before hardcoding. +- Do not rely on current status alone; the point is historical reopening. + +**Tests:** + +- Journal with status change Closed → New Message returns true. +- Journal with only New → Closed returns false. +- Known-spam close skips reopened-after-close tickets. +- Apply path records `spam-reopen-escalation` instead of `known-spam-close`. + +**Escalation behavior:** + +- Dry-run reports the escalation plan. +- Apply should update priority/due date only when explicit `--apply-escalations` or equivalent is provided; do not mix escalation writes into spam-close apply without a clear flag. + +--- + +### Task 4: Add `known-spam-scan` command + +**Objective:** Sync/search open tickets and report matches without closing. + +**Files:** + +- Modify: `redmine_tickets.py` +- Test: `tests/test_redmine_tickets.py` + +**Behavior:** + +- Fetch open `customer-service` tickets with `include=journals,helpdesk`. +- Cache ticket payloads. +- Evaluate known spam profiles. +- Fetch contacts for matched tickets only. +- Report: + - matched profile + - issue ID + - sender + - subject + - contact ID + - age / `updated_on` + - skipped reason if skipped +- Mark tickets older than 30 days and matching a verified `spam-promotional` profile as strong fast-close candidates. +- Newer tickets may still be candidates, but the report should separate them from stale 30+ day candidates. +- Record local action marker: + - `known-spam-scan = matched|matched-stale|skipped-customer|skipped-maybe-customer|skipped-reopened|not-matched` + +**Dry-run only:** This command should never write Redmine. + +--- + +### Task 5: Add `known-spam-close` command + +**Objective:** Close the previously scanned/planned known-spam tickets. + +**Files:** + +- Modify: `redmine_tickets.py` +- Test: `tests/test_redmine_tickets.py` + +**Behavior:** + +- Reads cache + scan markers. +- Only closes tickets with marker `known-spam-scan = matched`. +- Re-checks contact tags before close. Do not rely only on stale scan state. +- Updates ticket with: + +```json +{ + "status_id": 5, + "notes": "Closed as known promotional spam by redmine_tickets.py known-spam-close. Profile: ." +} +``` + +- Writes as `rebot`. +- Records: + - `known-spam-close = closed` + - profile name + - close payload + - verification pending + +**Tests:** + +- apply closes only matched markers +- second apply closes zero +- customer tag recheck blocks closure even if scan marker was matched +- impersonation defaults to `rebot` + +--- + +### Task 6: Add verification command/report + +**Objective:** Verify production writes after apply. + +**Files:** + +- Modify: `redmine_tickets.py` +- Test: `tests/test_redmine_tickets.py` + +**Behavior:** + +For every `known-spam-close = closed` marker: + +- fetch issue with `include=journals,helpdesk` +- verify `status.name == Closed` +- verify close note exists +- verify note author is `Harry B.` +- print failures clearly + +**Output:** + +```text +Verify known-spam close: checked=206 closed=206 author_harry_b=206 failures=0 +``` + +--- + +### Task 7: Seed first known-spam profile registry from verified production data + +**Objective:** Convert already-validated spam senders into conservative profiles. + +**Initial profile groups:** + +- Global Industrial promos +- Vistaprint promos +- QNAP newsletters/promos +- Business Journals / BizJournals promo/events +- TikTok Shop seller promos +- Popmenu restaurant promos +- Samsung promo newsletters +- Cintas mop/product promos +- Wings and Rings restaurant promos +- SellerEssentials promos + +**Do not include yet without explicit decision:** + +- `sales@supermarketdistribution.com` / stock/MOQ updates — may be vendor-ish; review first. +- QNAP security advisories — likely clutter, but security language can look operational; make it explicit in profile if included. +- Any sender with `Customer` or `maybe-customer` contacts. + +--- + +### Task 8: Production rollout procedure + +**Objective:** Define exact operator sequence for today. + +**Step A: Commit or checkpoint local code** + +Before production bulk close, ensure current code/tests/docs are checkpointed in git or at least `git diff` saved. + +```bash +cd /home/jason/projects/redmine +git status --short +python3 -m unittest discover -s tests -p 'test_redmine_tickets.py' -v +python3 -m py_compile redmine_tickets.py tests/test_redmine_tickets.py redmine_contacts.py +``` + +**Step B: First fast-path dry run** + +```bash +./redmine_tickets.py \ + --base-url https://rm.ldrprep.com \ + --project customer-service \ + --cache-dir .cache/redmine_tickets_prod_known_spam_$(date +%Y%m%dT%H%M%SZ) \ + known-spam-scan \ + --profile profiles/known-spam/customer-service.json \ + --status-id open \ + --max-tickets 2000 +``` + +**Step C: Human review** + +Review summary counts and sample candidate rows. If any candidate looks like customer/prospect/operational context, do not apply; add exclusion/profile refinement. + +**Step D: Apply** + +```bash +./redmine_tickets.py \ + --base-url https://rm.ldrprep.com \ + --project customer-service \ + --cache-dir \ + known-spam-close \ + --profile profiles/known-spam/customer-service.json \ + --close-status-id 5 \ + --impersonate-user rebot \ + --apply +``` + +**Step E: Verify** + +```bash +./redmine_tickets.py \ + --base-url https://rm.ldrprep.com \ + --project customer-service \ + --cache-dir \ + verify-known-spam-close +``` + +**Step F: Handoff update** + +Update: + +- `HANDOFF.md` +- `docs/ticket_spam_cleanup_handoff_2026-07-09.md` +- optionally `docs/known_spam_profile_registry.md` + +--- + +## Acceptance Criteria + +The implementation is acceptable when: + +1. Slow path still works and tests pass. +2. Fast path can dry-run all open production customer-service tickets without writes. +3. Fast path can report exact match profile per candidate. +4. Fast path skips contacts tagged `Customer`. +5. Fast path skips contacts tagged `maybe-customer` by default. +6. Fast path writes only when `--apply` is passed. +7. Fast path writes as `rebot`. +8. Second apply is idempotent. +9. Verification command proves status closed + Harry B. note author. +10. Profile registry is conservative and versioned in the repo. + +--- + +## Short-Term Recommendation for Today + +Before building the full fast-profile command, there is still an immediate safe action available: + +1. Apply the currently reviewed 27-ticket production close set from: + +```text +.cache/redmine_tickets_prod_200_promotional_eval_1783612885 +``` + +2. Verify closed / Harry B. / idempotency. +3. Then implement the fast known-spam profile path above. + +This keeps the broom moving while the turbo-broom gets brakes installed. Very advanced broom science. + +--- + +## Legacy Rebot Rule Extraction Status + +Completed source inspection of old Rebot spam/routing logic and captured it in: + +```text +docs/rebot_legacy_spam_rules.md +profiles/known-spam/customer-service.json +``` + +Validated profile seed: + +```text +JSON_OK +profiles 7 +globalindustrial-promos spam-promotional 1 sender regex / 1 subject regex +vistaprint-promos spam-promotional 1 / 1 +qnap-newsletters spam-promotional 1 / 1 +business-journals-promos spam-promotional 3 / 1 +legacy-rebot-promotional-senders spam-promotional 16 / 0 +legacy-rebot-operational-clutter automated-operational 4 / 10 +legacy-rebot-receipts-routing-reference routing-reference 4 / 1 +``` + +Important split: + +- `legacy-rebot-promotional-senders` can be considered for fast close after dry-run review. +- `legacy-rebot-operational-clutter` preserves old Rebot close rules but should remain report-only until Jason explicitly approves operational clutter closing. +- `legacy-rebot-receipts-routing-reference` should not be closed as spam; it is future routing/reference material. diff --git a/docs/raw-notes/Untitled.md b/docs/raw-notes/Untitled.md new file mode 100644 index 0000000..6e5ae14 --- /dev/null +++ b/docs/raw-notes/Untitled.md @@ -0,0 +1,33 @@ +@everyone I think we should do some tune-ups and checks on the "advanced" Redmine CLI located in ~/projects/redmine/ that is pointed to by ~/projects/redmine/bin/rtl + +We kind of had a history with this tool: + +- Initially, it kicked ass and was very fast and powerful, but it lacked a nice TUI +- We attempted to build a TUI using curses, and found that had severe limitations on the UI; the functionality was powerful, but the UI/UX sucked ass. +- We tried rebuilding a TUI using more modern libraries and apps as reference points, and somehow totally lost the plot. Not only did we end up with incomplete features agents swore they had completed, we ended up regressing the codebase terribly, to the point even the original CLI features became unreliable, didn't do what the docs/specs said they did, and all kinds of problems. Nobody involved was able to identify a straight path to fixing that. +- We only recently seem to have restored the basic CLI functionality of the tool. From what I understand, the majority of the problem(s) turned out to be: + +a) Unreachable code in CLI argument parsing +b) Something to do with order of arguments supplied before or after others would cause silent failures / weird behavior +c) Unverified, but we've seen evidence there is a bug that can leave the CACHE in an unusable state if the script is terminated before it finishes a sync, and the code is seems to be unable to recover from or _recognize_ that problem. On subsequent runs, it will act like there are no problems, it just doesn't give correct output if this happens. + +### As far as I know, at this point, the following things works correctly: + +- `jason@onyx > ./bin/rtl sync --remote-status-id '*' --remote-updated-since 30d --run-log --run-log-keep 20` run every 5-10 minutes to keep the local cache updated +- the `walk` verb, except it does not have a help menu for what commands it supports (n, p, q, r) +- view verb, but unclear if it displays local cache or fetches live +- `materialize-queues` seems to work +- `contacts-sync` + - Need to make sure this _does not_ delete old contacts if run, for instance, with something like `--nax0contacts 50` +- + +### Minor issues +- `draft` verb: + - doesn't have mechanism to _show_ a drafted change + - should have simple way to clear old drafts or something +- + +### Definitely not working: + +- triage verb +- \ No newline at end of file diff --git a/docs/raw-notes/features-for-redmine_tickets-sync.md b/docs/raw-notes/features-for-redmine_tickets-sync.md new file mode 100644 index 0000000..219c2ab --- /dev/null +++ b/docs/raw-notes/features-for-redmine_tickets-sync.md @@ -0,0 +1,332 @@ +# Features / Roadmap: power-user Redmine client (`redmine_tickets` plane) + +> [!todo] Keep this file as the **index**: status, priority, and short summaries. +> Long implementation detail lives in `docs/raw-notes/features/*` and is linked from here. +> Update checkboxes when work lands; do not re-expand full specs into this file. + +**Last status refresh:** 2026-07-15 (post browse-loop + attachments + entities + contacts bridge) + +--- + +## Product intent (what this is) + +A **scriptable power-user Redmine client** for humans and agents: + +- durable local ticket (+ contact) cache as the **ops workbench** +- property-based local search, saved filters, filesystem queues +- CLI / TUI-lite read path (Markdown + `glow`/`bat`/`fzf`) then **queued** writes +- place to invent/test filters, tools, and agent loops without living in the web UI + +**Not** the system of record (Redmine is). +**Not** the vault/wiki (curated memory). +**Not** Qdrant (optional later semantic layer). + +First project focus: **`customer-service`**. + +| Env | Redmine | Cache | Wrapper | +|-----|---------|--------|---------| +| Prod | `https://rm.ldrprep.com` | `/home/jason/projects/redmine/cache/redmine_tickets` | `bin/rtl` | +| Dev / mutation testing | `http://192.168.50.170` | `/home/jason/projects/redmine/cache-dev/redmine_tickets` | `bin/rtd` | + +**Safety defaults** + +- Writes/applies prefer LAN; prod needs parent `--allow-prod-writes` +- `note` never emails; `customer-email` always emails (ugly confirm flag) +- Browse/actions **queue drafts** by default (apply later) +- Dual tools OK for now: tickets workbench vs full `redmine_contacts.py` CRM CLI, with bridges + +--- + +## Priority stack (current) + +| Priority | Area | Status | +|----------|------|--------| +| P0 | Durable incremental ticket sync + remote filters | **Done** | +| P0 | Cron for CS refresh (+ queue rematerialize) | **Running** (6h window) | +| P1 | Property-based local search (`--where`) | **Done** | +| P1 | Markdown view + walk | **Done** | +| P1 | Browse loop (fzf → pager → action menu) | **MVP done** (polish open) | +| P2 | Saved filters + symlink queues | **Done** | +| P2 | Queued notes / customer-email / set-fields | **Done** | +| P2 | Config wrappers (`--config`, `rtl`/`rtd`) | **Minimal done** | +| P3 | Local control tags + suggest + JSONL export | **Done** (LLM refine open) | +| P3 | Contacts mirror + contact-set + ticket-contact | **Done** (cache unify optional) | +| P4 | Attachments list/download/upload | **Done** | +| P4 | Entity extract (patterns) + NMI bridge CLI | **MVP done** | +| Later | Full curses TUI / nested config DSL / master layout | Deferred | +| Adjacent | Semantic index / vault decompose / OCR rename skill | Separate tracks | + +--- + +## Capability map (what you can run today) + +| Need | Command(s) | +|------|------------| +| Refresh tickets | `sync` · cron: `cron-scripts/sync-live-customer-service.sh` | +| Search cache | `search --where '…'` · `@saved` · `--control-tag` · `--has-attachments` | +| Read ticket | `view ID` · `walk` · **`browse`** (loop) | +| Internal note | `note` (draft → `draft apply`) | +| Customer email | `customer-email` + confirm flag | +| Fields | `set-fields` (`--status Closed`, etc.) · `meta-refresh` | +| Local labels | `control-tag add|suggest|export|…` | +| Contacts | `contacts-sync` · `contacts-search` · `contact-show` · `contact-set` · `ticket-contact --fetch` | +| Files | `attachments list|download|upload` | +| Tracking/order hints | `entities show|scan` · `entities nmi` · browse `m` | +| OCR scan rename | `ocr-rename --issue-id` / `--where '@scanned-document'` | +| Categories | `categories list [--refresh]` | +| Queues on disk | `materialize-queues --config queues.example.json` | + +Cookbook: **`docs/cli_power_user_client.md`** + +--- + +## P0 — Sync foundation + +**Detail:** [[features/sync-foundation]] + +- [x] Stable cache path (not per-run timestamped dirs) +- [x] Incremental full-fetch skip when `updated_on` unchanged +- [x] Server-side `updated_on` + `status_id` (remote flags) +- [x] Local post-filters retained +- [x] Sync progress + `--quiet` for cron +- [x] Cache never auto-deletes old ticket JSON +- [x] Run-stats ring buffer (`--run-log`) +- [x] Cron: live + dev CS scripts; **6h** remote window; rematerialize queues after sync +- [x] Optional contacts cron: `cron-scripts/sync-live-contacts.sh` +- [ ] Confirm long-term cadence (currently ~every few minutes) is still desired +- [x] Full fetch includes `attachments` metadata (not just journals/helpdesk) + +```bash +./bin/rtl sync --remote-status-id '*' --remote-updated-since 2h --run-log --quiet +``` + +--- + +## P1 — Local search + +**Detail:** [[features/local-search-and-cli-client]] · `redmine_ticket_query.py` +**Historical thin search:** [[../2026-07-14_localsearch_before_sanity]] + +- [x] JSON path filters + `includes()` / `since()` / AND/OR / parens +- [x] Nested helpdesk/contact paths +- [x] `--ids-only`, ordering, max caps +- [x] Table columns: assignee, contact, control tags +- [x] `--has-attachments` (cache metadata) +- [x] **No default status filter** — closed tickets appear if cached +- [ ] Optional default “open-ish” profile (if desired later) + +--- + +## P1 — Read path + browse loop + +**Detail:** [[features/local-search-and-cli-client]] · `redmine_ticket_view.py` · `redmine_ticket_browse.py` + +- [x] Ticket → Markdown (`view`) + MD cache under `markdown/` +- [x] Pagers: glow / bat / batcat / cat +- [x] `walk` (n/p/q) +- [x] **`browse` loop:** fzf (or numbered) → pager → action menu → list + - Actions: view again, queue note/private note, queue fields, control-tag, contact handoff, entities, print id + - Mutations **queue only** + - `--once`, `--pager`, filters same family as search +- [ ] Browse polish (user-noted “minor things”) — collect below under **Open polish** +- [ ] Full curses/TUI (linear-tui style) still **later**, not blocking ops + +```bash +./bin/rtl browse --where 'assigned_to IS NULL' --max-tickets 80 --pager glow +``` + +--- + +## P2 — Filters, queues, mutations + +**Detail:** [[features/local-search-and-cli-client]] · drafts in `redmine_ticket_drafts.py` + +- [x] `saved-filters/*.expr` + `@name` +- [x] `materialize-queues` symlink folders +- [x] `note` vs `customer-email` (hard separation; Helpdesk `email_note` path) +- [x] Draft list/cancel/apply; draft binds `base_url`; obvious `draft_id=` +- [x] `set-fields` + name→id (`meta-refresh`) +- [x] Parent `--allow-prod-writes` (not subparser-shadowed) +- [x] Minimal `--config` + `bin/rtl` / `bin/rtd` / `bin/qaa` +- [ ] Nested config defaults (`search.max-tickets` etc.) — [[features/config-files]] +- [ ] Note body from file + optional pandoc→Textile (steering list) + +```bash +./bin/rtl note --issue-id 41934 --body 'staff context' --private +./bin/rtl customer-email --issue-id 41934 --body '…' --i-understand-this-emails-the-customer +./bin/rtl --allow-prod-writes draft apply +``` + +--- + +## P3 — Control tags + +**Detail:** [[features/control-tags]] · export: `redmine_ticket_control_export.py` + +- [x] Local wipeable store under `control-tags/tags.json` +- [x] add / remove / list / show / for-issue / wipe +- [x] suggest (`--where` rules, optional `--apply` local) +- [x] `export` → JSONL for training / refine +- [ ] LLM refine loop (consume export, propose tags, human correct/incorrect) + +--- + +## P3 — Contacts + +**Detail:** [[features/contacts-sync-integration]] · `redmine_ticket_contacts.py` +**Full CRM still:** `redmine_contacts.py` (separate cache/CLI by design for now) + +- [x] Durable mirror: `/contacts/` +- [x] `contacts-sync` / `contacts-search` / `contact-show` +- [x] `contact-set` draft/apply (tags, company, names, `--set`) +- [x] `ticket-contact` bridge; **`--fetch` implies show+refresh** (bugfix) +- [x] Contact column in ticket search +- [ ] Optional unify on-disk layout with `redmine_contacts.py` (only if dual-cache pain grows) + +--- + +## P4 — Attachments + +**Detail:** [[features/attachments]] · `redmine_ticket_attachments.py` + +- [x] `attachments list` (cache-first, `--refresh`) +- [x] `attachments download` (one / `--all`) +- [x] `attachments upload --apply` (POST `/uploads.json` + issue `uploads`; no email) +- [x] OCR rename workflow CLI (`ocr-rename`) — [[features/ocr-rename]] + +--- + +## P4 — Entities + NMI + +**Detail:** [[features/entity-type-mapper]] · `redmine_ticket_entities.py` · `redmine_ticket_nmi.py` + +- [x] Pattern extract: UPS/USPS, order/SKU/account + LDR `item_no` +- [x] Shown in Markdown view +- [x] `entities show|scan` +- [x] `entities nmi` bridge to `mrscClient-php bin/nmi` (cs-brief / resolve) +- [x] Noise filters on medium-confidence `order_id` hits +- [ ] Richer NMI membership index (known order/item windows) if CLI alone isn’t enough +- [x] One-key NMI from browse menu (`m` → private note draft) + +--- + +## Open polish / small bugs (browse + ops) + +- [x] Browse: fzf header/keybinding discoverability +- [x] Browse: optional preview window in fzf (`--no-preview` to disable) +- [x] Browse: louder draft queue banner (`announce_draft`) +- [x] Browse: `m` → entities nmi brief → private note draft +- [x] Entity noise filters + LDR 9-digit `item_no` +- [x] Note `--body-file` (already) + `--to-textile` (pandoc) +- [x] `search`/`browse`/`walk` `--open-only` +- [x] OCR attachment → rename (`ocr-rename`) MVP +- [ ] OCR cron pack for HR “Scanned document” inboxes +- [ ] categories create/update (list only for now) + +--- + +## Steering / next decisions (pick a lane) + +### A — Daily-driver polish — **done for now** +Reopen if browse UX still nags. + +### B — CS frontage / NMI / OCR — **MVP+ done** +Optional: OCR cron for HR; NMI membership index. + +### C — CRM / spam tooling **← good next lane** +1. Deeper `contact-set` (custom fields) or keep routing to `redmine_contacts.py` +2. Control-tag LLM refine from JSONL export +3. Issue categories create/update for triage taxonomy + +### D — Bigger product bets (defer unless intentional) +- Full curses TUI +- Unified contact cache +- Composable remote/local filter DSL +- Redmine project/version/custom-field admin + +--- + +## Explicitly out of this index (or later) + +| Item | Notes | +|------|--------| +| Semantic index v2 | `semantic_index/` — feed from this cache | +| Vault decompose | separate track | +| Redmine admin: versions / custom field *definitions* / project modules | not ticket ops | +| Master global JSON + symlink org | nice-to-have | + +--- + +## Related docs + +- `docs/cli_power_user_client.md` — **usage cookbook** +- `docs/ticket_sync_tool_plan.md` — original sync plan +- `docs/2026-07-14_localsearch_before_sanity.md` — historical +- `docs/ticket_spam_cleanup_handoff_2026-07-09.md` — spam consumers +- `docs/raw-notes/features/*.md` — detail files +- `configs/live.json`, `configs/dev.json` · `bin/rtl` · `bin/rtd` + +### Feature detail files + +| File | Topic | +|------|--------| +| [[features/sync-foundation]] | Sync / cache / remote filters | +| [[features/local-search-and-cli-client]] | Search, view, walk, queues, writes | +| [[features/control-tags]] | Local control tags | +| [[features/contacts-sync-integration]] | Contacts mirror + bridge | +| [[features/attachments]] | File attachments | +| [[features/entity-type-mapper]] | Entities + NMI | +| [[features/nmi-bridge]] | NMI env + browse `m` | +| [[features/ocr-rename]] | OCR subject rename | +| [[features/browse-loop]] | fzf browse loop | +| [[features/config-files]] | Config / aliases | + +--- + +## Maintenance rule + +1. Put full detail in `docs/raw-notes/features/.md` +2. Leave status + short bullets + link here +3. Prefer updating the feature file over re-pasting long prose into this index +4. When fixing a bug from **Bug reports**, check the box and leave the minimal repro snippet + +--- + +## Bug reports and QOL (resolved log) + +### --allow-prod-writes unrecognized / ignored +- [x] Parent flag only; subparser copies removed (child defaults overwrote parent) + +### Draft id not obvious +- [x] `draft_id=` / `apply_hint=` key=value output + +### Drafts ignore create-time base-url +- [x] Apply prefers draft `base_url`; conflict needs `--allow-base-url-override` + +### Configs / short aliases +- [x] Minimal `--config` + `bin/rtl|rtd|qaa` + +### `ticket-contact --fetch` alone did nothing +- [x] `--fetch` implies show + live refresh into contacts cache + +--- + +## Modules (code map) + +| Module | Role | +|--------|------| +| `redmine_tickets.py` | CLI entry + sync + routing | +| `redmine_ticket_query.py` | Local AST filters | +| `redmine_ticket_view.py` | Markdown render / pager | +| `redmine_ticket_browse.py` | fzf loop + action menu | +| `redmine_ticket_filters.py` | Saved filters / queues | +| `redmine_ticket_drafts.py` | Draft queue + write safety hosts | +| `redmine_ticket_meta.py` | Name→id meta cache | +| `redmine_ticket_control_tags.py` | Local tags store | +| `redmine_ticket_control_export.py` | Tag JSONL export | +| `redmine_ticket_contacts.py` | Contact mirror | +| `redmine_ticket_attachments.py` | Upload/download helpers | +| `redmine_ticket_entities.py` | Pattern extract | +| `redmine_ticket_nmi.py` | NMI CLI bridge | +| `redmine_ticket_ocr.py` | Attachment OCR + subject suggest | +| `redmine_contacts.py` | Full CRM CLI (sibling tool) | diff --git a/docs/raw-notes/features/attachments.md b/docs/raw-notes/features/attachments.md new file mode 100644 index 0000000..a98d461 --- /dev/null +++ b/docs/raw-notes/features/attachments.md @@ -0,0 +1,28 @@ +# Feature detail: Ticket attachments + +**Parent roadmap:** [[../features-for-redmine_tickets-sync]] +**Status:** MVP 2026-07-15 +**Module:** `redmine_ticket_attachments.py` + +## Commands + +```bash +./bin/rtl attachments list --issue-id 39305 --refresh +./bin/rtl attachments download --issue-id 39305 --attachment-id 55917 +./bin/rtl attachments download --issue-id 39305 --all +./bin/rtd attachments upload --issue-id 39305 --file ./scan.pdf # dry-run +./bin/rtd attachments upload --issue-id 39305 --file ./scan.pdf --apply --notes 'scan attached' +``` + +| Op | Emails customer? | Notes | +|----|------------------|--------| +| list | no | cache-first; `--refresh` pulls `include=attachments` | +| download | no | needs API key; default dir `//attachments//` | +| upload | **no** | Redmine `POST /uploads.json` + issue `uploads` token; prod needs `--allow-prod-writes` | + +Sync now also requests `attachments` on full issue fetch so metadata lands in cache over time. + +## Still open + +- OCR rename workflow (see [[../ocr-ticket-attachment-to-rewrite-update-ticket]]) +- Draft-queue binary uploads (paths) if desired diff --git a/docs/raw-notes/features/browse-loop.md b/docs/raw-notes/features/browse-loop.md new file mode 100644 index 0000000..0f9a482 --- /dev/null +++ b/docs/raw-notes/features/browse-loop.md @@ -0,0 +1,44 @@ +# Feature detail: Interactive browse loop + +**Parent roadmap:** [[../features-for-redmine_tickets-sync]] +**Status:** MVP+ polish 2026-07-15 +**Module:** `redmine_ticket_browse.py` + +## Flow + +1. Filtered ticket list via **fzf** (preview pane) or numbered pick +2. **Enter** → Markdown pager (`glow` / `bat` / `batcat` / `cat`) +3. After pager → **action menu** +4. Back to list until Esc/q + +## Actions (queue-safe) + +| Key | Behavior | +|-----|----------| +| v / Enter | View again | +| n / p | Queue internal / private note | +| f | Queue field update (names) | +| t | Local control-tag | +| c | ticket-contact handoff | +| e | Pattern entities | +| **m** | NMI cs-brief → confirm → **private** note draft | +| i | Print id | +| l / q | List / quit | + +## Flags + +- `--where`, `--status`, `--control-tag`, `--has-attachments`, **`--open-only`** +- `--max-tickets`, `--order`, `--pager`, `--once`, **`--no-preview`** + +## Done polish + +- [x] fzf header lists actions +- [x] fzf preview pane +- [x] loud draft banner +- [x] NMI menu action + +## Non-goals + +- Full curses app (separate spike) +- Customer-email from browse (CLI-only by design) +- Applying prod writes from browse without draft step diff --git a/docs/raw-notes/features/config-files.md b/docs/raw-notes/features/config-files.md new file mode 100644 index 0000000..8dc1ba8 --- /dev/null +++ b/docs/raw-notes/features/config-files.md @@ -0,0 +1,41 @@ +# Config files / short aliases (minimal) + +**Parent:** [[../features-for-redmine_tickets-sync]] +**Status:** minimal MVP (global defaults + bin wrappers). Nested per-command defaults deferred. + +## Runtime config + +```bash +./redmine_tickets.py --config configs/live.json search --max-tickets 10 +./redmine_tickets.py --config configs/dev.json note --issue-id 39305 --body 'x' --apply +``` + +JSON (always works) or YAML if PyYAML is installed. + +Supported top-level keys (also under `redmine_tickets:`): + +- `base-url` / `base_url` +- `project` +- `cache-dir` / `cache_dir` +- `allow-prod-writes` (bool) + +CLI flags already present win over config. + +## Short wrappers (`bin/`) + +| Command | Maps to | +|---------|---------| +| `bin/rtl` | live/prod CS (`configs/live.json`) | +| `bin/rtd` | LAN/dev CS (`configs/dev.json`) | +| `bin/qaa` | alias → `rtl` (muscle-memory example) | + +```bash +./bin/rtl search --where 'assigned_to IS NULL' --max-tickets 5 +./bin/rtl --allow-prod-writes draft apply +./bin/rtd note --issue-id 39305 --body 'LAN note' --apply +``` + +## Not yet + +- Nested `search.max-tickets` / command-specific defaults from YAML namespaces +- Auto-discovery of many named profiles beyond live/dev diff --git a/docs/raw-notes/features/contacts-handling.sh b/docs/raw-notes/features/contacts-handling.sh new file mode 100644 index 0000000..858ebe1 --- /dev/null +++ b/docs/raw-notes/features/contacts-handling.sh @@ -0,0 +1,5 @@ +## Contact Triage / Fixing + +- [ ] Properly link "call transcript" and "voicemail" tickets to the correct person by searching the contact database with the phone number involved +- [ ] Ingest call transcripts into our timeline +- [ ] Fix contact company is set to their email address, when the name of their business is easily found through context in tickets \ No newline at end of file diff --git a/docs/raw-notes/features/contacts-sync-integration.md b/docs/raw-notes/features/contacts-sync-integration.md new file mode 100644 index 0000000..e76f2f2 --- /dev/null +++ b/docs/raw-notes/features/contacts-sync-integration.md @@ -0,0 +1,38 @@ +# Feature detail: Contacts sync + mutation (tie-in) + +**Parent roadmap:** [[../features-for-redmine_tickets-sync]] +**Status:** Sync/search/show + draftable `contact-set` + `ticket-contact` bridge (2026-07-15) +**Module:** `redmine_ticket_contacts.py` +**Still separate from:** full `redmine_contacts.py` CRM CLI (batch/loops/custom-field tooling) + +## Why not one tool + +- Tickets workbench wants co-located per-id JSON + draft/apply safety next to tickets. +- `redmine_contacts.py` already owns heavy CRM workflows and a different cache layout. +- Do not rewrite/delete the CRM CLI; add bridges so a ticket-started workflow can end on contacts. + +## Implemented + +```bash +./bin/rtl contacts-sync --max-contacts 100 +./bin/rtl contacts-search --query ada +./bin/rtl contact-show --contact-id 14 --fetch + +# Bridge ticket → contact id +./bin/rtl ticket-contact --issue-id 41934 +./bin/rtl ticket-contact --issue-id 41934 --show --fetch + +# CRM mutation (draft by default; same prod/LAN gates as ticket writes) +./bin/rtl contact-set --contact-id 4515 --add-tag maybe-customer +./bin/rtd contact-set --contact-id 2284 --add-tag maybe-customer --apply --refresh-cache +``` + +- Durable path: `/contacts/.json` + `manifest.json` +- Draft kind: `contact_fields` (apply via `draft apply`) +- Handoff hints print `redmine_contacts.py show …` for full CRM when needed + +## Still open + +- Richer custom-field helpers (use `redmine_contacts.py` or extend `--set`) +- Optional cron for contacts next to ticket sync +- Shared library / single on-disk layout if dual-cache friction becomes painful diff --git a/docs/raw-notes/features/control-tags.md b/docs/raw-notes/features/control-tags.md new file mode 100644 index 0000000..a821837 --- /dev/null +++ b/docs/raw-notes/features/control-tags.md @@ -0,0 +1,31 @@ +# Feature detail: Local control tags (filter-assist labeling) + +**Parent roadmap:** [[../features-for-redmine_tickets-sync]] +**Status:** MVP implemented 2026-07-15 (`control-tag` CLI + `search --control-tag`) +**Module:** `redmine_ticket_control_tags.py` +**Critical distinction:** these tags are **local-only**. They are **not** Redmine issue tags, **not** contact CRM tags. + +## Storage + +`/control-tags/tags.json` — wipe without touching Redmine or ticket JSON. + +## CLI + +```bash +./redmine_tickets.py ... control-tag add --tag bavaria --issue-id 41934 \ + --property subject --property helpdesk_ticket.contact.company \ + --annotation 'BavariaTrachten shape' + +./redmine_tickets.py ... control-tag list +./redmine_tickets.py ... control-tag show --tag bavaria +./redmine_tickets.py ... control-tag for-issue --issue-id 41934 +./redmine_tickets.py ... control-tag remove --tag bavaria --issue-id 41934 +./redmine_tickets.py ... control-tag wipe --i-understand-this-wipes-local-control-tags + +./redmine_tickets.py ... search --control-tag bavaria +``` + +## Still open + +- Suggest/refine loop (LLM or rules + human correct/incorrect) +- Training-export format for classifier experiments diff --git a/docs/raw-notes/features/entity-type-mapper.md b/docs/raw-notes/features/entity-type-mapper.md new file mode 100644 index 0000000..b4c8ebc --- /dev/null +++ b/docs/raw-notes/features/entity-type-mapper.md @@ -0,0 +1,25 @@ +# Feature detail: External entity type mapper / identifier + +**Parent roadmap:** [[../features-for-redmine_tickets-sync]] +**Status:** pattern MVP + NMI CLI bridge + browse `m` (2026-07-15) +**Modules:** `redmine_ticket_entities.py`, `redmine_ticket_nmi.py` +**NMI env docs:** [[nmi-bridge]] + +## Commands + +```bash +./bin/rtl entities show --issue-id 41394 +./bin/rtl entities scan --where 'subject.includes("1Z")' --max-tickets 20 +./bin/rtl entities nmi --issue-id 41394 +``` + +## Done + +- Pattern types: UPS/USPS, FedEx candidates, Amazon order, order_id, item_or_sku, **item_no** (LDR 9-digit) +- Markdown view section: **Detected entities** +- NMI bridge + browse **`m`** → private note draft + +## Open + +- [ ] Optional membership index of recent NMI order/item ids +- [ ] Postal addresses / phones / social (much later) diff --git a/docs/raw-notes/features/local-search-and-cli-client.md b/docs/raw-notes/features/local-search-and-cli-client.md new file mode 100644 index 0000000..b1a0c9c --- /dev/null +++ b/docs/raw-notes/features/local-search-and-cli-client.md @@ -0,0 +1,39 @@ +# Feature detail: Local search + CLI Redmine client + +**Parent roadmap:** [[../features-for-redmine_tickets-sync]] +**Status:** MVP+ (2026-07-14 … 2026-07-15) — ops-usable +**Cookbook:** `docs/cli_power_user_client.md` + +## Modules + +| Module | Role | +|--------|------| +| `redmine_ticket_query.py` | Property AST (`--where`) | +| `redmine_ticket_view.py` | Markdown + pager | +| `redmine_ticket_browse.py` | fzf → pager → action menu loop | +| `redmine_ticket_filters.py` | Saved filters + symlink queues | +| `redmine_ticket_drafts.py` | Draft queue + host safety | +| `redmine_ticket_meta.py` | Name→id for set-fields | +| `redmine_ticket_ocr.py` | Attachment text extract + subject suggest | + +## Done + +- [x] Local search / `@saved` filters / queues +- [x] view / walk (`--open-only` on search/browse/walk) +- [x] note vs customer-email vs set-fields (draft/apply) +- [x] `--body-file` + `--to-textile` (pandoc) +- [x] browse **loop** with action menu (queue-only mutations + NMI `m`) +- [x] configs: `bin/rtl`, `bin/rtd`, `--config` +- [x] `ocr-rename` + `@scanned-document` +- [x] `categories list` + +## Open + +- [ ] Nested config defaults +- [ ] Full curses TUI (optional later) +- [ ] categories create/update + +## Non-goals + +- Replacing Redmine web UI for everything +- Silent customer email from browse diff --git a/docs/raw-notes/features/nmi-bridge.md b/docs/raw-notes/features/nmi-bridge.md new file mode 100644 index 0000000..bb9ed7f --- /dev/null +++ b/docs/raw-notes/features/nmi-bridge.md @@ -0,0 +1,41 @@ +# NMI bridge env + usage + +**Parent:** [[entity-type-mapper]] · module `redmine_ticket_nmi.py` + +## Environment + +| Variable | Default | Purpose | +|----------|---------|---------| +| `NMI_CLI` | `/home/jason/projects/mrscClient-php/bin/nmi` | Path to NMI PHP CLI | +| `NMI_ENV_FILE` | `/home/jason/projects/mrscClient-php/examples/modern/.env` | Env file for credentials/endpoint | +| `NMI_MODE` | `dev` | CLI `--mode` (dev/prod as supported by nmi) | + +Requires `php` on PATH. + +## CLI + +```bash +# Pattern extract only +./bin/rtl entities show --issue-id 41394 +./bin/rtl entities scan --type tracking_ups --max-tickets 20 + +# Resolve / cs-brief via NMI CLI +./bin/rtl entities nmi --issue-id 41394 +./bin/rtl entities nmi --issue-id 41394 --nmi-mode dev +./bin/rtl entities nmi --issue-id 41394 --as-private-note-draft # if wired; else use browse `m` +``` + +## Browse + +After opening a ticket in `browse`, press **`m`**: + +1. Runs NMI brief from extracted entities +2. Shows brief text +3. Confirms → queues **private** internal note draft + +Never emails the customer. + +## Safety + +- NMI is read-side ops context, not customer mail +- Private note drafts still need `draft apply` (+ prod flags) to land in Redmine diff --git a/docs/raw-notes/features/ocr-rename.md b/docs/raw-notes/features/ocr-rename.md new file mode 100644 index 0000000..be3115b --- /dev/null +++ b/docs/raw-notes/features/ocr-rename.md @@ -0,0 +1,46 @@ +# Feature detail: OCR attachment → subject rename + +**Parent roadmap:** [[../features-for-redmine_tickets-sync]] +**Status:** MVP 2026-07-15 +**Module:** `redmine_ticket_ocr.py` · CLI: `ocr-rename` + +## Intent + +Scanned docs land in Helpdesk as tickets titled “Scanned document…”. +Ops need: open attachment → understand contents → rename subject (+ optional notes). + +## Commands + +```bash +# One ticket: download OCR-able attachments, propose subject, queue private note (+ subject draft) +./bin/rtl ocr-rename --issue-id 12345 + +# Scan cache for generic scan subjects +./bin/rtl ocr-rename --where '@scanned-document' --only-generic-subjects --max-tickets 20 + +# Apply subject now (LAN/prod safety gates); still queues private OCR note by default +./bin/rtd ocr-rename --issue-id 12345 --apply-subject + +# Extract only (no drafts) +./bin/rtl ocr-rename --issue-id 12345 --no-queue-subject --no-queue-note --json +``` + +## Pipeline + +1. Load ticket + attachments (`include=attachments` refresh if needed) +2. Download PDF/image candidates (multi-URL fallback) +3. Extract text: `pdftotext` → else `pdftoppm` + `tesseract` +4. Heuristic `suggest_subject_from_text` (doc type, person/company, date) +5. Queue drafts: + - `fields` subject change (unless `--no-queue-subject`) + - private `internal_note` with OCR snippet (unless `--no-queue-note`) + +## Saved filter + +`saved-filters/scanned-document.expr` → `@scanned-document` + +## Still open + +- [ ] Stronger LLM subject summarization (optional; keep heuristics default) +- [ ] Cron wrapper that only processes new generic-scan tickets +- [ ] HR-project default project config example diff --git a/docs/raw-notes/features/p5-exploration.md b/docs/raw-notes/features/p5-exploration.md new file mode 100644 index 0000000..28ffcdd --- /dev/null +++ b/docs/raw-notes/features/p5-exploration.md @@ -0,0 +1,3 @@ + +- [ ] Download voicemails and call recordings from voip.ms automatically + - [ ] helper PHP client library https://git.int.ldrprep.com/dusk/php-voip.ms \ No newline at end of file diff --git a/docs/raw-notes/features/sync-foundation.md b/docs/raw-notes/features/sync-foundation.md new file mode 100644 index 0000000..0a0c033 --- /dev/null +++ b/docs/raw-notes/features/sync-foundation.md @@ -0,0 +1,91 @@ +# Feature detail: Ticket sync foundation + +**Parent roadmap:** [[../features-for-redmine_tickets-sync]] +**Status:** largely implemented (2026-07-14); cron wiring still open +**Primary script:** `redmine_tickets.py` +**Target project (first):** `customer-service` + +## Intent + +Durable, incremental local mirror of Redmine issues so humans and agents can work offline-first, cron-refresh frequently, and not re-pull full payloads for unchanged tickets. + +## Requirements + +- Sync tickets updated within a window (e.g. past 30 days) into a **stable** cache path (not timestamped per run). +- Skip full fetch when cached `updated_on` still matches the list stub. +- Support frequent refresh (every few minutes / couple times per hour) using a tight remote window after the initial warm-up. +- Never delete old ticket JSON unless explicitly asked. +- Optional small run-stats ring buffer for cron observability. + +## Cache locations + +| Environment | Path | +|-------------|------| +| Production (`https://rm.ldrprep.com`) | `/home/jason/projects/redmine/cache/redmine_tickets` | +| Dev / LAN (`http://192.168.50.170`) | `/home/jason/projects/redmine/cache-dev/redmine_tickets` | + +Layout: + +```text +/ + manifest.json + run-stats.json # optional, with --run-log + / + .json +``` + +## Server-side vs local filters (sync) + +| Flag | Side | Meaning | +|------|------|---------| +| `--status-id` / `--remote-status-id` | **remote** | Redmine API `status_id` (`*`, `open`, `closed`, numeric) | +| `--remote-updated-on` | **remote** | Raw Redmine syntax: `>=2026-06-14`, `>=2014-01-02T08:12:32Z`, `>=...` (YYYY-MM-DD, ISO, `30d`, `2 days ago`, …) | +| `--remote-updated-before` | **remote** | Convenience → `<=...`; with since → `>/run-stats.json`, keep last N (`--run-log-keep`, default 3). + +## Recommended commands + +```bash +cd /home/jason/projects/redmine + +# Initial 30-day window (server-side) +./redmine_tickets.py \ + --base-url https://rm.ldrprep.com \ + --project customer-service \ + --cache-dir /home/jason/projects/redmine/cache/redmine_tickets \ + sync \ + --remote-status-id '*' \ + --remote-updated-since 30d \ + --run-log + +# Frequent refresh +./redmine_tickets.py \ + --base-url https://rm.ldrprep.com \ + --project customer-service \ + --cache-dir /home/jason/projects/redmine/cache/redmine_tickets \ + sync \ + --remote-status-id '*' \ + --remote-updated-since 2h \ + --run-log \ + --quiet +``` + +Raw Redmine form: + +```bash +./redmine_tickets.py ... sync --remote-updated-on '>=2026-06-14' --status-id '*' +``` + +## Open + +- [x] Scripts under `cron-scripts/` installed + crontab running (live + dev) +- [ ] **ADJUST LATER (do not forget):** cron still uses `--remote-updated-since 30d` on a few-minute cadence. Fine while validating (~3–4s/run after warm cache) but should tighten to `2h`/`6h` (or marker-based since-last-success) so list pages stay cheap long-term. `run-log-keep 20` is already set. +- [ ] Optional: master storage + symlink organization of JSON (see parent roadmap) diff --git a/docs/raw-notes/ocr-ticket-attachment-to-rewrite-update-ticket.md b/docs/raw-notes/ocr-ticket-attachment-to-rewrite-update-ticket.md new file mode 100644 index 0000000..4d12c4d --- /dev/null +++ b/docs/raw-notes/ocr-ticket-attachment-to-rewrite-update-ticket.md @@ -0,0 +1,31 @@ +I want a skill (or set of skills) that accomplishes the following: + +## Context + +We have found that using networked printers like HPs to scan various documents have them emailed to inboxes wired up to the Redmine Helpdesk plugin a very useful way to avoid losing paperwork and speed up finding it later. That has been instrumental on several occassions, where we've had the printer in my office configured to email scanned files to hr@ldrprep.com, where they get imported as tickets with the scanned document as a file attachment. + +Even with that, to make it truly useful, somebody still has to typically do this: + + 1. Log into redmine and go to the hr project's issues. + 2. Identify scanned document emails (usually the subject like says something like "Scanned document...") + 3. Open the attachment to see what it is. + 4. Edit the ticket and change the subject line so if we search later on, we'll most likely be able to find the ticket with the attached file. + 5. Optionally, transcribe key points or info from the document and add them to the ticket. + +## Goal: automate this process + + +- [ ] **Convert target ticket**: When passed a redmine issue id and asked to peform this action, the following should happen: + - Fetch the ticket and check if it has any PDF files attached to it. + - Download the PDFs and use OCR or similar technology to extract the main topics and entities mentioned within the document(s). + - Update the ticket by changing it's subject line to reflect what the attached documents are. If they're mainly about one particular person or company, the subject line should include those. +- [ ] **Scan project**: When asked, search/scan a specified Redmine project to locate tickets where this hasn't been done yet, and then do it for all of the tickets found. +- [ ] **Cron-usable skill**: Once this is done, I want to have it as a skill I could easily add to an agentic cronjob, what will enable it to scan/check a specified list of Redmine projects whenever it runs, and then automatically perform this action. + +## Testing parameters + +- The test instance of redmine located at `192.168.50.170` can be used to test functionality. It supports the same authentication info as the live redmine. +- The email aspect of this is not important at all. It would be fine to simulate it as follows: + - Create a new ticket simulating what an ideal target would look like + - Upload a pdf to it + - Then, run the relevant workflow against the test ticket to check results diff --git a/docs/rebot_legacy_spam_rules.md b/docs/rebot_legacy_spam_rules.md new file mode 100644 index 0000000..1528489 --- /dev/null +++ b/docs/rebot_legacy_spam_rules.md @@ -0,0 +1,154 @@ +# Rebot Legacy Known-Spam Rules + +**Source inspected:** + +- `/home/jason/projects/rebot/rebot/RebotSlick.php` +- `/home/jason/projects/rebot/rebot/RebotSlick.php.save` +- `/home/jason/projects/rebot/rebot/RebotSlick.php.save.1` +- `/home/jason/projects/rebot/rebot/Classify/Receipts.php` + +## Summary + +The old Rebot spam cleanup used `RebotSlick::CleanSpam()` which iterated open customer-service issues and called `KnownSpam($issue)`. When `KnownSpam()` returned true, Rebot updated the issue to Closed and set category `Spam`. + +Current source behavior: + +```php +if ( $this->KnownSpam($issue)) { + $this->client->issue->update($issue['id'], [ + "status_id" => $this->statusByName['Closed'], + "category_id" => $this->categoriesByName['Spam'] + ]); +} +``` + +The saved older copies also used `KnownSpam()` and closed tickets, sometimes with note `Rebot identified as spam`. + +## Exact KnownSpam source logic + +### Direct sender equality + +```php +if ($from_address == 'noreply@rush-delivery.com') return true; +``` + +### Amazon package special case + +```php +if ($from_address == 'no-reply@amazon.com') { + if (strpos($subject, "Your Amazon package") !== false) return true; +} +``` + +### Current `badEmails` regex list + +```php +"/^.*@indeedemail.com/i" +"/^no-reply@indeed.com/i" +"/^.*@.*roku.com/i" +"/^no.reply@masterclass.com/i" +"/^donotreply@walmart.com/i" +"/^.*@uber.com/i" +"/^no-reply@inhome.walmart.com/i" +"/^.*wingsandrings.com/i" +"/mailnull@amazon.com/i" +"/^.*@tryfinally.co/i" +"/^.*crackerbarrel.com/i" +"/^.*shop.tiktok.com/i" +"/^noreply_samsung@sea.samsung.com/i" +"/^noreply@email.crunchbase.com/i" +"/^reply@globalindustrial.com/i" +"/^.*webstaurantstore.com/i" +"/^info@emailbiz.dukeenergy.com/i" +"/^support@helium10.com/i" +"/^noreply@popmenu.com/i" +"/^info@go.cbts.com/i" +``` + +### Older saved `badEmails` list + +The `.save` and `.save.1` variants had the earlier subset: + +```php +"/^.*@indeedemail.com/i" +"/^no-reply@indeed.com/i" +"/^.*@.*roku.com/i" +"/^no.reply@masterclass.com/i" +"/^donotreply@walmart.com/i" +"/^.*@uber.com/i" +"/^no-reply@inhome.walmart.com/i" +"/^.*@wingsandrings.com/i" +"/mailnull@amazon.com/i" +"/^.*@tryfinally.co/i" +"/^.*crackerbarrel.com/i" +"/^.*shop.tiktok.com/i" +``` + +The current source adds Samsung, Crunchbase, Global Industrial, Webstaurant Store, Duke Energy newsletters, Helium10, Popmenu, and CBTS. + +### `badSubjects` regex list + +```php +"/Your Amazon.com order.*has shipped/i" +"/^Your Duke energy statement is ready/i" +"/^Your Amazon.com order.*/i" +"/^UPS Billing: Payment Received.*/i" +"/^UPS Billing:.*new invoice.*/i" +"/^.applied on indeed.*/i" +"/^See results for the assessments you sent to*/i" +"/^Reminder: Leave feedback on your interview with.*/i" +"/^Pekin AutoPay Payment.*/i" +"/^Notification of Active Listings Report availability/i" +``` + +## Important interpretation for modern tooling + +The old Rebot called all of the above `spam`, but several of these are better understood today as **operational clutter** rather than promotional spam: + +- Amazon order/shipping/package notifications +- UPS billing/payment/invoice notices +- Duke Energy statements/newsletters +- Pekin AutoPay payments +- Indeed applicant/assessment/interview mail +- notification/report availability emails + +For the new fast path, preserve the rules but do **not** treat every legacy rule as `spam-promotional`. Recommended split: + +1. `legacy-rebot-known-spam-promotional` + - promotional/vendor/newsletter/sales senders + - safe for fast close if contact is not `Customer` / `maybe-customer` + +2. `legacy-rebot-operational-clutter` + - Amazon order/shipping, UPS billing, Duke/Pekin billing, Indeed workflow mail + - can be fast-closed only if Jason explicitly approves closing operational clutter as a separate lane + +3. `legacy-rebot-receipts-routing` + - from `Classify/Receipts.php`; these were routed to Receipts / Accounting / IT, not generic spam close + +## Receipts classifier reference + +`rebot/Classify/Receipts.php` used `GenericMail` to classify: + +### Email regexes + +```php +/culliganbillingoh@culliganwaterco.com/i +/accounting@amshippingsupplies.com/i +/bookkeeperasl@gmail.com/i +/bounces-noreply@mandrillapp.com/i +``` + +### Subject regexes + +```php +/You have a new bill from altafiber/i +``` + +### Actions + +- Default: status `Resolved`, category `Receipts` +- AMShipping invoices: category `Bill Needs Paid`, assigned Accounting +- JoeLevine/bookkeeper: category `Bill Needs Paid`, assigned Accounting +- Mandrill bounces: status `Resolved`, category `IT Issue`, assigned Technical Services + +These should not automatically enter the promotional spam fast-close registry, but they are useful future operational-clutter/routing profiles. diff --git a/docs/redmine_contacts_duplicate_merge_feasibility.md b/docs/redmine_contacts_duplicate_merge_feasibility.md new file mode 100644 index 0000000..07a404d --- /dev/null +++ b/docs/redmine_contacts_duplicate_merge_feasibility.md @@ -0,0 +1,69 @@ +# Redmine Contacts duplicate-merge feasibility (API vs script) + +## Scope +Assess whether duplicate contact merge (web flow: `/contacts//duplicates?project_id=` with `duplicate_id=`) can be safely automated from `redmine_contacts.py` or shell tooling. + +## Findings from local plugin source + +### 1) Merge flow exists as a controller action, not a JSON API endpoint +- Controller: `plugins/redmine_contacts/app/controllers/contacts_duplicates_controller.rb` +- Routes: `plugins/redmine_contacts/config/routes.rb` + - `match 'contacts/:contact_id/duplicates' => 'contacts_duplicates#index', :via => [:get, :post]` + - `match 'contacts_duplicates/:action' => 'contacts_duplicates', :via => [:get, :post]` + +`ContactsDuplicatesController` does **not** declare `accept_api_auth` and does **not** implement `format.api` responses. + +### 2) Merge behavior is server-side and destructive +Inside `merge` action: +- combines notes, deals, related_deals, issues, projects from source contact into duplicate target, +- unions email/phone/tag sets, +- saves target, +- destroys source contact. + +This matches the intended business behavior (source contact removed, linked records moved). + +### 3) Web form posts to merge action with `duplicate_id` +- View: `plugins/redmine_contacts/app/views/contacts_duplicates/index.html.erb` +- Uses Rails `form_tag` to `contacts_duplicates#merge` with `project_id`, `contact_id`, and posted `duplicate_id`. + +## Practical feasibility + +## A) Native Redmine REST API with API key +**Not currently feasible** for this merge operation. + +Reason: no API auth route/serializer for duplicate merge in this plugin path. + +## B) Scripted browser-equivalent flow (`curl`/requests with session cookies) +**Feasible with caution**. + +Required mechanics: +1. Authenticate as a normal Redmine user (session/cookie auth). +2. GET duplicate page for the source contact to obtain valid session + CSRF authenticity token. +3. POST to merge action with: + - `project_id` + - `contact_id` (source) + - `duplicate_id` (target) + - authenticity token +4. Verify redirect/result contact and audit that source contact no longer exists. + +## Risk notes +- This is a destructive operation (deletes source contact). +- Needs stronger guardrails than normal update operations: + - explicit `--apply` style confirmation, + - dry-run mode that only validates both contacts and project scope, + - optional `--test-instance-only` safety default, + - write operation log with source/target IDs and timestamp. + +## Recommendation for `redmine_contacts.py` +1. **Do not** add duplicate merge via API-key-only path unless plugin gains a dedicated JSON API action. +2. If we implement now, add a **separate command** using session-auth web flow (not mixed into normal contact update path), e.g.: + - `merge-duplicate --source-contact --target-contact --project [--apply]` +3. Keep it disabled-by-default for production base URL, with explicit override flag. + +## Suggested next technical step +Prototype a standalone script against `http://192.168.50.170` that: +- logs in, +- resolves CSRF token, +- performs merge POST, +- verifies resulting contact IDs/issue linkage, +- emits structured JSON result for later CLI integration. diff --git a/docs/redmine_contacts_search.md b/docs/redmine_contacts_search.md new file mode 100644 index 0000000..6904a4b --- /dev/null +++ b/docs/redmine_contacts_search.md @@ -0,0 +1,145 @@ +# Contact search (`redmine_contacts.py`) + +`./redmine_contacts.py search --help` prints three short examples. This page is the situation list. Gaps are in [redmine_contacts_search_proposals.md](redmine_contacts_search_proposals.md). + +This CLI was not changed to write this page. Commands below match the current `search` / `show` / `helpdesk-issues` behavior. + +## What `search` actually does + +1. Load the contact cache, or reload it with `--refresh`. +2. Apply exact filters: `--with-tag`, `--without-tag`, `--no-tags`, `--created-after`, `--created-before`, `--updated-after`, `--updated-before`. +3. Fuzzy-score the remaining contacts against the required positional query. +4. Print text lines, capped by `--limit` (default 20). + +It does not search tickets. It does not print JSON. A normal search still requires `REDMINE_API_KEY` even when it only reads the cache. `--offline-cache` is the exception. + +Default host is the LAN copy, `http://192.168.50.170`. Production: + +```bash +cd /home/jason/projects/redmine +./redmine_contacts.py --base-url https://rm.ldrprep.com search "Sahar Daod" +``` + +Cache file: `.cache/redmine_contacts/_.json`, relative to the current directory. A cache built for one host is rejected for another. This is not `bin/rtl contacts-search`. + +A hit is a substring of a scored field after lowercasing, or a fuzzy score of at least `0.68`. Scored fields: name, email, phone, company, job title, background, website, skype, tags, address, and custom-field values. A phone query is also matched on digits only, so `847-313-1342` hits `8473131342`. + +Output shape: `id: name | company | email | phone | score=1.00 via email`. Use `show ` for the record. + +`--refresh` pages through every project contact before scoring. Without it, a tag or field written since the last fetch is invisible. + +## Supported + +### 1. Find a person by name, email, or phone + +```bash +./redmine_contacts.py --base-url https://rm.ldrprep.com search "Sahar Daod" +./redmine_contacts.py --base-url https://rm.ldrprep.com search sahardaod4@groovinfulfillment.com +./redmine_contacts.py --base-url https://rm.ldrprep.com search 847-313-1342 +``` + +Then the record, including `tag_list`: + +```bash +./redmine_contacts.py --base-url https://rm.ldrprep.com show 4589 +``` + +`show` prints JSON. `--source api` skips a stale cache. `--source cache` does not call Redmine. + +### 2. Find people at a company, or everyone on one email domain + +Company and email are scored fields, not dedicated filters. A domain that appears in the address scores `1.00`. + +```bash +./redmine_contacts.py --base-url https://rm.ldrprep.com search gatsbylady.com --refresh +./redmine_contacts.py --base-url https://rm.ldrprep.com search "Cyberstorm" --min-score 0.9 +``` + +Raise `--min-score` when a short company name also fuzzy-matches unrelated contacts. There is no `--company` or `--email-domain` flag. An exact company filter is proposal 1. + +### 3. Exact tag + +`--with-tag` is exact and case-insensitive. The positional query is still required and still fuzzy, so repeat the tag as the query. Otherwise a contact with the tag can be filtered in and then dropped because the name did not fuzzy-match. + +```bash +./redmine_contacts.py --base-url https://rm.ldrprep.com \ + search promotional-spam --with-tag promotional-spam --refresh --limit 50 +``` + +Offline, against the cache already on disk for that host: + +```bash +./redmine_contacts.py --base-url https://rm.ldrprep.com \ + search promotional-spam --with-tag promotional-spam --offline-cache +``` + +Repeat `--with-tag` to require every tag. Repeat `--without-tag` to exclude any of those tags. + +```bash +./redmine_contacts.py --base-url https://rm.ldrprep.com \ + search customer --with-tag customer --without-tag promotional-spam --limit 20 +``` + +`--no-tags` cannot be combined with `--with-tag`. Listing every untagged contact is not this command. See proposal 2. + +### 4. A tag, narrowed by when the contact was created or updated + +Date flags are exact. The query is still required, so this finds "this text, inside this window," not "everyone touched this week." + +```bash +./redmine_contacts.py --base-url https://rm.ldrprep.com \ + search customer --with-tag customer --updated-after 2026-09-01 --refresh +``` + +Dates are `YYYY-MM-DD` or a full ISO datetime. A date-only `--updated-before` includes that whole UTC day. Listing by date with no text query is proposal 2. + +### 5. A known account number or other custom-field value + +Custom-field values are scored if they are already on the cached contact. Account Number is custom field 70. + +```bash +./redmine_contacts.py --base-url https://rm.ldrprep.com search 7385Y6 --refresh +``` + +This finds a value you already know. It cannot list customers whose Account Number is empty. That is proposal 3. + +### 6. Their tickets, once you have the contact id + +`search` stops at the contact. These follow-ups are live reads, not the contact cache. + +```bash +./redmine_contacts.py --base-url https://rm.ldrprep.com helpdesk-issues 4589 --limit 20 +./redmine_contacts.py --base-url https://rm.ldrprep.com helpdesk-timeline 4589 --limit 20 +``` + +Both print JSON from `/helpdesk_search/contacts/:id/...`. + +Cached Customer Service tickets, any status: + +```bash +bin/rtl search --order newest-updated-first --max-tickets 20 \ + --where 'helpdesk_ticket.contact_id == 4589 AND project.name == "Customer Service"' +``` + +Open New only: + +```bash +bin/rtl search --status New --open-only \ + --where 'helpdesk_ticket.contact_id == 4589' +``` + +`bin/rtl search` cannot filter on `tag_list`. The ticket's contact stub is `id`, `name`, `email`, `company`. This returns nothing and does not error: + +```bash +bin/rtl search --where 'helpdesk_ticket.contact.tag_list.includes("promotional-spam")' +``` + +One command that means "New tickets from contacts tagged promotional-spam" is proposal 4. `bin/rtl contacts-search --tag` is a different cache and can lag a live tag. + +## Not a search + +`fetch --with-tag customer` writes a cache and prints a count. It does not list contacts. + +`batch-update` can select by the same tag and date filters, but it requires an update payload and exists to write. Do not use it as a lister. + +`loop-run` is a triage worker, not a search. diff --git a/docs/redmine_contacts_search_proposals.md b/docs/redmine_contacts_search_proposals.md new file mode 100644 index 0000000..4b20076 --- /dev/null +++ b/docs/redmine_contacts_search_proposals.md @@ -0,0 +1,99 @@ +# Contact search proposals + +Situations `redmine_contacts.py search` does not cover, as of the current CLI. Do not treat these commands as existing. The working commands are in [redmine_contacts_search.md](redmine_contacts_search.md). + +None of these require changing ticket search. They belong on the contacts CLI, with a dry-run default and no write path. + +## 1. Exact company or email domain + +**Need:** "everyone at gatsbylady.com" or "company is UPS" without fuzzy misses and without a short string scoring unrelated names. + +**Today:** company and email are haystack fields. `search gatsbylady.com` works when that string is inside an email. `search UPS` also fuzzy-matches other fields above `0.68`. There is no `--company` or `--domain`. + +**Proposal:** + +```bash +./redmine_contacts.py --base-url https://rm.ldrprep.com \ + search --domain gatsbylady.com --refresh +./redmine_contacts.py --base-url https://rm.ldrprep.com \ + search --company UPS --company-exact --refresh +``` + +`--domain` matches the part after `@`, case-insensitive, across `emails[]`. `--company-exact` matches `company` after trim and case-fold, not a substring of the whole record. If either flag is set, the positional query becomes optional. Print the same text line `search` prints now, plus `--json` from proposal 5. + +## 2. List by filter, with no fuzzy query + +**Need:** every contact with no tags, or every contact updated since a date, or every `customer` tag, without inventing a query that happens to score. + +**Today:** `search` always requires `query`, then drops anyone under `--min-score`. `--no-tags` and the date flags only narrow that fuzzy set. `fetch --no-tags` caches matches and prints a count, not rows. + +**Proposal:** a `list` subcommand that reuses `filter_contacts` and does not call `search_contacts`. + +```bash +./redmine_contacts.py --base-url https://rm.ldrprep.com \ + list --no-tags --updated-after 2026-09-01 --refresh --limit 50 +./redmine_contacts.py --base-url https://rm.ldrprep.com \ + list --with-tag promotional-spam --refresh +``` + +Require at least one filter, same guard as `batch-update`, so `list` cannot dump the whole CRM by accident. Default `--limit 50`. Order by `updated_on` descending. Text lines should include id, name, email, and `tag_list`, because a bare fuzzy score is meaningless here. + +Do not overload `search` by making `query` optional. Operators already have to know that `search` is fuzzy. + +## 3. Tagged customers with an empty Account Number + +**Need:** billed or tagged customers who have no NMI account id in custom field 70 (`Account Number`), so the jumpstart can see who is unmatched. + +**Today:** a known account number can be fuzzy-searched if it is already stored. There is no empty-field filter. `show` is one id at a time. + +**Proposal:** + +```bash +./redmine_contacts.py --base-url https://rm.ldrprep.com \ + list --with-tag customer --missing-custom-field account-number --refresh +``` + +Resolve `account-number` with the existing custom-field slug (`slugify_custom_field_name`). Missing means the field is absent, null, or whitespace. Do not treat `"0"` as empty. Read-only. This is the list command from proposal 2 plus one predicate, not a new writer. + +## 4. Open tickets from contacts with a tag + +**Need:** New Customer Service tickets whose sender is tagged `promotional-spam`, without hand-joining ids. + +**Today:** ticket JSON does not store `tag_list`. `bin/rtl search --where 'helpdesk_ticket.contact.tag_list.includes(...)'` matches nothing. The working path is tag search, then `helpdesk_ticket.contact_id == N` per id. + +**Proposal:** a read-only join, not a ticket-cache field. + +```bash +./redmine_contacts.py --base-url https://rm.ldrprep.com \ + tagged-issues --with-tag promotional-spam --issue-status New --open-only --refresh +``` + +Steps: + +1. Resolve contact ids with the list filter from proposal 2, live or `--refresh`. +2. For each id, call the existing `helpdesk-issues` endpoint, or scan the ticket cache for `helpdesk_ticket.contact_id`. +3. Keep issues that are New, open, and in `customer-service`. +4. Print `issue_id contact_id email subject`, and stop after `--limit` issues (default 20). + +Do not close anything. Do not add `tag_list` to the ticket cache in this change. A stale contacts cache must be called out in the header line (`cache_refreshed=yes|no`). Refuse to run if the contact cache host does not match `--base-url`, which the current cache reader already does. + +## 5. Script output: ids and JSON + +**Need:** feed contact ids into `bin/rtl` or a later join without scraping `score=1.00 via email`. + +**Today:** `search` prints prose. `show` and `helpdesk-issues` print JSON for one id. There is no `--ids-only` on this CLI. `bin/rtl contacts-search --ids-only` is the other cache. + +**Proposal:** `--json` and `--ids-only` on `search` and on `list`. + +```bash +./redmine_contacts.py --base-url https://rm.ldrprep.com \ + search promotional-spam --with-tag promotional-spam --refresh --ids-only +``` + +`--json` prints one object per line: `id`, `name`, `email`, `tag_list`, and `score` when the command was fuzzy. `--ids-only` prints one integer per line. Neither flag changes which rows match. + +## Out of scope + +Duplicate detection (same email or phone on two contacts) needs a separate scan and a human merge step. Do not fold it into `search`. The existing duplicate-merge note stays the place for that design. + +Do not make `batch-update` printable as a search. It requires a mutation payload on purpose. diff --git a/docs/ticket_decomposition_knowledge_ingestion_plan.md b/docs/ticket_decomposition_knowledge_ingestion_plan.md new file mode 100644 index 0000000..3769d12 --- /dev/null +++ b/docs/ticket_decomposition_knowledge_ingestion_plan.md @@ -0,0 +1,788 @@ +# Ticket Decomposition & Knowledge Ingestion Implementation Plan + +**Date:** 2026-07-09 +**Project repo:** `/home/jason/projects/redmine` +**Primary vault:** `/home/jason/lilaBuild/vaults/obsidian-ldr` +**Primary Redmine project to start:** `customer-service` +**Production Redmine:** `https://rm.ldrprep.com` +**Onyx vault URL prefix:** `https://onyx.int.ldrprep.com/projects/obsidian-ldr/` + +## Goal + +Build a ticket decomposition process that converts meaningful Redmine tickets into durable organizational memory: timeline event notes, entity/contact notes, Redmine contact enrichment, cross-links, tags, and gbrain-ingestable vault content. + +The immediate wedge is Redmine tickets, especially `customer-service`, but the final interface should support: + +```text +decompose +``` + +where `` may be: + +- Redmine issue ID / Redmine project query +- email text +- PDF/document file +- pasted text +- cached ticket JSON + +## Scope + +This plan is about **knowledge ingestion and decomposition**, not spam closure. It should build on the ticket sync substrate, not replace it. + +Existing substrate to reuse: + +- `/home/jason/projects/redmine/redmine_tickets.py` +- `/home/jason/projects/redmine/redmine_contacts.py` +- Redmine issue API with `include=journals,helpdesk` +- Obsidian LDR vault +- Onyx browser links +- gbrain source/index tooling + +--- + +## Ticket Buckets Worth Decomposing + +A ticket should be eligible for decomposition if it appears to fit one or more of these buckets: + +### 1. Definite customer ticket with meaningful event + +Customer ticket where: + +- a decision was made +- a complaint was involved +- a new project started +- a workflow/process implication was exposed +- an account/service/pricing change happened + +Examples: + +- customer complaint about missing photos, missing package, wrong condition, wrong item +- pricing/service inquiry that caused a meaningful answer or decision +- customer onboarding/project kickoff +- customer-specific exception or policy decision + +### 2. Durable purchase receipt + +Receipt about a durable item or business asset worth remembering. + +Examples: + +- equipment +- tools +- technology/hardware +- warehouse/office asset +- anything likely to matter for warranty, accounting, maintenance, or future context + +### 3. Regular utility / recurring service receipt or invoice + +Receipt/invoice/payment confirmation about a regular service. + +Examples: + +- utilities +- insurance +- phone/VoIP +- shipping/billing services +- cloud/SaaS subscriptions +- recurring vendors + +This may not need rich event notes every time, but should enrich vendor/entity records and optionally create monthly financial/operations timeline entries. + +### 4. Partnership / inbound opportunity / buy-from-us request + +Not a customer and not a sales person trying to sell us something, but someone proposing a relationship or wanting to buy something **from us**. + +Examples: + +- buy Gaylords +- buy liquidation +- advertise/affiliate/blog partnership +- ask about partnership or marketplace cooperation +- use LDR as a source/supplier + +These should become opportunity/entity notes, not spam. + +### 5. Any meaningful operational event + +Catch-all for tickets that refer to a meaningful event: + +- policy change +- operational incident +- vendor problem +- staff/customer decision +- process drift or known-failure-method recurrence +- new obligation or future follow-up + +--- + +## Non-Goals / Safety Boundaries + +1. Do not ingest every Redmine ticket into the vault as a polished note. Use cache/archive for raw data; decompose only meaningful tickets. +2. Do not overwrite existing entity pages destructively. +3. Do not invent facts. Use evidence blocks and confidence levels. +4. Do not send customer-visible Helpdesk replies. +5. Do not update Redmine contacts until the proposed changes are explicit and reviewable. +6. Do not let spam-cleanup closures trigger decomposition unless the ticket also has meaningful non-spam content. + +--- + +## Vault Information Architecture + +### Timeline root + +Create a new timeline tree: + +```text +/home/jason/lilaBuild/vaults/obsidian-ldr/Timeline/ +``` + +Monthly folders: + +```text +Timeline/2026-07/ +Timeline/2026-08/ +``` + +Monthly index: + +```text +Timeline/2026-07/index.md +``` + +Event notes: + +```text +Timeline/2026-07/2026-07-09_customer-name-short-subject.md +``` + +Filename rules: + +- kebab-case subject slug +- no spaces +- prefix `YYYY-MM-DD_` +- keep short but human-readable + +Example: + +```text +Timeline/2026-07/2026-07-09_repackify-gaylord-inquiry.md +Timeline/2026-07/2026-07-09_u2wear-returns-provider-followup.md +Timeline/2026-07/2026-07-09_cyberstorm-order-return-confirmation.md +``` + +### Monthly index format + +`Timeline/YYYY-MM/index.md` should contain: + +```markdown +# YYYY-MM Timeline Index + +## Key events + +- [[2026-07-09_repackify-gaylord-inquiry]] — Partnership / buy-from-us inquiry. Redmine #41424. +- [[2026-07-09_u2wear-returns-provider-followup]] — Existing/prospective customer follow-up. Redmine #41448. + +## Decisions + +- ... + +## Complaints / service failures + +- ... + +## Purchases / receipts + +- ... + +## Partnerships / opportunities + +- ... +``` + +### Entity notes + +Use the existing vault `entities/` tree as the default entity home. + +Proposed paths: + +```text +entities/people/.md +entities/organizations/.md +entities/concepts/.md +entities/places/.md +``` + +If the repo prefers flat `entities/.md`, the first implementation can create flat notes and later migrate. But the decomposition code should centralize path decisions in one function, e.g. `entity_note_path(entity)`. + +### Entity note frontmatter + +Minimum frontmatter for Redmine contacts: + +```yaml +--- +type: organization +name: RePackify +redmine_contact_id: 4470 +redmine_contact_url: https://rm.ldrprep.com/contacts/4470 +onyx_url: https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md +tags: + - redmine-contact + - maybe-customer + - partnership-opportunity +aliases: + - RePackify +--- +``` + +For a person: + +```yaml +--- +type: person +name: Sarah Johnson +organization: [[repackify]] +redmine_contact_id: 4470 +redmine_contact_url: https://rm.ldrprep.com/contacts/4470 +onyx_url: https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/people/sarah-johnson.md +tags: + - redmine-contact + - contact-person +--- +``` + +### Entity body format + +```markdown +# Entity Name + +## Summary + +Brief factual summary with confidence level. + +## Redmine links + +- Contact: https://rm.ldrprep.com/contacts/4470 +- Related tickets: + - https://rm.ldrprep.com/issues/41424 + +## Timeline + +- [[Timeline/2026-07/2026-07-09_repackify-gaylord-inquiry]] + +## Known facts + +- Fact with source link. + +## Open questions + +- Question that needs human/agent follow-up. +``` + +--- + +## Event Note Format + +Each event note should be evidence-first and dialectic. + +```markdown +--- +type: timeline-event +date: 2026-07-09 +source_type: redmine-ticket +redmine_issue_id: 41424 +redmine_url: https://rm.ldrprep.com/issues/41424 +helpdesk_contact_id: 4470 +entities: + - [[repackify]] +tags: + - timeline + - redmine + - partnership-opportunity + - buy-from-us +confidence: medium +--- + +# 2026-07-09 — RePackify Gaylord Inquiry + +## One-line summary + +RePackify asked whether LDR has used Gaylord boxes available to purchase. + +## Source evidence + +- Redmine: https://rm.ldrprep.com/issues/41424 +- From: support@repackify.com +- Subject: Gaylord inquiry-RePackify +- Key excerpt: + > We’re interested in purchasing used Gaylord boxes and wanted to see if you have any available. + +## Classification + +- Bucket: partnership / buy-from-us request +- Customer status: not confirmed customer +- Suggested contact tag: maybe-customer or partnership-opportunity + +## Dialectic analysis + +### Claim + +This is not spam; it is a potential inbound business/opportunity ticket. + +### Supporting evidence + +- The sender wants to buy something from LDR, not sell LDR a generic service. +- The request references a concrete asset class: used Gaylord boxes. + +### Counterpoints / uncertainty + +- It may be a one-off opportunistic buyer, not a durable customer relationship. +- Needs human judgment on whether LDR wants to sell Gaylords. + +### Decision / outcome + +- No final business decision captured yet. +- Needs follow-up if LDR wants a liquidation/Gaylord sale process. + +## Entities to update + +- [[repackify]] + +## Redmine contact enrichment proposal + +```json +{ + "contact_id": 4470, + "add_tags": ["maybe-customer", "partnership-opportunity", "buy-from-us"], + "obsidian_article_url": "https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md" +} +``` + +## Follow-up candidates + +- Decide whether LDR sells used Gaylords. +- If yes, define a standard response/process. +``` + +--- + +## Dialectic Reasoning Requirements + +Each decomposition should include: + +1. **Claim** — what seems to be true or meaningful. +2. **Evidence** — exact ticket excerpts, sender, date, Redmine link. +3. **Counterpoints** — why the interpretation might be wrong. +4. **Decision/outcome** — what was decided or what remains undecided. +5. **Recommended graph updates** — timeline note, entity notes, contact updates, tags. +6. **Confidence** — low/medium/high. + +This prevents the vault from becoming a pile of confident hallucinated summaries. A small mercy in a cruel world. + +--- + +## Redmine Contact Cross-Linking + +### Vault → Redmine + +Entity frontmatter should include: + +```yaml +redmine_contact_id: 4470 +redmine_contact_url: https://rm.ldrprep.com/contacts/4470 +``` + +### Redmine → Vault + +The Redmine contact should be updated with an Onyx URL, not an `obsidian://` link. + +Preferred URL format: + +```text +https://onyx.int.ldrprep.com/projects/obsidian-ldr/.md +``` + +Example: + +```text +https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md +``` + +Implementation detail: + +- If a Redmine contact custom field exists for `Obsidian Article`, use `redmine_contacts.py --set-custom-field`. +- If no suitable custom field exists, append a clearly delimited line to contact background/notes, e.g.: + +```text +Obsidian article: https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md +``` + +Do not guess custom field IDs. Discover available custom fields from `get_contact()` payloads first. + +--- + +## `decompose ` Architecture + +### KarpathyWiki plugin reference + +Jason identified the Obsidian `karpathywiki` plugin as a close match for the desired behavior. Inspection notes are saved at: + +```text +/home/jason/projects/redmine/docs/karpathywiki_plugin_inspection.md +``` + +Finding: the plugin does not expose a true standalone CLI. It registers Obsidian commands and its `WikiEngine` depends on Obsidian `App/Vault/TFile/metadataCache` APIs. However, its logic is highly relevant and portable as a design reference: + +- source analysis prompt / JSON schema +- entity vs concept decision tree +- alias handling +- duplicate/semantic entity resolution +- entity/concept/source page generation prompts +- contradiction and related-page fields +- graph-based query/retrieval ideas + +Recommended implementation path for Redmine decomposition: build a Python headless mirror integrated with `redmine_tickets.py`, using KarpathyWiki's prompts/schema as reference. Consider a TypeScript adapter later only if maintaining a Python mirror becomes too costly. + +### Compatibility contract with the Obsidian KarpathyWiki plugin + +The LDR vault already contains a plugin-managed wiki tree: + +```text +wiki/entities/ +wiki/concepts/ +wiki/sources/ +wiki/index.md +wiki/log.md +``` + +The headless Redmine decomposer must be a **companion**, not a competing owner of that tree. + +Default rules: + +1. Do **not** write into `wiki/` by default. +2. Do **not** edit `wiki/index.md` or `wiki/log.md`; those are plugin-owned. +3. Do **not** overwrite or reformat plugin-generated `wiki/entities/*`, `wiki/concepts/*`, or `wiki/sources/*` pages. +4. Write Redmine-derived timeline/event artifacts into LDR-owned paths: + - `Timeline/YYYY-MM/...` + - `entities/organizations/...` + - `entities/people/...` + - `Processes/Redmine/...` +5. Use Obsidian-compatible wikilinks so plugin-generated pages and headless-generated pages can link to each other. +6. If linking to plugin pages, use explicit paths such as `[[wiki/concepts/returns-management|returns management]]` rather than ambiguous bare names. +7. Use plugin-compatible frontmatter concepts where practical (`type`, `created`, `updated`, `tags`, `sources`, `confidence`, `contradictions`) but allow LDR-specific types such as `timeline-event` outside `wiki/`. +8. If we ever want the plugin itself to ingest Redmine-derived material, create a deliberate handoff/export file outside `wiki/` first and let the plugin ingest it through Obsidian; do not silently write into plugin-owned `sources/`. + +This keeps Obsidian’s KarpathyWiki plugin useful in the desktop vault while allowing Redmine decomposition to run headlessly/cron/agent-side without fighting plugin state. + +Concrete upstream files to mirror first: + +- `src/wiki/prompts/ingestion.ts` — extraction prompt / JSON schema +- `src/wiki/prompts/generation.ts` — page generation prompts +- `src/wiki/prompts/merge.ts` — merge/update triage prompts +- `src/wiki/source-analyzer.ts` — source analysis normalization/provenance behavior +- `src/wiki/page-factory.ts` — slug/alias/page merge behavior + +The first Python implementation should not attempt full parity. It should clone only the useful minimum: + +1. analyze source into JSON entities/concepts/key points/contradictions; +2. resolve entity slugs against existing vault files; +3. generate or update entity/event notes; +4. produce a reviewable contact-update proposal; +5. leave graph retrieval to Obsidian/gbrain for now. + +### Input abstraction + +Create a common normalized input model: + +```python +@dataclass +class DecompositionSource: + source_type: str # redmine-ticket, email, pdf, text, file + source_id: str | None + title: str + body: str + date: str | None + sender: str | None + redmine_issue_id: int | None + redmine_contact_id: int | None + attachments: list[Path] + metadata: dict[str, Any] +``` + +Adapters: + +- `from_redmine_issue(issue_json)` +- `from_email_text(raw_text)` +- `from_pdf(path)` +- `from_plain_text(text)` +- `from_file(path)` + +### Decomposition output model + +```python +@dataclass +class DecompositionPlan: + source: DecompositionSource + buckets: list[str] + timeline_events: list[TimelineEvent] + entities: list[EntityUpdate] + redmine_contact_updates: list[ContactUpdate] + tags: list[str] + gbrain_actions: list[GBrainAction] + confidence: str + review_required: bool +``` + +### Apply phases + +1. `decompose --dry-run` — produce JSON/Markdown plan only. +2. `decompose --write-vault` — write timeline/entity notes. +3. `decompose --update-redmine-contact` — update Redmine contact with Onyx article link/tags. +4. `decompose --gbrain-sync` — run/queue gbrain sync/index. + +Each phase should be separately idempotent. + +--- + +## CLI Proposal + +### Decompose a Redmine ticket + +```bash +./redmine_tickets.py \ + --base-url https://rm.ldrprep.com \ + --project customer-service \ + decompose \ + --issue-id 41424 \ + --dry-run +``` + +### Decompose from cached tickets matching a bucket + +```bash +./redmine_tickets.py \ + --cache-dir .cache/redmine_tickets_prod_meaningful_20260709 \ + --project customer-service \ + decompose-cached \ + --bucket partnership-opportunity \ + --dry-run +``` + +### Decompose arbitrary text + +```bash +./decompose.py text \ + --title "Supplier proposal" \ + --input /tmp/message.txt \ + --dry-run +``` + +### Write vault notes after review + +```bash +./redmine_tickets.py \ + --base-url https://rm.ldrprep.com \ + --project customer-service \ + decompose \ + --issue-id 41424 \ + --write-vault +``` + +--- + +## gbrain Integration + +Initial approach should be simple and explicit: + +1. Write/update Obsidian notes. +2. Run or queue gbrain sync for the LDR vault source. +3. Verify that the new notes are searchable. + +Commands need source discovery first: + +```bash +gbrain sources list --json +gbrain stats +``` + +Then likely: + +```bash +gbrain sync --source obsidian-ldr --no-embed --json +gbrain embed --stale +gbrain extract --stale +gbrain search "RePackify Gaylord inquiry" +``` + +Do not assume the exact source ID is `obsidian-ldr` until verified by `gbrain sources list --json`. + +--- + +## Implementation Phases + +### Phase 0 — Immediate schema/docs checkpoint + +**Goal:** Preserve this architecture and make future implementation straightforward. + +Tasks: + +1. Save this plan. +2. Link it from Redmine `HANDOFF.md`. +3. Optionally create an Obsidian process note at: + +```text +/home/jason/lilaBuild/vaults/obsidian-ldr/Processes/Redmine/ticket-decomposition-and-knowledge-ingestion.md +``` + +Definition of done: + +- Plan exists. +- Handoff links it. +- Next worker can start implementation without transcript context. + +### Phase 1 — Manual pilot decomposition for 3 tickets + +**Goal:** Prove note structure and cross-linking manually before automating. + +Suggested pilot tickets: + +- RePackify Gaylord inquiry (`#41424`) +- u2wear returns-provider follow-up (`#41448`) +- Snuggle Puppy / 3PL & FBA Returns (`#41504`) + +Tasks: + +1. Fetch each ticket with `include=journals,helpdesk`. +2. Write one timeline event note per ticket. +3. Create/update entity notes. +4. Prepare Redmine contact update payloads, dry-run first. +5. Review Onyx links. +6. Optionally run gbrain sync/search smoke test. + +Definition of done: + +- 3 timeline notes exist. +- Monthly index links them. +- Entity notes exist or are updated. +- Redmine contact update payloads are reviewed. +- No customer-visible Redmine replies are sent. + +### Phase 2 — Implement decomposition dry-run engine + +**Goal:** Generate decomposition plans from cached Redmine tickets without writing vault/Redmine. + +**Status — 2026-07-10:** First deterministic slice implemented in `redmine_tickets.py decompose`. It supports `--issue-id`, `--input-json`, and `--format json|markdown`. It produces dry-run plans for the three pilot fixtures and respects the KarpathyWiki compatibility boundary by planning writes outside `wiki/`. A KarpathyWiki-style structured extraction layer now cleans Helpdesk/forwarded-message noise and emits `analysis.summary`, `analysis.entities`, `analysis.concepts`, `analysis.key_points`, `analysis.contradictions`, `analysis.related_pages`, and `analysis.clean_text`. Tests pass (`23` tests). Current limitation: extraction is still deterministic/rule-backed rather than LLM-backed; next slice should add prompt-driven JSON extraction for broader ticket types before vault-write mode. + +Files: + +- Modify: `/home/jason/projects/redmine/redmine_tickets.py` +- Add tests: `/home/jason/projects/redmine/tests/test_ticket_decomposition.py` or extend `test_redmine_tickets.py` + +Tasks: + +1. Add `DecompositionSource` and `DecompositionPlan` models. +2. Add `source_from_redmine_issue(issue)`. +3. Add bucket classifier: + - customer-decision + - customer-complaint + - customer-new-project + - durable-purchase-receipt + - recurring-utility-receipt + - partnership-buy-from-us + - meaningful-event +4. Add dialectic plan generator. +5. Add JSON/Markdown dry-run output. +6. Add KarpathyWiki-inspired source cleanup and LLM JSON extraction for higher-quality summaries/entities. + +Definition of done: + +- Unit tests cover each bucket. +- Dry-run output includes source evidence, entities, proposed notes, and proposed contact updates. +- Output quality is comparable to the hand-written pilot notes for the first three fixtures. + +### Phase 3 — Implement vault writer + +**Goal:** Write idempotent timeline/entity notes. + +Tasks: + +1. Add path builder for `Timeline/YYYY-MM/YYYY-MM-DD_slug.md`. +2. Add monthly index updater. +3. Add entity path resolver. +4. Add frontmatter merge/update helper. +5. Add idempotency marker using source Redmine issue ID in frontmatter. + +Definition of done: + +- Re-running same decomposition does not duplicate notes or index bullets. +- Existing entity notes are updated, not overwritten. + +### Phase 4 — Implement Redmine contact updater + +**Goal:** Update Redmine contacts with Onyx article links and useful tags. + +Tasks: + +1. Discover contact custom fields. +2. Decide field for Obsidian/Onyx article URL. +3. Add dry-run contact update report. +4. Add `--apply-contact-updates` using `rebot` only when approved. +5. Verify contact after update. + +Definition of done: + +- Contact has vault article URL. +- Vault article has `redmine_contact_id`. +- Writes are impersonated appropriately. + +### Phase 5 — gbrain sync/index integration + +**Goal:** Make decomposed notes searchable in gbrain. + +Tasks: + +1. Discover LDR vault gbrain source ID. +2. Add optional command wrapper or instruction to run sync/embed/extract. +3. Verify search for a newly created entity/event. + +Definition of done: + +- gbrain search finds the decomposed event/entity. + +### Phase 6 — Generalize `decompose ` + +**Goal:** Support non-ticket inputs with the same decomposition outputs. + +Tasks: + +1. Add plain-text adapter. +2. Add email adapter. +3. Add PDF adapter using existing OCR/document pipeline if needed. +4. Keep output model the same. + +Definition of done: + +- Same event/entity/timeline/contact note logic works for non-ticket sources. + +--- + +## Recommended First Execution Step + +Do **Phase 1 manual pilot** before writing more automation. + +Use the 3 known “meaningful but not spam” tickets from the recent spam-cleanup work: + +```text +#41424 RePackify Gaylord inquiry +#41448 u2wear returns-provider follow-up +#41504 Snuggle Puppy / 3PL & FBA Returns +``` + +That gives us immediate signal on: + +- note format quality +- entity matching +- Onyx link usefulness +- Redmine contact update shape +- gbrain ingestion friction + +Only after that should we automate the decomposition engine. Otherwise we risk building a very elegant machine for generating annoying notes. We have enough of those already. diff --git a/docs/ticket_spam_cleanup_handoff_2026-07-09.md b/docs/ticket_spam_cleanup_handoff_2026-07-09.md new file mode 100644 index 0000000..9e384ba --- /dev/null +++ b/docs/ticket_spam_cleanup_handoff_2026-07-09.md @@ -0,0 +1,824 @@ +# Ticket-Centric Spam Cleanup Handoff — 2026-07-09 + +## Purpose + +This handoff captures the current state of the new ticket-centric Redmine maintenance substrate and the spam cleanup workflow for the `customer-service` project. + +The immediate business goal is to reduce visible spam clutter in `customer-service` without disrupting existing NMI production scripts or doing unsafe bulk Redmine writes. + +## Important Safety / Operating Rules + +1. **Do not modify NMI scripts right now.** Existing NMI cron scripts are reference material only. +2. **Dry-run first.** Redmine mutations require explicit `--apply`. +3. **Reads do not impersonate.** Ordinary sync/search/dry-run planning should use normal API access. +4. **Writes impersonate `rebot`.** Spam cleanup / ticket maintenance writes should use `X-Redmine-Switch-User: rebot` (Harry Botter), because Redmine activity logs should show automated cleanup under that account, not Jason. +5. **Treat impersonation like sudo.** Use it only where the write is supposed to happen under that agent account. +6. **LAN first.** The validated target so far is the LAN Redmine copy at `http://192.168.50.170`, project `customer-service`. +7. **Confirm status IDs before applying.** Existing evidence suggests `status_id=5` is closed, but verify before first real apply if uncertain. + +## Relevant Files + +- Tool: `/home/jason/projects/redmine/redmine_tickets.py` +- Tests: `/home/jason/projects/redmine/tests/test_redmine_tickets.py` +- Plan: `/home/jason/projects/redmine/docs/ticket_sync_tool_plan.md` +- General project handoff: `/home/jason/projects/redmine/HANDOFF.md` +- Helpdesk API include docs: `/home/jason/projects/redmine/docs/redmine_issue_api_helpdesk_include.md` +- Existing contact tool: `/home/jason/projects/redmine/redmine_contacts.py` + +## Why `redmine_tickets.py` Exists + +The earlier `redmine_contacts.py` contact conveyor is useful for contact enrichment and contact-level routing, but it is not the best first tool for clearing visible spam from `customer-service` because the UI pain is ticket-level: the top/open tickets themselves are clutter. + +`redmine_tickets.py` is the new ticket-centric substrate. It syncs enriched issue payloads, searches cached tickets, and runs idempotent ticket actions. + +## Modern Helpdesk Access + +Do **not** recreate RebotSlick `PullHelpdeskInfo` behavior. The old separate helpdesk pull/cache pattern is obsolete because the local Redmine Helpdesk plugin now supports: + +```text +GET /issues/:id.json?include=journals,helpdesk +``` + +This returns normal Redmine issue data plus embedded `helpdesk_ticket` metadata, including contact/email/source fields. + +## Current Architecture + +### Ticket payload cache + +Per-ticket JSON files are stored under a configurable cache directory, e.g.: + +```text +.cache/redmine_tickets_lan/customer-service/41788.json +.cache/redmine_tickets_lan/customer-service/41789.json +``` + +### Ticket/action state + +State is currently manifest-backed: + +```text +.cache/redmine_tickets_lan/manifest.json +``` + +But the code now separates payload cache from state storage: + +- `TicketCache` handles ticket JSON cache + delegates state operations. +- `ManifestTicketStateStore` is the current state backend. +- The abstraction is intentionally small so SQLite can replace the manifest later for thousands of tickets. + +Required state-store methods: + +```python +get_ticket_state(issue_id) +upsert_ticket_state(issue_id, fields) +has_action(issue_id, action_name) +record_action(issue_id, action_name, result, metadata=None) +``` + +## Implemented Commands + +### Sync enriched tickets + +```bash +cd /home/jason/projects/redmine + +./redmine_tickets.py \ + --base-url http://192.168.50.170 \ + --project customer-service \ + --cache-dir .cache/redmine_tickets_lan \ + sync \ + --status-id open \ + --max-tickets 10 \ + --order newest-updated-first +``` + +Behavior: +- fetches issue stubs/pages from Redmine +- fetches full issue payloads with `include=journals,helpdesk` +- writes only missing/updated tickets to cache +- second identical run skips unchanged tickets + +Verified output: + +```text +Sync complete: seen=10 fetched=10 skipped_cached=0 written=10 +Sync complete: seen=10 fetched=0 skipped_cached=10 written=0 +``` + +### Search cached tickets + +```bash +./redmine_tickets.py \ + --cache-dir .cache/redmine_tickets_lan \ + --project customer-service \ + search \ + --max-tickets 10 \ + --order newest-updated-first +``` + +Validated that this matches the UI top-open-ticket pattern and includes spam-looking tickets such as: + +```text +#41788 Up to 20% Off Select Fans & Blowers +#41789 Up to 20% Off Select Fans & Blowers +``` + +### Spam check cached tickets + +```bash +./redmine_tickets.py \ + --cache-dir .cache/redmine_tickets_lan \ + --project customer-service \ + spam-check +``` + +Validated result: + +```text +#41788 SPAM score=2 from=reply@globalindustrial.com Up to 20% Off Select Fans & Blowers +#41789 SPAM score=2 from=reply@globalindustrial.com Up to 20% Off Select Fans & Blowers +``` + +Record local idempotency only (no Redmine mutation): + +```bash +./redmine_tickets.py \ + --cache-dir .cache/redmine_tickets_lan \ + --project customer-service \ + spam-check \ + --apply +``` + +After this, a second `spam-check` checks 0 tickets because `spam-check` action markers are recorded. + +### Plan closure for spam tickets + +Dry-run: + +```bash +./redmine_tickets.py \ + --base-url http://192.168.50.170 \ + --project customer-service \ + --cache-dir .cache/redmine_tickets_lan \ + close-if-spam \ + --close-status-id 5 \ + --contact-ticket-limit 10 +``` + +Validated output: + +```text +#41788 close_payload={"notes": "Closed as likely spam by redmine_tickets.py close-if-spam.", "status_id": 5} contact_id=621 contact_spam=4/10 likely_spammer=True Up to 20% Off Select Fans & Blowers +#41789 close_payload={"notes": "Closed as likely spam by redmine_tickets.py close-if-spam.", "status_id": 5} contact_id=621 contact_spam=4/10 likely_spammer=True Up to 20% Off Select Fans & Blowers +Close-if-spam complete: planned=2 apply=False +Dry run only. Re-run with --apply to close Redmine issues and record idempotency markers. +``` + +No Redmine mutation has been applied yet. + +## Actual Closure Command (Pending Approval) + +After confirming `status_id=5` is the correct closed status on the LAN copy, this is the command that would close the two planned tickets and record idempotency markers: + +```bash +cd /home/jason/projects/redmine + +./redmine_tickets.py \ + --base-url http://192.168.50.170 \ + --project customer-service \ + --cache-dir .cache/redmine_tickets_lan \ + close-if-spam \ + --close-status-id 5 \ + --contact-ticket-limit 10 \ + --impersonate-user rebot \ + --apply +``` + +Expected write behavior: + +- sends `PUT /issues/:id.json` +- payload: + ```json + { + "issue": { + "status_id": 5, + "notes": "Closed as likely spam by redmine_tickets.py close-if-spam." + } + } + ``` +- includes header: + ```text + X-Redmine-Switch-User: rebot + ``` +- records local actions: + - `close-if-spam = closed` + - `contact-spam-review = likely-spammer` or `single-ticket-spam` + +## Tests / Verification + +Current test command: + +```bash +python3 -m unittest discover -s tests -p 'test_redmine_tickets.py' -v +``` + +Last verified: + +```text +Ran 11 tests +OK +``` + +Compile command: + +```bash +python3 -m py_compile redmine_tickets.py tests/test_redmine_tickets.py redmine_contacts.py +``` + +Last verified: OK. + +## 25-Ticket LAN Evaluation — 2026-07-09 + +Jason confirmed `status_id=5` is Closed. A fresh throwaway cache was used so old action markers would not hide results: + +```text +.cache/redmine_tickets_lan_25_eval +``` + +Commands run: + +```bash +./redmine_tickets.py \ + --base-url http://192.168.50.170 \ + --project customer-service \ + --cache-dir .cache/redmine_tickets_lan_25_eval \ + sync \ + --status-id open \ + --max-tickets 25 \ + --order newest-updated-first + +./redmine_tickets.py \ + --cache-dir .cache/redmine_tickets_lan_25_eval \ + --project customer-service \ + spam-check \ + --apply + +./redmine_tickets.py \ + --base-url http://192.168.50.170 \ + --project customer-service \ + --cache-dir .cache/redmine_tickets_lan_25_eval \ + close-if-spam \ + --close-status-id 5 \ + --contact-ticket-limit 10 +``` + +Verified output summary: + +```text +Sync complete: seen=25 fetched=25 skipped_cached=0 written=25 +Spam check complete: checked=25 apply=True +Close-if-spam complete: planned=5 apply=False +SUMMARY cached_tickets=25 +SUMMARY spam=5 not_spam=20 +``` + +Closure candidates produced by dry-run: + +```text +#41753 Automated unfulfillable FBA inventory removal notification — from fba-noreply@amazon.com — contact_id=51 — contact_spam=10/10 — likely_spammer=True +#41760 Celebrate America's 250th: Save Up To 40% Off — from reply@globalindustrial.com — contact_id=621 — contact_spam=4/10 — likely_spammer=True +#41761 Celebrate America's 250th: Save Up To 40% Off — from reply@globalindustrial.com — contact_id=621 — contact_spam=4/10 — likely_spammer=True +#41767 — — from enews-noreply@qnap.com — contact_id=3162 — contact_spam=10/10 — likely_spammer=True +#41776 Your favorite sale ENDS THURSDAY ⏳ — from noreply@mail.vistaprint.com — contact_id=3795 — contact_spam=10/10 — likely_spammer=True +``` + +Important finding from first run: the scorer conflated **spam/promotional clutter** with **automated operational notifications**. Example: `#41753` from `fba-noreply@amazon.com` is likely an automated Amazon/FBA operational notification, not normal spam. + +No Redmine close mutation was applied during this 25-ticket evaluation. + +## Refined Promotional-Junk Evaluation — 2026-07-09 + +The scorer was refined to target Jason's immediate goal: quickly and reliably close **obvious unsolicited promotional/sales junk**, such as Walmart/Uline/GlobalIndustrial/Vistaprint/QNAP-style promotional mail and lead-gen/sales outreach from people trying to sell LDR something without an established relationship. + +False-positive protections added: + +- `automated-operational`: Amazon/FBA/order/API/payment-style notifications are excluded from the default auto-close class. +- Customer/vendor operational threads are protected from staff-reply contamination: LDR staff-authored journal notes are no longer scored as spam evidence. +- Standalone `SEO` in customer operational context is not treated as a sales trigger. + +Regression tests now cover: + +- promotional sale email => `spam-promotional` +- cleaning-service sales pitch => `spam-promotional` +- Amazon/FBA notification => `automated-operational`, not spam +- real FBA returns inquiry with LDR pricing reply => not spam +- customer operational context mentioning SEO => not spam + +Fresh final cache: + +```text +.cache/redmine_tickets_lan_25_promotional_eval_final_1783600119 +``` + +Final verified summary: + +```text +Sync complete: seen=25 fetched=25 skipped_cached=0 written=25 +Close-if-spam complete: planned=6 apply=False +SUMMARY categories={'unknown-review': 17, 'automated-operational': 2, 'spam-promotional': 6} +``` + +Current safe close candidates from the final dry-run: + +```text +#41767 — from enews-noreply@qnap.com — QNAP newsletter/promo +#41776 Your favorite sale ENDS THURSDAY ⏳ — from noreply@mail.vistaprint.com +#41788 Up to 20% Off Select Fans & Blowers — from reply@globalindustrial.com +#41789 Up to 20% Off Select Fans & Blowers — from reply@globalindustrial.com +#41804 Do you serve e-commerce brands? — from d.elliott@fulfillmentviewpro.com — 3PL lead-gen/sales pitch +#41834 Japan’s “JC-STAR Level 1” — from enews-noreply@qnap.com — QNAP promotional/newsletter item +``` + +No Redmine close mutation was applied during this final evaluation. + +### LAN apply verification — 2026-07-09 + +Jason approved moving toward closing the obvious junk. The following command was run against the final 25-ticket cache: + +```bash +./redmine_tickets.py \ + --base-url http://192.168.50.170 \ + --project customer-service \ + --cache-dir .cache/redmine_tickets_lan_25_promotional_eval_final_1783600119 \ + close-if-spam \ + --close-status-id 5 \ + --contact-ticket-limit 10 \ + --impersonate-user rebot \ + --apply +``` + +Applied closures: + +```text +#41767 +#41776 +#41788 +#41789 +#41804 +#41834 +``` + +Idempotency check: + +```text +Close-if-spam complete: planned=0 apply=True +``` + +API verification after apply: + +```text +41767 status=Closed close_note_author=Harry B. close_note_found=True +41776 status=Closed close_note_author=Harry B. close_note_found=True +41788 status=Closed close_note_author=Harry B. close_note_found=True +41789 status=Closed close_note_author=Harry B. close_note_found=True +41804 status=Closed close_note_author=Harry B. close_note_found=True +41834 status=Closed close_note_author=Harry B. close_note_found=True +``` + +## Current Limitations / Known Next Improvements + +1. `manifest.json` is acceptable for the current tracer, but SQLite should be considered once caches/actions cover thousands of tickets. +2. Spam scoring is still intentionally simple and reviewable. It now distinguishes `spam-promotional`, `automated-operational`, and `unknown-review`; future classes can be added as patterns emerge. +3. Contact-level spammer handling is only advisory right now. The tool reports `likely_spammer=True`, but does not yet tag the contact. +4. First real close apply for each newly expanded batch should happen on LAN only, then immediately re-sync/verify status and activity author. +5. The production Redmine path should remain out of scope until LAN write behavior is proven and reviewed. + +## 50-Ticket Refined Promotional-Junk Dry Run — 2026-07-09 + +Fresh cache: + +```text +.cache/redmine_tickets_lan_50_promotional_eval_refined_1783600628 +``` + +Verified summary: + +```text +Sync complete: seen=50 fetched=50 skipped_cached=0 written=50 +Close-if-spam complete: planned=9 apply=False +SUMMARY categories={'unknown-review': 32, 'spam-promotional': 9, 'automated-operational': 9} +``` + +Promotional close candidates: + +```text +#41685 Samsung promo — noreply_samsung@sea.samsung.com — "Jason, we’ve missed you. We just can't stop innovating." +#41687 QNAP promo — enews-noreply@qnap.com — "Qsync Now Supports File Backup on Mac" +#41698 Vistaprint promo — noreply@mail.vistaprint.com — "Life’s best moments, treasured forever" +#41705 Popmenu/restaurant promo — noreply@popmenu.com — "Take a break from cooking!" +#41722 3PL lead-gen/sales pitch — hello@offer.search3pl.com — "Live fulfilment leads waiting for you to connect" +#41738 Vistaprint promo — noreply@mail.vistaprint.com — "Up to 15% off top business products that’ll draw them in" +#41746 Business Journals events/nominations promo — replycincinnati@engaged.bizjournals.com — "Hi LDR, here are our upcoming events and nominations" +#41760 Global Industrial sale — reply@globalindustrial.com — "Celebrate America's 250th: Save Up To 40% Off" +#41761 Global Industrial sale — reply@globalindustrial.com — "Celebrate America's 250th: Save Up To 40% Off" +``` + +Operational/system notifications excluded from auto-close: + +```text +#41694 USPS eReceipt +#41704 2024 UNPAID INVOICES $3,413.55 +#41714 Undelivered Mail Returned to Sender +#41716 Critical security alert +#41726 UPS Invoice Now Available +#41753 Automated unfulfillable FBA inventory removal notification +#41765 Amazon Listings API changes +#41772 Cintas Payment Status +#41823 Amazon order confirmation +``` + +No Redmine mutation was applied during this 50-ticket dry run. + +### 50-ticket LAN apply verification — 2026-07-09 + +Jason approved applying the 9-candidate promotional set. The following command was run against the exact reviewed cache: + +```bash +./redmine_tickets.py \ + --base-url http://192.168.50.170 \ + --project customer-service \ + --cache-dir .cache/redmine_tickets_lan_50_promotional_eval_refined_1783600628 \ + close-if-spam \ + --close-status-id 5 \ + --contact-ticket-limit 10 \ + --impersonate-user rebot \ + --apply +``` + +Applied closures: + +```text +#41685 +#41687 +#41698 +#41705 +#41722 +#41738 +#41746 +#41760 +#41761 +``` + +Idempotency check: + +```text +Close-if-spam complete: planned=0 apply=True +``` + +API verification after apply: + +```text +41685 status=Closed close_note_author=Harry B. close_note_found=True +41687 status=Closed close_note_author=Harry B. close_note_found=True +41698 status=Closed close_note_author=Harry B. close_note_found=True +41705 status=Closed close_note_author=Harry B. close_note_found=True +41722 status=Closed close_note_author=Harry B. close_note_found=True +41738 status=Closed close_note_author=Harry B. close_note_found=True +41746 status=Closed close_note_author=Harry B. close_note_found=True +41760 status=Closed close_note_author=Harry B. close_note_found=True +41761 status=Closed close_note_author=Harry B. close_note_found=True +``` + +## Recommended Next Session Steps + +1. If Jason approves the current 27-candidate production 200-ticket set, apply closures on production using `--impersonate-user rebot` and the exact cache `.cache/redmine_tickets_prod_200_promotional_eval_1783612885`. +2. Optionally apply the 3 `maybe-customer` contact tags from the same cache. +3. Re-sync/inspect via Redmine API/UI and verify closed status, close-note author `Harry B.`, second-apply idempotency, and local action markers. +4. Continue with another bounded production dry-run batch. + +## Customer identification update — 2026-07-09 + +Jason confirmed the following contacts/patterns are customers and should be tagged `Customer`: + +```text +koki-takeda@shinkiya-inc.com +hello@bohoflow.co +hi5@bavariatrachten.com +maia@solride.org +mike@solride.org +info@u2wear.com +sales@u2wear.com +any Cyberstorm email +Esire-family emails +``` + +Important correction: do **not** treat incidental substrings such as `desiree` in Indeed/applicant email aliases as Esire-family customer matches. + +Production contact tagging was applied with `impersonate_user='rebot'`: + +```text +MATCHED_COUNT 31 +UPDATED_COUNT 25 +SKIPPED_ALREADY_CUSTOMER 6 +``` + +Two accidental applicant matches were corrected immediately: + +```text +4253 conversation-desireethrasher-customerreturnsinspect-0n3@indeedemail.com: removed Customer, left applicant +3934 conversation-desireethrasher-customerreturnsinspect-9nv@indeedemail.com: removed Customer, left applicant +``` + +After customer tagging, re-running the production 200 close plan still produced 27 candidates and kept `#41238` removed from the plan. + +## Customer-tag safety gate — 2026-07-09 + +New hard rule: before closing any ticket as spam, `redmine_tickets.py` fetches the Helpdesk contact and checks `tag_list`. If the contact has tag `customer`, the ticket is **not** closed as spam. Absence of the `customer` tag does **not** imply spam. + +Jason also clarified the closure risk model: + +- Closed tickets are not deleted. +- If the other party replies to a closed Helpdesk ticket, Redmine reopens it as `New Message`. +- Therefore, stale 30+ day tickets that match a verified `spam-promotional` profile are reasonable fast-close candidates once contact safety gates pass. +- However, if a ticket was previously closed and then reopened because the contact messaged again, do **not** auto-close it again. Escalate instead: bump priority and optionally set a due date 1–3 days out. + +A separate action now exists to tag contacts needing manual/customer review: + +```bash +./redmine_tickets.py \ + --base-url https://rm.ldrprep.com \ + --project customer-service \ + --cache-dir \ + tag-maybe-customer [--apply] +``` + +This adds `maybe-customer` to contacts with inbound/established-business inquiry evidence, using `--impersonate-user rebot` for writes. + +Tests now cover 20 cases, including the customer-tag skip and `maybe-customer` tag action. + +## Production 200-Ticket Dry Run With Customer Gate — 2026-07-09 + +Production cache: + +```text +.cache/redmine_tickets_prod_200_promotional_eval_1783612885 +``` + +Re-running the close plan after adding the contact `customer` safety gate produced: + +```text +Close-if-spam complete: planned=27 apply=False +``` + +The customer-tag gate removed `#41238` from the close plan, which was the suspicious `manager@sonisha.com` / `Account Queries` candidate. + +Current 27 close candidates: + +```text +#41098 +#41114 +#41121 +#41122 +#41127 +#41128 +#41153 +#41164 +#41167 +#41168 +#41201 +#41222 +#41226 +#41227 +#41242 +#41248 +#41269 +#41293 +#41297 +#41315 +#41331 +#41340 +#41341 +#41347 +#41368 +#41373 +#41850 +``` + +Maybe-customer dry-run produced 3 contact tag candidates: + +```text +contact_id=4470 source_issue_id=41424 tag_list=["maybe-customer"] +contact_id=3919 source_issue_id=41448 tag_list=["maybe-customer"] +contact_id=4476 source_issue_id=41504 tag_list=["maybe-customer"] +``` + +No production mutation was applied after adding this guard. + +## Production 100-Ticket Promotional-Junk Dry Run — 2026-07-09 + +Production URL: + +```text +https://rm.ldrprep.com +``` + +Fresh cache: + +```text +.cache/redmine_tickets_prod_100_promotional_eval_1783609390 +``` + +Verified summary: + +```text +Sync complete: seen=100 fetched=100 skipped_cached=0 written=100 +Close-if-spam complete: planned=20 apply=False +SUMMARY categories={'unknown-review': 59, 'automated-operational': 21, 'spam-promotional': 20} +``` + +Promotional close candidates: + +```text +#41397 Business Journals events/nominations promo — replycincinnati@engaged.bizjournals.com +#41430 Vistaprint sale — noreply@mail.vistaprint.com +#41431 Business Summit promo — replycincinnati@partner.bizjournals.com +#41657 Global Industrial Kee Safety promo — reply@globalindustrial.com +#41658 Global Industrial Kee Safety promo — reply@globalindustrial.com +#41685 Samsung promo — noreply_samsung@sea.samsung.com +#41687 QNAP promo — enews-noreply@qnap.com +#41698 Vistaprint promo — noreply@mail.vistaprint.com +#41705 Popmenu restaurant promo — noreply@popmenu.com +#41722 3PL lead-gen sales pitch — hello@offer.search3pl.com +#41738 Vistaprint promo — noreply@mail.vistaprint.com +#41746 Business Journals events/nominations promo — replycincinnati@engaged.bizjournals.com +#41760 Global Industrial sale — reply@globalindustrial.com +#41761 Global Industrial sale — reply@globalindustrial.com +#41767 QNAP newsletter/promo — enews-noreply@qnap.com +#41776 Vistaprint sale — noreply@mail.vistaprint.com +#41788 Global Industrial sale — reply@globalindustrial.com +#41789 Global Industrial sale — reply@globalindustrial.com +#41804 3PL lead-gen sales pitch — d.elliott@fulfillmentviewpro.com +#41834 QNAP promotional/newsletter item — enews-noreply@qnap.com +``` + +No production mutation was applied during this dry run. + +### Production apply verification — 2026-07-09 + +Jason approved applying the 20-candidate promotional set. The following command was run against the exact reviewed production cache: + +```bash +./redmine_tickets.py \ + --base-url https://rm.ldrprep.com \ + --project customer-service \ + --cache-dir .cache/redmine_tickets_prod_100_promotional_eval_1783609390 \ + close-if-spam \ + --close-status-id 5 \ + --contact-ticket-limit 10 \ + --impersonate-user rebot \ + --apply +``` + +Applied closures: + +```text +#41397 +#41430 +#41431 +#41657 +#41658 +#41685 +#41687 +#41698 +#41705 +#41722 +#41738 +#41746 +#41760 +#41761 +#41767 +#41776 +#41788 +#41789 +#41804 +#41834 +``` + +Idempotency check: + +```text +Close-if-spam complete: planned=0 apply=True +``` + +API verification after apply: + +```text +41397 status=Closed close_note_author=Harry B. close_note_found=True +41430 status=Closed close_note_author=Harry B. close_note_found=True +41431 status=Closed close_note_author=Harry B. close_note_found=True +41657 status=Closed close_note_author=Harry B. close_note_found=True +41658 status=Closed close_note_author=Harry B. close_note_found=True +41685 status=Closed close_note_author=Harry B. close_note_found=True +41687 status=Closed close_note_author=Harry B. close_note_found=True +41698 status=Closed close_note_author=Harry B. close_note_found=True +41705 status=Closed close_note_author=Harry B. close_note_found=True +41722 status=Closed close_note_author=Harry B. close_note_found=True +41738 status=Closed close_note_author=Harry B. close_note_found=True +41746 status=Closed close_note_author=Harry B. close_note_found=True +41760 status=Closed close_note_author=Harry B. close_note_found=True +41761 status=Closed close_note_author=Harry B. close_note_found=True +41767 status=Closed close_note_author=Harry B. close_note_found=True +41776 status=Closed close_note_author=Harry B. close_note_found=True +41788 status=Closed close_note_author=Harry B. close_note_found=True +41789 status=Closed close_note_author=Harry B. close_note_found=True +41804 status=Closed close_note_author=Harry B. close_note_found=True +41834 status=Closed close_note_author=Harry B. close_note_found=True +``` + +## 100-Ticket Refined Promotional-Junk Dry Run — 2026-07-09 + +Fresh cache: + +```text +.cache/redmine_tickets_lan_100_promotional_eval_refined_1783601115 +``` + +Verified summary: + +```text +Sync complete: seen=100 fetched=100 skipped_cached=0 written=100 +Close-if-spam complete: planned=9 apply=False +SUMMARY categories={'unknown-review': 68, 'spam-promotional': 9, 'automated-operational': 23} +``` + +Promotional close candidates: + +```text +#41227 Global Industrial clearance — reply@globalindustrial.com — "Clearance: Bestsellers At Best Prices" +#41297 Vistaprint sale — noreply@mail.vistaprint.com — "Up to 30% off select customer faves, like mugs, postcards & more" +#41340 Global Industrial AC/fans — reply@globalindustrial.com — "Advanced AC, Wall Fans, And Cooling Units" +#41373 TikTok Shop promo — taylor@shop.tiktok.com — "The simplest way to start selling this June" +#41397 Business Journals events/nominations promo — replycincinnati@engaged.bizjournals.com — "Hi LDR, here are our upcoming events and nominations" +#41430 Vistaprint sale — noreply@mail.vistaprint.com — "Email exclusive! Up to 40% off select top products ENDS TOMORROW" +#41431 Business Summit promo — replycincinnati@partner.bizjournals.com — "Regional Business Summit Returns August 5" +#41657 Global Industrial Kee Safety promo — reply@globalindustrial.com — "'Kee Safety' Modular & Durable Fall Prevention Equipment" +#41658 Global Industrial Kee Safety promo — reply@globalindustrial.com — "'Kee Safety' Modular & Durable Fall Prevention Equipment" +``` + +False-positive protections added/verified during this 100-ticket run: + +- VoIP voicemail / call transcription tickets are `automated-operational`, not promotional spam. +- Inbound buying/relationship inquiries like RePackify Gaylord purchase interest and u2wear returns-provider follow-up remain `unknown-review`, not auto-close. +- Receipts, bounces, security alerts, invoices, Amazon/FBA/order notices remain `automated-operational`, not auto-close. + +No Redmine mutation was applied during this 100-ticket dry run. + +### 100-ticket LAN apply verification — 2026-07-09 + +Jason approved applying the 9-candidate promotional set. The following command was run against the exact reviewed cache: + +```bash +./redmine_tickets.py \ + --base-url http://192.168.50.170 \ + --project customer-service \ + --cache-dir .cache/redmine_tickets_lan_100_promotional_eval_refined_1783601115 \ + close-if-spam \ + --close-status-id 5 \ + --contact-ticket-limit 10 \ + --impersonate-user rebot \ + --apply +``` + +Applied closures: + +```text +#41227 +#41297 +#41340 +#41373 +#41397 +#41430 +#41431 +#41657 +#41658 +``` + +Idempotency check: + +```text +Close-if-spam complete: planned=0 apply=True +``` + +API verification after apply: + +```text +41227 status=Closed close_note_author=Harry B. close_note_found=True +41297 status=Closed close_note_author=Harry B. close_note_found=True +41340 status=Closed close_note_author=Harry B. close_note_found=True +41373 status=Closed close_note_author=Harry B. close_note_found=True +41397 status=Closed close_note_author=Harry B. close_note_found=True +41430 status=Closed close_note_author=Harry B. close_note_found=True +41431 status=Closed close_note_author=Harry B. close_note_found=True +41657 status=Closed close_note_author=Harry B. close_note_found=True +41658 status=Closed close_note_author=Harry B. close_note_found=True +``` diff --git a/docs/ticket_sync_tool_plan.md b/docs/ticket_sync_tool_plan.md new file mode 100644 index 0000000..dffc829 --- /dev/null +++ b/docs/ticket_sync_tool_plan.md @@ -0,0 +1,191 @@ +# Ticket Sync Tool — Architecture & Implementation Plan + +**Status**: Phase 1–3 tracer implemented and LAN-smoke-tested (2026-07-09) +**Goal**: Build a reliable, incremental ticket sync + query foundation (`redmine_tickets.py`) that replaces the obsolete RebotSlick `PullHelpdeskInfo` pattern with the modern `include=helpdesk` capability. This becomes the core data layer for all future agentic ticket workflows (spam cleanup, enrichment, dropped-ball detection, etc.). + +**Primary target**: `customer-service` project (currently >1700 tickets). Tool must work for any project. + +**Key constraints** +- Do **not** modify any NMI components. +- Reuse existing `RedmineClient` and helpdesk methods from `redmine_contacts.py`. +- Leverage the `include=helpdesk` (and `include=journals,helpdesk`) patch already deployed. +- Cache must be idempotent and cheap to re-run. +- Tool must support “do nothing but sync what’s new” when run with no action. + +--- + +## 1. Core Requirements (from user spec) + +1. **Incremental sync** + - Recursively fetch tickets with optional criteria (`--updated-since`, `--category`, `--status`, `--project`, `--with-tag`, etc.). + - Store full enriched payload (issue + journals + helpdesk_ticket + messages) as local JSON. + - Only re-fetch when the ticket’s `updated_on` is newer than the cached file mtime (or use a small manifest). + +2. **No-op default behavior** + - Running with no filters/action does a safe, incremental sync of everything (or a sensible default window). + +3. **Filter/search mode** + - When filters are supplied but no action is chosen, behave as a powerful search/report tool (list matching tickets, optionally write JSONL). + +4. **Idempotent action layer (future)** + - Higher-level workers (spam-check, close, enrich, etc.) must be able to mark a ticket as “already processed” so repeated runs skip it. + - Suggested mechanism: sidecar manifest or lightweight SQLite table (`ticket_id`, `action_name`, `last_run`, `result`). + +5. **Helpdesk enrichment** + - Always request `include=journals,helpdesk` (or configurable) so every cached ticket already contains the rich contact/source metadata. + +--- + +## 2. Proposed Architecture + +### 2.1 Tool location & reuse +- New script: `/home/jason/projects/redmine/redmine_tickets.py` +- Reuses the existing `RedmineClient` class (especially `issue_with_helpdesk`, `get_issue`, list methods). +- Same argument patterns, dry-run safety, `--apply`, error handling, and completion scripts as `redmine_contacts.py`. + +### 2.2 Cache layout (local filesystem) +Recommended structure (under a configurable `--cache-dir`, default `/home/jason/projects/redmine/cache/tickets/` or `/tmp/redmine-tickets-cache/`): + +``` +cache/tickets/ +├── manifest.json # { "ticket_id": { "updated_on": "...", "last_synced": "...", "actions": {...} } } +├── customer-service/ +│ ├── 12345.json # full enriched payload +│ ├── 12346.json +│ └── ... +└── other-project/ + └── ... +``` + +- One JSON file per ticket (stable, human-inspectable, easy to diff). +- `manifest.json` tracks `updated_on` + per-action idempotency state. +- Simple mtime + `updated_on` comparison is sufficient for v1; manifest makes it robust. +- Implementation note: ticket payload storage and ticket/action state storage are separated in code. `ManifestTicketStateStore` is the current implementation, but `TicketCache` only depends on a small state-store method set (`get_ticket_state`, `upsert_ticket_state`, `has_action`, `record_action`). If the manifest becomes cumbersome with thousands of tickets, replace it with SQLite or another backend without rewriting sync/search/action logic. + +### 2.3 CLI structure (v1 scope) + +```bash +./redmine_tickets.py \ + --base-url http://192.168.50.170 \ + --project customer-service \ + [sync|search] \ + [--updated-since YYYY-MM-DD] \ + [--updated-before YYYY-MM-DD] \ + [--status open|closed|*] \ + [--category "Spam"] \ + [--max-tickets N] \ + [--order newest-updated-first] \ + [--cache-dir ...] \ + [--dry-run] [--apply] \ + [--json] [--jsonl] +``` + +**Commands** +- `sync` (default if omitted): Perform incremental fetch + cache update. +- `search`: Same selection logic, but only output matching tickets (no cache write unless `--write-cache`). + +**Safety** +- Dry-run by default for any write. +- `--apply` required to persist cache or perform mutations. + +### 2.4 Idempotency & action hooks (v1 + v2) + +- Manifest stores per-ticket action state: + ```json + "actions": { + "spam-check": { "last_run": "2026-07-09", "result": "not-spam" }, + "close-if-spam": { "last_run": "...", "result": "skipped" } + } + ``` +- Future workers read the manifest before acting and update it after. + +--- + +## 3. Implementation Phases + +### Phase 1 — Core sync + cache (implemented tracer) +- Implement `RedmineClient` enhancements if needed (list issues with `include=journals,helpdesk`, pagination). +- Basic `sync` command that: + - Accepts date/status/project filters. + - Fetches pages of issues. + - For each issue, writes `cache//.json` only if newer than manifest entry. + - Updates manifest. +- `--max-tickets` safety limit. +- Dry-run shows what would be fetched/updated. + +**Verified 2026-07-09**: `redmine_tickets.py sync` pulled 10 open `customer-service` tickets from LAN Redmine (`192.168.50.170`), wrote enriched per-ticket JSON, and a second identical run fetched 0 / skipped 10 unchanged cached tickets. + +### Phase 2 — Search / filter-only mode (implemented tracer) +- `search` subcommand that reuses the same selection engine. +- Output options: human table, `--json`, `--jsonl` (for downstream agents). +- Ability to pipe results into later tools. + +**Verified 2026-07-09**: cached search over the 10-ticket LAN cache listed the same newest open tickets seen in the UI, including promo candidates like `#41788` and `#41789`. + +### Phase 3 — Idempotency manifest + action skeleton (implemented tracer) +- Add manifest read/write helpers. +- Skeleton `--action` interface (initially just records “action X run on ticket Y”). +- Example action: `spam-check` (calls the logic we started in `spam_promo_scorer.py` but at ticket level). + +**Verified 2026-07-09**: `spam-check` identified `#41788` and `#41789` as SPAM based on promotional subject language, recorded local manifest idempotency with `--apply`, and a second `spam-check` run checked 0 tickets. `close-if-spam` dry-run planned closures for those two tickets and reviewed sender `contact_id=621`, finding 4/10 recent tickets scored as spam (`likely_spammer=True`). No Redmine close mutation has been applied yet. + +### Phase 4 — Spam cleanup worker (next) +- Concrete `spam-check` + `close-if-spam` actions. +- Rules based on sender patterns, subject language, lack of human replies, existing contact tags, etc. +- Produces a report + optional `--apply` to close tickets and add `spam` tag to contact. + +Preferred spam policy: +1. When a ticket is high-confidence spam, close that ticket. +2. Use `helpdesk_ticket.contact.id` to fetch/analyze the last 5–10 tickets from the same contact. +3. Decide whether the sender/contact appears to be a recurring spammer or whether only the specific ticket/email is spam. +4. Only mark/tag the contact as a spammer when the contact-level lookback supports that conclusion. +5. Record both `close-if-spam` and `contact-spam-review` idempotency markers so repeated runs do not re-close/re-review the same work. +6. Treat Redmine impersonation like `sudo`: do not use it for ordinary reads or dry-run planning, but all spam-cleanup write calls should impersonate `rebot` / “Harry Botter” so Redmine activity logs clearly attribute automated cleanup work. + +--- + +## 4. Non-Goals (for this phase) + +- Do not replace or modify any NMI cron scripts. +- Do not build full agent orchestration yet (this is the data layer). +- Do not implement complex NLP classifiers in v1 (keep scoring simple and reviewable). +- Cache does not need to be a database (filesystem + manifest is sufficient and inspectable). + +--- + +## 5. Success Criteria + +- Running the tool twice in a row with the same filters produces zero API calls on the second run (everything is cached and up-to-date). +- A filter-only search for “tickets updated in last 30 days with no human reply” returns results in < 5 seconds after initial sync. +- The same tool can be pointed at any project and produce usable enriched ticket JSON. +- Higher-level workers can be written as thin Python scripts that consume the cache/manifest without needing their own Redmine client logic. + +--- + +## 6. Next Immediate Action + +The implementation tracer is complete through dry-run `close-if-spam`. + +Next safe execution sequence: + +1. Confirm LAN issue `status_id=5` is the intended closed status. +2. Run `close-if-spam --apply` against the LAN copy only, impersonating `rebot` (the CLI default). +3. Re-sync or inspect the affected LAN tickets. +4. Verify: + - `#41788` and `#41789` are closed, + - Redmine activity author is `rebot` / Harry Botter, + - local state records `close-if-spam` and `contact-spam-review`, + - a second `close-if-spam --apply` does nothing. +5. Only then expand the batch size beyond the initial 10-ticket cache. + +Detailed handoff: `docs/ticket_spam_cleanup_handoff_2026-07-09.md`. + +--- + +**References** +- `docs/redmine_issue_api_helpdesk_include.md` (the key enabler) +- `redmine_contacts.py` (client patterns, filtering, dry-run UX) +- `docs/contact_triage_loop_*` (overall workflow philosophy) +- RebotSlick `RebotSlick.php` (historical cache idea — modernized) + +This plan deliberately keeps scope tight while giving us a clean, extensible foundation for the spam-reduction work that is currently blocking the customer-service UI. \ No newline at end of file diff --git a/docs/visual/helpdesk-ticket-rendering.md b/docs/visual/helpdesk-ticket-rendering.md new file mode 100644 index 0000000..628474f --- /dev/null +++ b/docs/visual/helpdesk-ticket-rendering.md @@ -0,0 +1,11 @@ +![[message (1).eml]]## Helpdesk ticket rendering notes + +**Ticket 41753** is used here as an example. + +![rendered helpdesk ticket 41753 with circled areas for annotation](./rendered-helpdesk-ticket-41753.png) + +## Ticket https://rm.ldrprep.com/issues/43025 +The resulting subject is just "Let's", and it seems like this is caused by UTF-8 characters. The actual email text is as follows: +```text +Subject: =?UTF-8?B?TGV04oCZcw==?= update your password +``` diff --git a/docs/visual/original-helpdesk-ticket-41753.eml b/docs/visual/original-helpdesk-ticket-41753.eml new file mode 100644 index 0000000..962f368 --- /dev/null +++ b/docs/visual/original-helpdesk-ticket-41753.eml @@ -0,0 +1,948 @@ +Delivered-To: contact@ldrprep.com +Received: by 2002:a05:6918:5408:b0:476:29de:ede8 with SMTP id s8csp5332841ysl; + Mon, 6 Jul 2026 09:49:14 -0700 (PDT) +X-Received: by 2002:a05:622a:4015:b0:51b:feee:b08f with SMTP id d75a77b69052e-51c73f516b5mr19056341cf.1.1783356554502; + Mon, 06 Jul 2026 09:49:14 -0700 (PDT) +ARC-Seal: i=1; a=rsa-sha256; t=1783356554; cv=none; + d=google.com; s=arc-20260327; + b=jXuQyGgK9DIPjBYAbzAJ20wdcyzgO5pw27qvr4DQZcBYquyPpTbDvGYJNhRRYZeuce + vq4r5c/lCmDLRVaYF23nVaq+mcj9xttCKoWMphIJB2nW9EwsDSiPBKRCNBkeOGxxoG6F + uFFkrLi952oy+B7oc/pW1PeXINIoMqa4+6Ggp0aTfrCj6rYVopN8AmpR1KF/gsKxoDy6 + G5q4nbmOH/QD9yNTHDgCRQKRbzFg8Qh1cXZJa0UQATPL+C91e/BbQyK4/TK4AGJrbvwx + 9AxR1asvqr6TLk2WkyWEbDa7sr38cK8YDBtEyL3U/4kJcJJ4kIQ25snOwPRE0T9tZv65 + LW1g== +ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; + h=feedback-id:bounces-to:mime-version:subject:message-id:to:reply-to + :from:date:dkim-signature:dkim-signature; + bh=R3n7y8rm5NjqL3jRteMIf7Tu+z0CzAY1J5Yk3jgKsq8=; + fh=EfWMslCv/1V8JdxM9Auar9N+FfH8Z6xRox18ZgotYNQ=; + b=gTQr3GOnpaPlTobB0OGtJ0D9XzXAIZOw6fckc5GbfhAx0YEUPO+WkAQA7Ikxh22emG + weiP4UEax/mOo0n+RryW3tH6lItdnKadqTYPhORRc++dHSeVpE32C/cPjuNgXnOwCtI5 + mb+9qJr/pTzyr0kBJTSvB8xw4KaD5zpktbnJ7FMml9wySGxhhNPs3brN5MW9oWgHuiTk + xPooIa/Dt3iNOBtmLlRNEyTtcU2OAnZ6AsafDvkkY79S/Oxv+rZP8h+DdKp8xbtEtFy9 + sLaNrAwyK47pT17yFm3f+AGOk4fG7KZNuJQqxj8mid3nwsN5GW9He7xoQUYDOxUo0ECX + XRUA==; + dara=google.com +ARC-Authentication-Results: i=1; mx.google.com; + dkim=pass header.i=@amazon.com header.s=pwdll6lpa66mxupg2aa3uosx2zizclvc header.b="06YN//A6"; + dkim=pass header.i=@amazonses.com header.s=224i4yxa5dv7c2xz3womw6peuasteono header.b=D4Muravu; + spf=pass (google.com: domain of 0100019f38557b1a-ce103552-25f4-4d40-a76a-d8f8cf98fa1b-000000@amazonses.com designates 76.223.143.195 as permitted sender) smtp.mailfrom=0100019f38557b1a-ce103552-25f4-4d40-a76a-d8f8cf98fa1b-000000@amazonses.com; + dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.com +Return-Path: <0100019f38557b1a-ce103552-25f4-4d40-a76a-d8f8cf98fa1b-000000@amazonses.com> +Received: from c143-195.smtp-out.amazonses.com (c143-195.smtp-out.amazonses.com. [76.223.143.195]) + by mx.google.com with ESMTPS id d75a77b69052e-51c41b26f47si99447201cf.88.2026.07.06.09.49.14 + for + (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); + Mon, 06 Jul 2026 09:49:14 -0700 (PDT) +Received-SPF: pass (google.com: domain of 0100019f38557b1a-ce103552-25f4-4d40-a76a-d8f8cf98fa1b-000000@amazonses.com designates 76.223.143.195 as permitted sender) client-ip=76.223.143.195; +Authentication-Results: mx.google.com; + dkim=pass header.i=@amazon.com header.s=pwdll6lpa66mxupg2aa3uosx2zizclvc header.b="06YN//A6"; + dkim=pass header.i=@amazonses.com header.s=224i4yxa5dv7c2xz3womw6peuasteono header.b=D4Muravu; + spf=pass (google.com: domain of 0100019f38557b1a-ce103552-25f4-4d40-a76a-d8f8cf98fa1b-000000@amazonses.com designates 76.223.143.195 as permitted sender) smtp.mailfrom=0100019f38557b1a-ce103552-25f4-4d40-a76a-d8f8cf98fa1b-000000@amazonses.com; + dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.com +DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; + s=pwdll6lpa66mxupg2aa3uosx2zizclvc; d=amazon.com; t=1783356554; + h=Date:From:Reply-To:To:Message-ID:Subject:MIME-Version:Content-Type; + bh=+oIkeaNXB+80Md33rZGcSEaAg+nS01oDB42PknjIXIk=; + b=06YN//A69cb10ZreMsauMMCcF/nQHOCjBwAbnJFQAC6+ijGUmCmpkGXHtRnTidQ0 + mQdNdy2Sx2E438pdWtouKSi5xuhUN+vNqit+YwMVd6a7IC0YuyLTAtRbBQinoYAr6Tl + UJKXEIVYvmanrUjMjMuGo2kf7pz9EpdosXCngYs7e3swBVESKnT8D109Ia+Kmpk7leD + J+VmxC3sBtQVzEvY5unzq027vIZdQZ0K2GAdpyAHWy0+BCcvesB9lowLj5nea/L8HOd + cJsPffRGUm2G6wQNQm6fo2QybowSdPCA18k/VCT+Iz2l2oyjzFkFmX1ycoKM7O/dXwT + iFfC9dkUGA== +DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; + s=224i4yxa5dv7c2xz3womw6peuasteono; d=amazonses.com; t=1783356554; + h=Date:From:Reply-To:To:Message-ID:Subject:MIME-Version:Content-Type:Feedback-ID; + bh=+oIkeaNXB+80Md33rZGcSEaAg+nS01oDB42PknjIXIk=; + b=D4MuravuFghRiehrC6vr6KgCKcqFQl99m0hY50EceMrsHg0fm+0TeZof3DJINLbn + HlVVLcr30EjkLgVPGuN4/zf5jlTqmFNQQLrMNITJguluM4N712Ih3dsGN31M06m1rBw + G5a4Wy34Ss11oIvUH7Bba+sanCQb8m2vsSO5m8as= +Date: Mon, 6 Jul 2026 16:49:14 +0000 +From: Fulfillment by Amazon Operations +Reply-To: Fulfillment by Amazon Operations +To: contact@ldrprep.com +Message-ID: <0100019f38557b1a-ce103552-25f4-4d40-a76a-d8f8cf98fa1b-000000@email.amazonses.com> +Subject: Automated unfulfillable FBA inventory removal notification +MIME-Version: 1.0 +Content-Type: multipart/alternative; + boundary="----=_Part_196911_132258034.1783356554001" +Bounces-to: RTE+NE-null-de34cA07089053HVB1V8PA3SYZ@sellernotifications.amazon.com +X-Space-Message-ID: 387030323632580 +X-Marketplace-ID: ATVPDKIKX0DER +X-Space-Notification-Type: PRODUCT_REMOVAL_ORDER_CREATED_AUTO +Feedback-ID: ::1.us-east-1.apt6UHTqs3880kGtYdHRf9uSLRw5TOiel8eOJZxLSu8=:AmazonSES +X-SES-Outgoing: 2026.07.06-76.223.143.195 + +------=_Part_196911_132258034.1783356554001 +Content-Type: text/html; charset=UTF-8 +Content-Transfer-Encoding: quoted-printable + + + + + += +=09 + + + + + +=09=09 +=09 + + + + + + + + + + + + +=20 + + + + +
+
+ + +
+ + +=09=09 +=09=09 +=09=09 + +
+ +
+
+ + + + +=09=09=09=09=09 +=09=09=09=09=09=09 +=09=09=09=09=09 +=09=09=09=09=09
+ =09 +=09=09=09=09=09=09=09=09 +=09=09=09=09=09=09=09=09=09 +=09=09=09=09=09=09=09=09=09=09 + + +
+=09=09=09=09=09=09=09=09=09=09=09 +=09=09=09=09=09=09=09=09=09=09=09 +=09=09=09=09=09=09=09=09=09=09=09 + + =09=09=09=09=09 + +
+ 3D"" +
+
+=09=09=09=09=09=09
+ + + +
+ + + + +=09=09 +=09=09
+ + + +=09 + +
+=09=09 +=09=09=09 +=09=09=09 +=09=09=09=09 +=09=09=09 +=09=09=09 +=09=09
+=09=09=09=09 +=09=09=09=09=09 +=09=09=09=09=09 + + + + + +=09=09=09=09=09 +=09=09=09=09=09 +=09=09=09=09
+
+=09Hello peleya,

The following automated unfulfillable FBA removal o= +rders have been created based on your automated settings. +=09
+=09=09 +=09=09=09=09 +=09=09 +=09=09 +=09=09=09
Removal order ID kFAkH567aC
+=09
+=09You can view the removals on Removal Order Details i= +n your seller account by using the Search tool.

The next removal ord= +er will be created as scheduled on July 20, 2026 in accordance with your se= +ttings if you have unfulfillable inventory at that time.

To make cha= +nges to the shipping address, frequency of your automated unfulfillable rem= +ovals, or other settings, go to Autom= +ated unfulfillable removal settings in your seller account. +=09

+=09Thank you.

+The Fulfillment by Amazon team

+Please note: This email was sent from a notification-only address that cann= +ot accept incoming email. Please do not reply to this email. +
+
+=09=09=09=09
+=09
+ + + + +=09=09=09=09=09 +=09=09=09=09=09 +=09=09=09=09=09 +=09=09=09=09=09 +=09=09=09=09=09 +=09=09=09=09=09
+=09=09=09=09 +=09=09=09=09=09 +=09=09=09=09=09 +=09=09=09=09=09 +=09=09=09=09=09=09 +=09=09=09=09=09 +=09=09=09=09=09
+ =09 +=09=09=09=09=09=09=09=09 +=09=09=09=09=09=09=09=09=09 +=09=09=09=09=09=09=09=09=09=09 + + +
+=09=09=09=09=09=09=09=09=09=09=09
+ =09 + Report an issue with this email + +

If you have any que= +stions visit: Seller Central

+=09=09=09=09=09=09=09=09=09=09=09 To change your email preferences visit= +: Notification Preferences

+=09=09=09=09=09=09=09=09=09=09=09=09=09=09=09=09=09=09Copyright 2026 Amazo= +n, Inc, or its affiliates. All rights reserved.
=20 +=09=09=09=09=09=09=09=09=09=09=09=09Amazon.com, 410 Terry Avenue North, Sea= +ttle, WA 98109-5210

+=09=09=09=09=09=09=09=09=09=09=09 +=09=09=09=09=09=09=09=09=09=09=09 + =20 + =20 + =20 +=09=09=09=09=09=09=09=09=09=09=09 + =09=09=09=09=09 + +
+
+=09=09=09=09=09=09
+=09=09=09=09=09 +=09=09=09=09=09
+=09=09=09=09=09 +=09=09=09=09=09 +=09=09=09=09=09 +=09=09=09=09=09 +=09=09=09=09=09 +=09=09=09=09=09
+=09=09=09=09 +=09=09=09=09=09
+=09=09=09=09=09 +=09=09=09 +=09=09=09
+ +
+
+

SPC-USAmazon-387030323632580

+
+------=_Part_196911_132258034.1783356554001-- diff --git a/docs/visual/rendered-helpdesk-ticket-41753.png b/docs/visual/rendered-helpdesk-ticket-41753.png new file mode 100644 index 0000000..d5fbc06 Binary files /dev/null and b/docs/visual/rendered-helpdesk-ticket-41753.png differ diff --git a/docs/visual/webview-helpdesk-ticket-41753.png b/docs/visual/webview-helpdesk-ticket-41753.png new file mode 100644 index 0000000..f32654a Binary files /dev/null and b/docs/visual/webview-helpdesk-ticket-41753.png differ diff --git a/fix_globals.py b/fix_globals.py new file mode 100644 index 0000000..20a9573 --- /dev/null +++ b/fix_globals.py @@ -0,0 +1,19 @@ +#!/usr/bin/env python3 +"""Fix global flag placement for subcommands like tui.""" + +with open("redmine_tickets.py") as f: + txt = f.read() + +# 1. Add --allow-prod-writes to the parent value options +old_parent = '_PARENT_VALUE_OPTIONS = frozenset({"--base-url", "--project", "--cache-dir", "--config"})' +new_parent = '_PARENT_VALUE_OPTIONS = frozenset({"--base-url", "--project", "--cache-dir", "--config", "--allow-prod-writes"})' +if old_parent in txt: + txt = txt.replace(old_parent, new_parent) + print("1. Updated _PARENT_VALUE_OPTIONS to include --allow-prod-writes") +else: + print("1. PARENT set change skipped (not exact match)") + +with open("redmine_tickets.py", "w") as f: + f.write(txt) + +print("Saved parent options update") diff --git a/fix_postparse.py b/fix_postparse.py new file mode 100644 index 0000000..572e7b7 --- /dev/null +++ b/fix_postparse.py @@ -0,0 +1,35 @@ +#!/usr/bin/env python3 +with open("redmine_tickets.py") as f: + txt = f.read() + +old = ''' normalized_argv = inject_default_sync_command(raw_argv) + parser = build_parser() + args = parser.parse_args(normalized_argv) + + cache = TicketCache(args.cache_dir, args.project) +''' + +new = ''' normalized_argv = inject_default_sync_command(raw_argv) + parser = build_parser() + args = parser.parse_args(normalized_argv) + + # Post-process common globals so subparsers (tui/browse/...) that re-declare them + # (to support "cmd --allow-prod-writes ...") don't leave them as None. + if getattr(args, "cache_dir", None) in (None, ""): + from pathlib import Path as _P + args.cache_dir = _P(".cache/redmine_tickets") + if getattr(args, "project", None) in (None, ""): + args.project = "customer-service" + if getattr(args, "base_url", None) in (None, ""): + args.base_url = os.getenv("REDMINE_BASE_URL", "https://rm.ldrprep.com") + + cache = TicketCache(args.cache_dir, args.project) +''' + +if old in txt: + txt = txt.replace(old, new) + with open("redmine_tickets.py", "w") as f: + f.write(txt) + print("Inserted post-parse normalizer for common globals") +else: + print("Anchor not found for post-parse fix") diff --git a/fix_syntax.py b/fix_syntax.py new file mode 100644 index 0000000..20e9704 --- /dev/null +++ b/fix_syntax.py @@ -0,0 +1,23 @@ +#!/usr/bin/env python3 +with open("redmine_tickets.py") as f: + txt = f.read() + +# Fix the missing ) after no-preview arg +old = ''' help="Disable fzf right-hand preview pane.", + if add_tui_parser is not None:''' +new = ''' help="Disable fzf right-hand preview pane.", + ) + + if add_tui_parser is not None:''' +if old in txt: + txt = txt.replace(old, new, 1) + print("Replaced missing ) block") +else: + print("Old not found, trying looser fixes") + # Ensure ) before the if + txt = txt.replace('pane.",\n if add', 'pane.",\n )\n\n if add') + +with open("redmine_tickets.py", "w") as f: + f.write(txt) + +print("Saved") diff --git a/fix_tui_registration.py b/fix_tui_registration.py new file mode 100644 index 0000000..8d8fad0 --- /dev/null +++ b/fix_tui_registration.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python3 +import re + +with open("redmine_tickets.py") as f: + txt = f.read() + +# Clean any previous broken insertions around the area +txt = re.sub( + r'(\s*)\s*if add_tui_parser is not None:\s*add_tui_parser\(subparsers\)\s*\n\s*\)', + r'\1)', + txt +) + +marker = ' help="Disable fzf right-hand preview pane.",\n )\n\n ocr_parser = subparsers.add_parser(' +insert = ''' help="Disable fzf right-hand preview pane.", + ) + + if add_tui_parser is not None: + add_tui_parser(subparsers) + + ocr_parser = subparsers.add_parser(''' + +if marker in txt: + txt = txt.replace(marker, insert, 1) + print("Inserted cleanly via marker") +else: + # Find by context and insert before ocr + ocr_pos = txt.find('ocr_parser = subparsers.add_parser(\n "ocr-rename"') + no_prev_pos = txt.find('--no-preview', 0) + if ocr_pos > 0 and no_prev_pos > 0 and ocr_pos > no_prev_pos: + # insert just before ocr + insert_block = ''' + if add_tui_parser is not None: + add_tui_parser(subparsers) + +''' + txt = txt[:ocr_pos] + insert_block + txt[ocr_pos:] + print("Inserted before ocr via fallback") + else: + print("Could not find good spot") + +with open("redmine_tickets.py", "w") as f: + f.write(txt) + +print("Done fixing registration") diff --git a/icons/box-icon.png b/icons/box-icon.png new file mode 100644 index 0000000..5f9ae52 Binary files /dev/null and b/icons/box-icon.png differ diff --git a/icons/box-isometric-pallet.png b/icons/box-isometric-pallet.png new file mode 100644 index 0000000..418ce01 Binary files /dev/null and b/icons/box-isometric-pallet.png differ diff --git a/icons/box-motion-pallet.png b/icons/box-motion-pallet.png new file mode 100644 index 0000000..bb70033 Binary files /dev/null and b/icons/box-motion-pallet.png differ diff --git a/icons/box-shield-pallet.png b/icons/box-shield-pallet.png new file mode 100644 index 0000000..cbfc384 Binary files /dev/null and b/icons/box-shield-pallet.png differ diff --git a/icons/box-shield-stacked.png b/icons/box-shield-stacked.png new file mode 100644 index 0000000..fc37d66 Binary files /dev/null and b/icons/box-shield-stacked.png differ diff --git a/icons/redmine-semantic-64px-conduit.png b/icons/redmine-semantic-64px-conduit.png new file mode 100644 index 0000000..8418dec Binary files /dev/null and b/icons/redmine-semantic-64px-conduit.png differ diff --git a/icons/redmine-semantic-64px-emblem.png b/icons/redmine-semantic-64px-emblem.png new file mode 100644 index 0000000..8b69d80 Binary files /dev/null and b/icons/redmine-semantic-64px-emblem.png differ diff --git a/icons/redmine-semantic-64px-magnifier.png b/icons/redmine-semantic-64px-magnifier.png new file mode 100644 index 0000000..41abb83 Binary files /dev/null and b/icons/redmine-semantic-64px-magnifier.png differ diff --git a/icons/redmine-semantic-conduit-badass.png b/icons/redmine-semantic-conduit-badass.png new file mode 100644 index 0000000..315a21b Binary files /dev/null and b/icons/redmine-semantic-conduit-badass.png differ diff --git a/icons/redmine-semantic-emblem-badass.png b/icons/redmine-semantic-emblem-badass.png new file mode 100644 index 0000000..7bd79d0 Binary files /dev/null and b/icons/redmine-semantic-emblem-badass.png differ diff --git a/icons/redmine-semantic-fast-professional-1.png b/icons/redmine-semantic-fast-professional-1.png new file mode 100644 index 0000000..af9af30 Binary files /dev/null and b/icons/redmine-semantic-fast-professional-1.png differ diff --git a/icons/redmine-semantic-fast-professional-2.png b/icons/redmine-semantic-fast-professional-2.png new file mode 100644 index 0000000..6f61d3a Binary files /dev/null and b/icons/redmine-semantic-fast-professional-2.png differ diff --git a/icons/redmine-semantic-fast-professional-3.png b/icons/redmine-semantic-fast-professional-3.png new file mode 100644 index 0000000..63d8574 Binary files /dev/null and b/icons/redmine-semantic-fast-professional-3.png differ diff --git a/icons/redmine-semantic-hybrid-badass.png b/icons/redmine-semantic-hybrid-badass.png new file mode 100644 index 0000000..846940f Binary files /dev/null and b/icons/redmine-semantic-hybrid-badass.png differ diff --git a/icons/request-icon.png b/icons/request-icon.png new file mode 100644 index 0000000..39aba45 Binary files /dev/null and b/icons/request-icon.png differ diff --git a/icons/request-manifest-tablet.png b/icons/request-manifest-tablet.png new file mode 100644 index 0000000..e1aaffe Binary files /dev/null and b/icons/request-manifest-tablet.png differ diff --git a/icons/request-route-buildings.png b/icons/request-route-buildings.png new file mode 100644 index 0000000..98ac877 Binary files /dev/null and b/icons/request-route-buildings.png differ diff --git a/icons/request-shield-grouped.png b/icons/request-shield-grouped.png new file mode 100644 index 0000000..7551132 Binary files /dev/null and b/icons/request-shield-grouped.png differ diff --git a/icons/request-shield-manifest.png b/icons/request-shield-manifest.png new file mode 100644 index 0000000..7ccb503 Binary files /dev/null and b/icons/request-shield-manifest.png differ diff --git a/queues.example.json b/queues.example.json new file mode 100644 index 0000000..5a3025f --- /dev/null +++ b/queues.example.json @@ -0,0 +1,22 @@ +{ + "queues": [ + { + "name": "unassigned", + "where": ["@unassigned"], + "max_tickets": 100, + "order": "newest-updated-first" + }, + { + "name": "triage-needed-5d", + "where": ["@triage-needed-5d"], + "max_tickets": 100, + "order": "newest-updated-first" + }, + { + "name": "helpdesk-email-eq-company", + "where": ["@helpdeskEmailMatchCompany"], + "max_tickets": 200, + "order": "newest-updated-first" + } + ] +} diff --git a/redMCP/README.md b/redMCP/README.md index 958f088..912913c 100644 --- a/redMCP/README.md +++ b/redMCP/README.md @@ -220,6 +220,22 @@ The stdio server supports both MCP framing styles used by clients in the wild: Responses mirror the detected input framing mode for compatibility. +### JSON-RPC response normalization (2026-07-10) + +When running under Hermes's current Python MCP client, JSON-RPC responses with +`result` as a top-level JSON array (for example `[]`) can trigger repeated +Pydantic validation errors in `mcp.client.stdio`, followed by reconnect loops. + +To prevent that log spam and reconnect churn, redMCP now normalizes list-style +JSON-RPC results at the dispatcher boundary: + +- associative arrays (objects) pass through unchanged +- list arrays are wrapped as `{ "items": [...] }` +- empty list results are wrapped as `{ "items": [] }` + +This keeps response payloads object-shaped for broad client compatibility while +preserving the underlying data. + For local testing, run the Streamable HTTP server: ```sh @@ -336,6 +352,24 @@ customer-visible mail require an explicit tool call such as `redmine_send_helpdesk_response` or `redmine_update_issue` with `send_helpdesk_email=true`. +All MCP tools also accept optional `impersonate_user` to run the call as a +specific Redmine login when the configured API account has permission to switch +users. When omitted, calls run as the configured API user. + +```json +{ + "name": "redmine_update_issue", + "arguments": { + "impersonate_user": "jason", + "issue_id": 39858, + "notes": "Replying as Jason via MCP" + } +} +``` + +This applies to the entire tool call, including multi-step actions such as +`redmine_update_issue` with `send_helpdesk_email=true`. + Run the local no-network query normalizer checks with: ```sh diff --git a/redMCP/app/McpDispatcher.php b/redMCP/app/McpDispatcher.php index 33a9e55..4a132b8 100644 --- a/redMCP/app/McpDispatcher.php +++ b/redMCP/app/McpDispatcher.php @@ -10,6 +10,11 @@ use Throwable; final class McpDispatcher { private const PROJECT_ID_DESCRIPTION = 'Redmine project identifier or numeric id. If unsure, call redmine_find_project first and use project_id_to_use.'; + private const IMPERSONATE_USER_ARG = 'impersonate_user'; + private const IMPERSONATE_USER_SCHEMA = [ + 'type' => 'string', + 'description' => 'Optional Redmine login to impersonate for this tool call. Requires switch-user permission on the authenticated API account.', + ]; private const ISSUE_FIELD_ARGUMENT_KEYS = [ 'project_id', @@ -64,8 +69,9 @@ final class McpDispatcher try { $result = $this->dispatch($method, $params); + $normalizedResult = $this->normalizeJsonRpcResult($result); $this->logCall($context, $method, $params, true, $started); - return ['jsonrpc' => '2.0', 'id' => $id, 'result' => $result]; + return ['jsonrpc' => '2.0', 'id' => $id, 'result' => $normalizedResult]; } catch (Throwable $exception) { $this->logCall($context, $method, $params, false, $started, $exception->getMessage()); return [ @@ -79,6 +85,27 @@ final class McpDispatcher } } + /** + * @param array $result + * + * @return array + */ + private function normalizeJsonRpcResult(array $result): array + { + // JSON-RPC 2.0 allows any JSON value, but the Python MCP client used by + // Hermes currently validates response.result as an object/dict. + // Normalize list-style results so stdio parsing does not fail. + if ($result === []) { + return ['items' => []]; + } + + if (array_is_list($result)) { + return ['items' => $result]; + } + + return $result; + } + /** * @param array $params * @@ -348,6 +375,8 @@ final class McpDispatcher */ private function tool(string $name, string $description, array $properties, array $required = []): array { + $properties[self::IMPERSONATE_USER_ARG] = self::IMPERSONATE_USER_SCHEMA; + return [ 'name' => $name, 'description' => $description, @@ -369,109 +398,84 @@ final class McpDispatcher { $name = (string) ($params['name'] ?? ''); $arguments = is_array($params['arguments'] ?? null) ? $params['arguments'] : []; + $impersonateUser = $this->optionalStringArg($arguments, self::IMPERSONATE_USER_ARG); + unset($arguments[self::IMPERSONATE_USER_ARG]); - switch ($name) { - case 'redmine_list_projects': - $result = $this->redmine->listProjects(ListQueryNormalizer::listParams($arguments)); - break; - case 'redmine_find_project': - $result = $this->findProject($this->stringArg($arguments, 'query'), $this->intArg($arguments, 'limit', 10)); - break; - case 'redmine_get_project': - $result = $this->redmine->project($this->resolvedProjectIdArg($arguments, 'project_id', 'redmine_get_project'), $this->objectArg($arguments, 'params')); - break; - case 'redmine_list_project_memberships': - $result = $this->redmine->projectMemberships($this->resolvedProjectIdArg($arguments, 'project_id', 'redmine_list_project_memberships'), ListQueryNormalizer::listParams($arguments)); - break; - case 'redmine_list_users': - $result = $this->redmine->listUsers(ListQueryNormalizer::userParams($arguments)); - break; - case 'redmine_get_user': - $result = $this->redmine->user($this->intArg($arguments, 'user_id'), $this->objectArg($arguments, 'params')); - break; - case 'redmine_list_issues': - $result = $this->redmine->filterIssues(ListQueryNormalizer::issueFilters($this->resolvedProjectArgument($arguments, 'redmine_list_issues'))); - break; - case 'redmine_search': - $result = $this->redmine->search($this->stringArg($arguments, 'query'), ListQueryNormalizer::searchParams($this->resolvedProjectArgument($arguments, 'redmine_search'))); - break; - case 'redmine_search_issues': - $result = $this->redmine->searchIssues($this->stringArg($arguments, 'query'), ListQueryNormalizer::searchParams($this->resolvedProjectArgument($arguments, 'redmine_search_issues'))); - break; - case 'redmine_get_issue': - $result = $this->redmine->issue($this->intArg($arguments, 'issue_id'), $this->stringListArg($arguments, 'include', ['journals', 'attachments'])); - break; - case 'redmine_list_issue_relations': - $result = $this->redmine->listIssueRelations($this->intArg($arguments, 'issue_id')); - break; - case 'redmine_get_issue_relation': - $result = $this->redmine->issueRelation($this->intArg($arguments, 'relation_id')); - break; - case 'redmine_create_issue_relation': - $result = $this->redmine->createIssueRelation($this->intArg($arguments, 'issue_id'), $this->objectArg($arguments, 'fields')); - break; - case 'redmine_remove_issue_relation': - $result = ['ok' => $this->redmine->removeIssueRelation($this->intArg($arguments, 'relation_id'))]; - break; - case 'redmine_list_issue_children': - $filters = ListQueryNormalizer::issueFilters($arguments); - unset($filters['parent_id']); - $result = $this->redmine->listIssueChildren($this->intArg($arguments, 'issue_id'), $filters); - break; - case 'redmine_set_issue_parent': - $result = ['ok' => $this->redmine->setIssueParent($this->intArg($arguments, 'issue_id'), $this->intArg($arguments, 'parent_issue_id'))]; - break; - case 'redmine_clear_issue_parent': - $result = ['ok' => $this->redmine->clearIssueParent($this->intArg($arguments, 'issue_id'))]; - break; - case 'redmine_issue_with_helpdesk': - $result = $this->redmine->issueWithHelpdesk( - $this->intArg($arguments, 'issue_id'), - $this->intArg($arguments, 'message_limit', 100), - $this->stringListArg($arguments, 'include', ['journals', 'attachments']) - ); - break; - case 'redmine_create_issue': - $result = $this->redmine->createIssue($this->issueFieldsArg($arguments, 'redmine_create_issue')); - break; - case 'redmine_update_issue': - $result = ['ok' => $this->redmine->updateIssue($this->intArg($arguments, 'issue_id'), $this->issueFieldsArg($arguments, 'redmine_update_issue'), $this->objectArg($arguments, 'options'))]; - break; - case 'redmine_list_project_issue_categories': - $result = $this->redmine->listProjectIssueCategories($this->resolvedProjectIdArg($arguments, 'project_id', 'redmine_list_project_issue_categories')); - break; - case 'redmine_get_issue_category': - $result = $this->redmine->issueCategory($this->intArg($arguments, 'category_id')); - break; - case 'redmine_create_issue_category': - $result = $this->redmine->createIssueCategory($this->resolvedProjectIdArg($arguments, 'project_id', 'redmine_create_issue_category'), $this->objectArg($arguments, 'fields')); - break; - case 'redmine_update_issue_category': - $result = $this->redmine->updateIssueCategory($this->intArg($arguments, 'category_id'), $this->objectArg($arguments, 'fields')); - break; - case 'redmine_get_attachment': - $result = $this->redmine->attachment($this->intArg($arguments, 'attachment_id')); - break; - case 'redmine_upload_attachment': - $result = $this->redmine->uploadAttachment($arguments); - break; - case 'redmine_download_attachment': - $result = $this->redmine->downloadAttachment( - $this->intArg($arguments, 'attachment_id'), - $this->stringArg($arguments, 'path'), - $this->boolArg($arguments, 'include_base64', false), - $this->intArg($arguments, 'max_base64_bytes', 262144) - ); - break; - case 'redmine_update_attachment': - $result = $this->redmine->updateAttachment($this->intArg($arguments, 'attachment_id'), $this->objectArg($arguments, 'fields')); - break; - case 'redmine_send_helpdesk_response': - $result = $this->redmine->sendHelpdeskIssueResponse($this->intArg($arguments, 'issue_id'), $this->stringArg($arguments, 'content'), $this->objectArg($arguments, 'options')); - break; - default: - throw new RuntimeException('Unknown tool: ' . $name); - } + $result = $this->redmine->withImpersonation($impersonateUser, function () use ($name, $arguments) { + switch ($name) { + case 'redmine_list_projects': + return $this->redmine->listProjects(ListQueryNormalizer::listParams($arguments)); + case 'redmine_find_project': + return $this->findProject($this->stringArg($arguments, 'query'), $this->intArg($arguments, 'limit', 10)); + case 'redmine_get_project': + return $this->redmine->project($this->resolvedProjectIdArg($arguments, 'project_id', 'redmine_get_project'), $this->objectArg($arguments, 'params')); + case 'redmine_list_project_memberships': + return $this->redmine->projectMemberships($this->resolvedProjectIdArg($arguments, 'project_id', 'redmine_list_project_memberships'), ListQueryNormalizer::listParams($arguments)); + case 'redmine_list_users': + return $this->redmine->listUsers(ListQueryNormalizer::userParams($arguments)); + case 'redmine_get_user': + return $this->redmine->user($this->intArg($arguments, 'user_id'), $this->objectArg($arguments, 'params')); + case 'redmine_list_issues': + return $this->redmine->filterIssues(ListQueryNormalizer::issueFilters($this->resolvedProjectArgument($arguments, 'redmine_list_issues'))); + case 'redmine_search': + return $this->redmine->search($this->stringArg($arguments, 'query'), ListQueryNormalizer::searchParams($this->resolvedProjectArgument($arguments, 'redmine_search'))); + case 'redmine_search_issues': + return $this->redmine->searchIssues($this->stringArg($arguments, 'query'), ListQueryNormalizer::searchParams($this->resolvedProjectArgument($arguments, 'redmine_search_issues'))); + case 'redmine_get_issue': + return $this->redmine->issue($this->intArg($arguments, 'issue_id'), $this->stringListArg($arguments, 'include', ['journals', 'attachments'])); + case 'redmine_list_issue_relations': + return $this->redmine->listIssueRelations($this->intArg($arguments, 'issue_id')); + case 'redmine_get_issue_relation': + return $this->redmine->issueRelation($this->intArg($arguments, 'relation_id')); + case 'redmine_create_issue_relation': + return $this->redmine->createIssueRelation($this->intArg($arguments, 'issue_id'), $this->objectArg($arguments, 'fields')); + case 'redmine_remove_issue_relation': + return ['ok' => $this->redmine->removeIssueRelation($this->intArg($arguments, 'relation_id'))]; + case 'redmine_list_issue_children': + $filters = ListQueryNormalizer::issueFilters($arguments); + unset($filters['parent_id']); + return $this->redmine->listIssueChildren($this->intArg($arguments, 'issue_id'), $filters); + case 'redmine_set_issue_parent': + return ['ok' => $this->redmine->setIssueParent($this->intArg($arguments, 'issue_id'), $this->intArg($arguments, 'parent_issue_id'))]; + case 'redmine_clear_issue_parent': + return ['ok' => $this->redmine->clearIssueParent($this->intArg($arguments, 'issue_id'))]; + case 'redmine_issue_with_helpdesk': + return $this->redmine->issueWithHelpdesk( + $this->intArg($arguments, 'issue_id'), + $this->intArg($arguments, 'message_limit', 100), + $this->stringListArg($arguments, 'include', ['journals', 'attachments']) + ); + case 'redmine_create_issue': + return $this->redmine->createIssue($this->issueFieldsArg($arguments, 'redmine_create_issue')); + case 'redmine_update_issue': + return ['ok' => $this->redmine->updateIssue($this->intArg($arguments, 'issue_id'), $this->issueFieldsArg($arguments, 'redmine_update_issue'), $this->objectArg($arguments, 'options'))]; + case 'redmine_list_project_issue_categories': + return $this->redmine->listProjectIssueCategories($this->resolvedProjectIdArg($arguments, 'project_id', 'redmine_list_project_issue_categories')); + case 'redmine_get_issue_category': + return $this->redmine->issueCategory($this->intArg($arguments, 'category_id')); + case 'redmine_create_issue_category': + return $this->redmine->createIssueCategory($this->resolvedProjectIdArg($arguments, 'project_id', 'redmine_create_issue_category'), $this->objectArg($arguments, 'fields')); + case 'redmine_update_issue_category': + return $this->redmine->updateIssueCategory($this->intArg($arguments, 'category_id'), $this->objectArg($arguments, 'fields')); + case 'redmine_get_attachment': + return $this->redmine->attachment($this->intArg($arguments, 'attachment_id')); + case 'redmine_upload_attachment': + return $this->redmine->uploadAttachment($arguments); + case 'redmine_download_attachment': + return $this->redmine->downloadAttachment( + $this->intArg($arguments, 'attachment_id'), + $this->stringArg($arguments, 'path'), + $this->boolArg($arguments, 'include_base64', false), + $this->intArg($arguments, 'max_base64_bytes', 262144) + ); + case 'redmine_update_attachment': + return $this->redmine->updateAttachment($this->intArg($arguments, 'attachment_id'), $this->objectArg($arguments, 'fields')); + case 'redmine_send_helpdesk_response': + return $this->redmine->sendHelpdeskIssueResponse($this->intArg($arguments, 'issue_id'), $this->stringArg($arguments, 'content'), $this->objectArg($arguments, 'options')); + default: + throw new RuntimeException('Unknown tool: ' . $name); + } + }); $prepared = $this->redactSensitive($result); if ($this->sanitizeToolText) { @@ -721,6 +725,20 @@ final class McpDispatcher return $value; } + /** + * @param array $arguments + */ + private function optionalStringArg(array $arguments, string $key): ?string + { + if (!array_key_exists($key, $arguments)) { + return null; + } + + $value = trim((string) $arguments[$key]); + + return $value === '' ? null : $value; + } + /** * @param array $arguments */ diff --git a/redMCP/app/RedmineClient.php b/redMCP/app/RedmineClient.php index 9ee001d..ec815c9 100644 --- a/redMCP/app/RedmineClient.php +++ b/redMCP/app/RedmineClient.php @@ -22,6 +22,31 @@ final class RedmineClient $this->client = $client; } + /** + * Run a callback while impersonating an existing Redmine username. + * + * Redmine impersonation only applies when the authenticated API user has + * permission to switch users. + * + * @param callable $callback + * + * @return mixed + */ + public function withImpersonation(?string $username, callable $callback) + { + $normalized = is_string($username) ? trim($username) : ''; + if ($normalized === '') { + return $callback(); + } + + $this->client->startImpersonateUser($normalized); + try { + return $callback(); + } finally { + $this->client->stopImpersonateUser(); + } + } + public static function fromCredentials(string $url, string $apiKeyOrUsername, ?string $password = null): self { return new self(new NativeCurlClient($url, $apiKeyOrUsername, $password)); diff --git a/redMCP/bin/test-mcp-http-handler.php b/redMCP/bin/test-mcp-http-handler.php index 98fac01..551722d 100755 --- a/redMCP/bin/test-mcp-http-handler.php +++ b/redMCP/bin/test-mcp-http-handler.php @@ -54,7 +54,7 @@ try { assertContains('text/event-stream', $sse['headers'], 'SSE POST returns event stream content type', $assertions); assertContains('X-Accel-Buffering: no', $sse['headers'], 'SSE POST disables proxy buffering', $assertions); assertContains("event: message\n", $sse['body'], 'SSE POST emits a message event', $assertions); - assertContains('data: {"jsonrpc":"2.0","id":1,"result":[]}', $sse['body'], 'SSE POST emits JSON-RPC response data', $assertions); + assertContains('data: {"jsonrpc":"2.0","id":1,"result":{"items":[]}}', $sse['body'], 'SSE POST emits JSON-RPC response data', $assertions); $json = httpRequest( 'POST', diff --git a/redMCP/bin/test-redmine-structure.php b/redMCP/bin/test-redmine-structure.php index ba1d691..0d8fb02 100755 --- a/redMCP/bin/test-redmine-structure.php +++ b/redMCP/bin/test-redmine-structure.php @@ -19,6 +19,9 @@ final class RecordingClient implements Client, HttpClient /** @var array */ public array $requests = []; + /** @var array */ + public array $impersonationEvents = []; + /** @var array */ private array $responses = []; @@ -54,8 +57,15 @@ final class RecordingClient implements Client, HttpClient throw new RuntimeException('Unexpected vendor API call for ' . $name); } - public function startImpersonateUser(string $username): void {} - public function stopImpersonateUser(): void {} + public function startImpersonateUser(string $username): void + { + $this->impersonationEvents[] = 'start:' . $username; + } + + public function stopImpersonateUser(): void + { + $this->impersonationEvents[] = 'stop'; + } public function requestGet(string $path): bool { return false; } public function requestPost(string $path, string $body): bool { return false; } public function requestPut(string $path, string $body): bool { return false; } @@ -82,6 +92,8 @@ final class RedmineStructureTest $this->testMcpGetProjectShowsHelpfulErrorForAmbiguousHumanName(); $this->testMcpSearchSanitizesNoisyTextFields(); $this->testMcpSearchCanDisableTextSanitization(); + $this->testMcpToolCallSupportsImpersonationAndOmitsArgumentFromPayload(); + $this->testMcpToolCallStopsImpersonationOnFailure(); $this->testCreateRelationDefaultsToRelatesAndRequiresTarget(); $this->testAttachmentUploadSupportsPathAndBase64(); $this->testAttachmentUploadAcceptsPdfDataUrl(); @@ -347,6 +359,49 @@ final class RedmineStructureTest ); } + private function testMcpToolCallSupportsImpersonationAndOmitsArgumentFromPayload(): void + { + $http = new RecordingClient(); + $http->queueJson(['issue' => ['id' => 90210]], 201); + $dispatcher = new McpDispatcher(new RedmineClient($http)); + + $result = $this->callToolJson($dispatcher, 'redmine_create_issue', [ + 'impersonate_user' => 'jason', + 'project_id' => 'quality-tracker', + 'subject' => 'Impersonation create smoke', + 'description' => 'created via MCP test', + ]); + + $payload = $this->json($http->requests[0]['content']); + $this->assertSame(90210, $result['id'], 'impersonated create issue returns created issue payload'); + $this->assertSame(['start:jason', 'stop'], $http->impersonationEvents, 'impersonation wraps full tool call'); + $this->assertSame(false, isset($payload['issue']['impersonate_user']), 'impersonate_user is not forwarded as issue field'); + } + + private function testMcpToolCallStopsImpersonationOnFailure(): void + { + $http = new RecordingClient(); + $http->queueJson(['errors' => ['forbidden']], 500); + $dispatcher = new McpDispatcher(new RedmineClient($http)); + + $response = $dispatcher->handleMessage([ + 'jsonrpc' => '2.0', + 'id' => 1, + 'method' => 'tools/call', + 'params' => [ + 'name' => 'redmine_create_issue', + 'arguments' => [ + 'impersonate_user' => 'jason', + 'project_id' => 'quality-tracker', + 'subject' => 'Will fail intentionally', + ], + ], + ]); + + $this->assertSame(true, is_array($response) && isset($response['error']), 'failed tool call returns MCP error response'); + $this->assertSame(['start:jason', 'stop'], $http->impersonationEvents, 'impersonation is cleared when tool call fails'); + } + private function testAttachmentUploadSupportsPathAndBase64(): void { $path = sys_get_temp_dir() . '/redmcp-upload-test.txt'; @@ -507,6 +562,7 @@ final class RedmineStructureTest } $projectDescription = (string) $createIssueTool['inputSchema']['properties']['project_id']['description']; $this->assertStringContains('redmine_find_project', $projectDescription, 'project_id schema points agents to project resolver'); + $this->assertSame(true, isset($createIssueTool['inputSchema']['properties']['impersonate_user']), 'tool schema advertises impersonate_user'); } /** diff --git a/redmine_contacts.py b/redmine_contacts.py index 8cc69b8..0230ca1 100755 --- a/redmine_contacts.py +++ b/redmine_contacts.py @@ -7,20 +7,44 @@ import argparse import json import os import re +import subprocess import sys import time import urllib.error import urllib.parse import urllib.request from dataclasses import dataclass +from datetime import datetime, timezone from difflib import SequenceMatcher from pathlib import Path -from typing import Any +from typing import Any, Iterable DEFAULT_BASE_URL = "http://192.168.50.170" DEFAULT_PROJECT = "customer-service" DEFAULT_CACHE_DIR = Path(".cache/redmine_contacts") +SEARCH_EXAMPLES_DOC = "docs/redmine_contacts_search.md" +SEARCH_EXAMPLES = f"""\ +examples: + Name, email, or phone. Fuzzy; default min-score is 0.68. + Production must pass --base-url (the default host is the LAN copy): + ./redmine_contacts.py --base-url https://rm.ldrprep.com search "Sahar Daod" + + Exact tag. The positional query is still required; repeat the tag so the + tag text itself matches. --refresh reloads the cache first: + ./redmine_contacts.py --base-url https://rm.ldrprep.com \\ + search promotional-spam --with-tag promotional-spam --refresh + + Same tag lookup with no API call. The on-disk cache can be stale: + ./redmine_contacts.py --base-url https://rm.ldrprep.com \\ + search promotional-spam --with-tag promotional-spam --offline-cache + +This searches contacts, not tickets. Ticket search cannot see contact tags. +After you have an id: + bin/rtl search --status New --open-only --where 'helpdesk_ticket.contact_id == 4589' + +More examples: {SEARCH_EXAMPLES_DOC} +""" SEARCH_FIELDS = ( "first_name", "middle_name", @@ -46,6 +70,15 @@ UPDATE_FIELDS = { "email", "tag_list", } +BATCH_ORDER_CHOICES = ( + "as-is", + "oldest-updated-first", + "newest-updated-first", + "oldest-created-first", + "newest-created-first", + "id-asc", + "id-desc", +) class RedmineError(RuntimeError): @@ -64,6 +97,7 @@ class RedmineClient: path: str, params: dict[str, Any] | None = None, payload: dict[str, Any] | None = None, + impersonate_user: str | None = None, ) -> dict[str, Any]: url = self._url(path, params) body = None @@ -71,6 +105,8 @@ class RedmineClient: "Accept": "application/json", "X-Redmine-API-Key": self.api_key, } + if impersonate_user: + headers["X-Redmine-Switch-User"] = impersonate_user if payload is not None: body = json.dumps(payload).encode("utf-8") headers["Content-Type"] = "application/json" @@ -80,7 +116,7 @@ class RedmineClient: with urllib.request.urlopen(request, timeout=30) as response: raw = response.read() except urllib.error.HTTPError as exc: - detail = exc.read().decode("utf-8", errors="replace") + detail = (exc.read() or b'').decode("utf-8", errors="replace") raise RedmineError(f"HTTP {exc.code} from Redmine: {detail[:500]}") from exc except urllib.error.URLError as exc: raise RedmineError(f"Could not reach Redmine: {exc.reason}") from exc @@ -92,15 +128,22 @@ class RedmineClient: except json.JSONDecodeError as exc: raise RedmineError(f"Redmine returned non-JSON response from {url}") from exc - def fetch_contacts(self, limit: int = 100) -> list[dict[str, Any]]: + def fetch_contacts( + self, + limit: int = 100, + extra_params: dict[str, Any] | None = None, + ) -> list[dict[str, Any]]: contacts: list[dict[str, Any]] = [] offset = 0 total = None while total is None or offset < total: + params = {"limit": limit, "offset": offset} + if extra_params: + params.update(extra_params) data = self.request( "GET", f"/projects/{self.project}/contacts.json", - {"limit": limit, "offset": offset}, + params, ) page = data.get("contacts", []) contacts.extend(page) @@ -111,8 +154,47 @@ class RedmineClient: offset += received return contacts - def update_contact(self, contact_id: int, fields: dict[str, Any]) -> None: - self.request("PUT", f"/contacts/{contact_id}.json", payload={"contact": fields}) + def get_contact(self, contact_id: int) -> dict[str, Any]: + data = self.request("GET", f"/contacts/{contact_id}.json") + contact = data.get("contact") + if isinstance(contact, dict): + return contact + if isinstance(data, dict) and data.get("id") == contact_id: + return data + raise RedmineError(f"Redmine contact payload for id={contact_id} was missing 'contact'.") + + def get_issue(self, issue_id: int, includes: Iterable[str] | None = None) -> dict[str, Any]: + params: dict[str, Any] = {} + if includes: + params["include"] = ",".join(includes) + data = self.request("GET", f"/issues/{issue_id}.json", params=params or None) + issue = data.get("issue") + if isinstance(issue, dict): + return issue + if isinstance(data, dict) and data.get("id") == issue_id: + return data + raise RedmineError(f"Redmine issue payload for id={issue_id} was missing 'issue'.") + + def issue_with_helpdesk(self, issue_id: int, message_limit: int = 100) -> dict[str, Any]: + issue = self.get_issue(issue_id, includes=("journals", "attachments", "children", "relations", "changesets")) + response: dict[str, Any] = {"issue": issue} + try: + response["helpdesk_ticket"] = self.helpdesk_ticket_by_issue(issue_id) + except RedmineError as exc: + response["helpdesk_ticket_error"] = str(exc) + try: + response["helpdesk_messages"] = self.helpdesk_messages_by_issue(issue_id, message_limit) + except RedmineError as exc: + response["helpdesk_messages_error"] = str(exc) + return response + + def update_contact(self, contact_id: int, fields: dict[str, Any], impersonate_user: str | None = None) -> None: + self.request( + "PUT", + f"/contacts/{contact_id}.json", + payload={"contact": fields}, + impersonate_user=impersonate_user, + ) def helpdesk_ticket_by_issue(self, issue_id: int) -> dict[str, Any]: return self.request("GET", f"/helpdesk_search/issues/{issue_id}/ticket") @@ -134,65 +216,282 @@ class RedmineClient: return url +class ContactsHelpFormatter( + argparse.ArgumentDefaultsHelpFormatter, + argparse.RawDescriptionHelpFormatter, +): + """Show defaults, and keep example blocks in description/epilog.""" + + def main() -> int: - parser = argparse.ArgumentParser(description="Search and update Redmine CRM contacts.") - parser.add_argument("--base-url", default=os.getenv("REDMINE_BASE_URL", DEFAULT_BASE_URL)) - parser.add_argument("--project", default=os.getenv("REDMINE_PROJECT", DEFAULT_PROJECT)) + parser = argparse.ArgumentParser( + description="Search and update Redmine CRM contacts with safe dry-run-first workflows.", + epilog=( + "Contact search examples: ./redmine_contacts.py search --help\n" + f"Cookbook: {SEARCH_EXAMPLES_DOC}" + ), + formatter_class=ContactsHelpFormatter, + ) + parser.add_argument( + "--base-url", + default=os.getenv("REDMINE_BASE_URL", DEFAULT_BASE_URL), + help="Redmine base URL.", + ) + parser.add_argument( + "--project", + default=os.getenv("REDMINE_PROJECT", DEFAULT_PROJECT), + help="Redmine project identifier for contacts list calls.", + ) parser.add_argument("--cache", type=Path, help="Override the cache file path.") subparsers = parser.add_subparsers(dest="command", required=True) - fetch_parser = subparsers.add_parser("fetch", help="Fetch all visible contacts into the local cache.") - fetch_parser.add_argument("--limit", type=int, default=100) + fetch_parser = subparsers.add_parser( + "fetch", + help="Fetch contacts and cache them locally.", + description="Fetch contacts from Redmine, optionally filter for reporting, and cache results.", + ) + fetch_parser.add_argument( + "--limit", + type=int, + default=100, + help="Page size for Redmine pagination requests.", + ) + fetch_parser.add_argument( + "--cache-mode", + choices=("auto", "full", "filtered"), + default="auto", + help=( + "Cache mode: auto uses filtered when any contact filter is provided, otherwise full; " + "full always caches all fetched contacts; filtered caches only matches." + ), + ) + add_contact_filter_args(fetch_parser) - search_parser = subparsers.add_parser("search", help="Search cached contacts.") - search_parser.add_argument("query") - search_parser.add_argument("--refresh", action="store_true", help="Refresh cache before searching.") + search_parser = subparsers.add_parser( + "search", + help="Search contacts from cache.", + description=( + "Fuzzy-search cached CRM contacts by name, email, phone, company, or tag text.\n" + "This does not search tickets. Exact tags use --with-tag, not the query alone.\n" + f"Short examples are below. Full set: {SEARCH_EXAMPLES_DOC}" + ), + epilog=SEARCH_EXAMPLES, + formatter_class=ContactsHelpFormatter, + ) + search_parser.add_argument( + "query", + help=( + "Required fuzzy text (name, email, phone, company, or tag text). " + "Not an exact tag filter; pair with --with-tag for that." + ), + ) + search_parser.add_argument( + "--refresh", + action="store_true", + help="Reload the contact cache from Redmine before searching. Without this, tags added since the last fetch are invisible.", + ) search_parser.add_argument( "--offline-cache", action="store_true", help="Search only the local cache without requiring REDMINE_API_KEY.", ) - search_parser.add_argument("--limit", type=int, default=20) - search_parser.add_argument("--min-score", type=float, default=0.68) + search_parser.add_argument("--limit", type=int, default=20, help="Maximum number of matches to print.") + search_parser.add_argument( + "--min-score", + type=float, + default=0.68, + help="Minimum fuzzy match score threshold (0.0-1.0).", + ) + add_contact_filter_args(search_parser) - update_parser = subparsers.add_parser("update", help="Update safe fields on one contact.") - update_parser.add_argument("contact_id", type=int) + show_parser = subparsers.add_parser( + "show", + help="Show one contact JSON object by id.", + description="Print a contact JSON object from cache/API with optional cache refresh.", + ) + show_parser.add_argument("contact_id", type=int, help="Contact id.") + show_parser.add_argument( + "--source", + choices=("auto", "cache", "api"), + default="auto", + help="Read from cache, API, or auto-fallback to API when cache miss.", + ) + show_parser.add_argument("--refresh-cache", action="store_true", help="Refresh cache before reading.") + + update_parser = subparsers.add_parser( + "update", + help="Update safe fields on one contact.", + description="Build and optionally apply one-contact update payloads (dry run by default).", + ) + update_parser.add_argument("contact_id", type=int, help="Contact id.") update_parser.add_argument( "--set", action="append", default=[], metavar="FIELD=VALUE", - help="Set a contact field. Repeat for multiple fields.", + help="Set a contact field (allowed keys are restricted). Repeat for multiple fields.", + ) + update_parser.add_argument("--add-tag", action="append", default=[], metavar="TAG", help="Tag to add.") + update_parser.add_argument( + "--remove-tag", + action="append", + default=[], + metavar="TAG", + help="Tag to remove.", + ) + update_parser.add_argument( + "--set-custom-field", + action="append", + default=[], + metavar="NAME=VALUE", + help="Set one custom field by kebab-case name, preserving other custom fields.", + ) + update_parser.add_argument( + "--contact-source", + choices=("api", "cache", "auto"), + default="api", + help="Source used to read existing tags/custom-fields before building update payload.", ) update_parser.add_argument("--apply", action="store_true", help="Actually send the update.") + batch_update_parser = subparsers.add_parser( + "batch-update", + help="Apply one payload to a filtered contact batch.", + description=( + "Select contacts by filters, print one payload per contact, and only write when --apply is passed. " + "At least one filter is required as a safety guard." + ), + ) + batch_update_parser.add_argument( + "--max-contacts", + type=int, + default=20, + help="Maximum number of selected contacts to process.", + ) + batch_update_parser.add_argument( + "--order", + choices=BATCH_ORDER_CHOICES, + default="as-is", + help=( + "Deterministic selection order. Use oldest-updated-first/newest-updated-first for campaign sweeps." + ), + ) + batch_update_parser.add_argument( + "--source", + choices=("cache", "api"), + default="cache", + help="Read candidate contacts from cache or live API.", + ) + batch_update_parser.add_argument( + "--refresh-cache", + action="store_true", + help="Refresh the cache from API before selecting contacts.", + ) + add_contact_filter_args(batch_update_parser) + batch_update_parser.add_argument( + "--set", + action="append", + default=[], + metavar="FIELD=VALUE", + help="Set a contact field (allowed keys are restricted). Repeat for multiple fields.", + ) + batch_update_parser.add_argument("--add-tag", action="append", default=[], metavar="TAG", help="Tag to add.") + batch_update_parser.add_argument( + "--remove-tag", + action="append", + default=[], + metavar="TAG", + help="Tag to remove.", + ) + batch_update_parser.add_argument( + "--set-custom-field", + action="append", + default=[], + metavar="NAME=VALUE", + help="Set one custom field by kebab-case name, preserving other custom fields.", + ) + batch_update_parser.add_argument("--apply", action="store_true", help="Actually send updates.") + + loop_run_parser = subparsers.add_parser( + "loop-run", + help="Run an extensible per-contact FILTER-1/FILTER-2 triage loop.", + description=( + "Select contacts by filters, run a profile-defined FILTER-1 gate, fetch recent helpdesk issues, " + "call a FILTER-2 command worker, and apply generic queue-state tag updates only with --apply. " + "At least one contact filter is required as a safety guard." + ), + ) + loop_run_parser.add_argument("--profile", type=Path, required=True, help="JSON loop profile path.") + loop_run_parser.add_argument( + "--max-contacts", + type=int, + default=10, + help="Maximum number of selected contacts to process.", + ) + loop_run_parser.add_argument( + "--order", + choices=BATCH_ORDER_CHOICES, + default="oldest-updated-first", + help="Deterministic selection order for loop candidates.", + ) + loop_run_parser.add_argument( + "--source", + choices=("cache", "api"), + default="cache", + help="Read candidate contacts from cache or live API.", + ) + loop_run_parser.add_argument( + "--refresh-cache", + action="store_true", + help="Refresh the contact cache from API before selecting contacts.", + ) + add_contact_filter_args(loop_run_parser) + loop_run_parser.add_argument("--apply", action="store_true", help="Actually apply runner-owned tag updates.") + ticket_parser = subparsers.add_parser( "helpdesk-ticket", help="Fetch helpdesk ticket metadata for an issue.", + description="Fetch helpdesk ticket metadata for one issue id.", + ) + ticket_parser.add_argument("issue_id", type=int, help="Issue id.") + + issue_full_parser = subparsers.add_parser( + "issue-full", + help="Fetch a full issue object with journals/relations plus helpdesk metadata.", + description="Fetch full issue data and merge in helpdesk ticket + message context.", + ) + issue_full_parser.add_argument("issue_id", type=int, help="Issue id.") + issue_full_parser.add_argument( + "--message-limit", + type=int, + default=100, + help="Maximum helpdesk journal messages to include.", ) - ticket_parser.add_argument("issue_id", type=int) contact_issues_parser = subparsers.add_parser( "helpdesk-issues", help="List helpdesk issues for a contact.", + description="List helpdesk issue records for one contact id.", ) - contact_issues_parser.add_argument("contact_id", type=int) - contact_issues_parser.add_argument("--limit", type=int, default=100) + contact_issues_parser.add_argument("contact_id", type=int, help="Contact id.") + contact_issues_parser.add_argument("--limit", type=int, default=100, help="Maximum records to return.") issue_messages_parser = subparsers.add_parser( "helpdesk-messages", help="List helpdesk journal message metadata for an issue.", + description="List helpdesk journal-message metadata for one issue id.", ) - issue_messages_parser.add_argument("issue_id", type=int) - issue_messages_parser.add_argument("--limit", type=int, default=100) + issue_messages_parser.add_argument("issue_id", type=int, help="Issue id.") + issue_messages_parser.add_argument("--limit", type=int, default=100, help="Maximum records to return.") timeline_parser = subparsers.add_parser( "helpdesk-timeline", help="Show a contact's helpdesk ticket/message timeline.", + description="Return timeline-style helpdesk ticket/message metadata for one contact id.", ) - timeline_parser.add_argument("contact_id", type=int) - timeline_parser.add_argument("--limit", type=int, default=100) + timeline_parser.add_argument("contact_id", type=int, help="Contact id.") + timeline_parser.add_argument("--limit", type=int, default=100, help="Maximum records to return.") args = parser.parse_args() @@ -200,36 +499,120 @@ def main() -> int: if args.command == "fetch": client = client_from_args(args) contacts = client.fetch_contacts(limit=args.limit) + filtered_contacts = filter_contacts_from_args(contacts, args) cache_path = resolve_cache_path(args) - write_cache(cache_path, args, contacts) - print(f"Cached {len(contacts)} contacts in {cache_path}") + effective_cache_mode = resolve_fetch_cache_mode(args) + + cache_contacts = filtered_contacts if effective_cache_mode == "filtered" else contacts + write_cache(cache_path, args, cache_contacts) + + if effective_cache_mode == "filtered": + print( + f"Cached {len(filtered_contacts)} filtered contacts " + f"(from {len(contacts)} fetched) in {cache_path}" + ) + elif has_contact_filters(args): + print( + f"Cached {len(contacts)} contacts in {cache_path}. " + f"{len(filtered_contacts)} contacts currently match the supplied filters." + ) + else: + print(f"Cached {len(contacts)} contacts in {cache_path}") return 0 if args.command == "search": contacts = load_or_refresh(args) + contacts = filter_contacts_from_args(contacts, args) matches = search_contacts(contacts, args.query, args.min_score) print_matches(matches[: args.limit]) return 0 + if args.command == "show": + contact = resolve_contact_from_args(args) + print_json(contact) + return 0 + if args.command == "update": - fields = parse_update_fields(args.set) + client = client_from_args(args) + fields = build_update_fields(args, client) if not fields: - raise RedmineError("No fields supplied. Use --set FIELD=VALUE.") + raise RedmineError( + "No updates supplied. Use --set, --add-tag, --remove-tag, or --set-custom-field." + ) print(f"Contact {args.contact_id} update payload:") print(json.dumps({"contact": fields}, indent=2, sort_keys=True)) if not args.apply: print("Dry run only. Re-run with --apply to write to Redmine.") return 0 - client = client_from_args(args) client.update_contact(args.contact_id, fields) print("Update sent.") return 0 + if args.command == "batch-update": + if args.max_contacts <= 0: + raise RedmineError("--max-contacts must be >= 1.") + if not has_contact_filters(args): + raise RedmineError( + "batch-update requires at least one filter (date/tag/no-tags) to avoid sweeping all contacts." + ) + if not has_update_changes(args): + raise RedmineError( + "No updates supplied. Use --set, --add-tag, --remove-tag, or --set-custom-field." + ) + + contacts = load_contacts_for_batch(args) + selected = select_batch_contacts(contacts, args) + if not selected: + print("No contacts matched filters.") + return 0 + + client = client_from_args(args) + print(f"Selected {len(selected)} contact(s) for batch update.") + updated_count = 0 + for contact in selected: + contact_raw_id = contact.get("id") + if contact_raw_id is None: + raise RedmineError("Encountered contact without id in selected batch.") + contact_id = int(contact_raw_id) + fields = build_update_fields_for_contact(contact, args) + print(f"Contact {contact_id} update payload:") + print(json.dumps({"contact": fields}, indent=2, sort_keys=True)) + if args.apply: + client.update_contact(contact_id, fields) + updated_count += 1 + + if args.apply: + print(f"Applied updates to {updated_count} contact(s).") + else: + print("Dry run only. Re-run with --apply to write updates.") + return 0 + + if args.command == "loop-run": + outcomes = run_loop_from_args(args) + if not outcomes: + print("No contacts matched filters.") + return 0 + print(f"Processed {len(outcomes)} contact(s) with loop profile {args.profile}.") + for outcome in outcomes: + print( + f"Contact {outcome['contact_id']}: {outcome['action']} " + f"applied={outcome['applied']} reason={outcome['reason']}" + ) + print(json.dumps({"contact": outcome["fields"]}, indent=2, sort_keys=True)) + if not args.apply: + print("Dry run only. Re-run with --apply to write runner-owned tag updates.") + return 0 + if args.command == "helpdesk-ticket": client = client_from_args(args) print_json(client.helpdesk_ticket_by_issue(args.issue_id)) return 0 + if args.command == "issue-full": + client = client_from_args(args) + print_json(client.issue_with_helpdesk(args.issue_id, message_limit=args.message_limit)) + return 0 + if args.command == "helpdesk-issues": client = client_from_args(args) print_json(client.helpdesk_issues_by_contact(args.contact_id, args.limit)) @@ -258,6 +641,48 @@ def client_from_args(args: argparse.Namespace) -> RedmineClient: return RedmineClient(base_url=args.base_url, api_key=api_key, project=args.project) +def add_contact_filter_args(parser: argparse.ArgumentParser) -> None: + parser.add_argument( + "--created-after", + metavar="ISO_DATE", + help="Keep contacts created on/after this date or datetime (ISO format).", + ) + parser.add_argument( + "--created-before", + metavar="ISO_DATE", + help="Keep contacts created on/before this date or datetime (ISO format).", + ) + parser.add_argument( + "--updated-after", + metavar="ISO_DATE", + help="Keep contacts updated on/after this date or datetime (ISO format).", + ) + parser.add_argument( + "--updated-before", + metavar="ISO_DATE", + help="Keep contacts updated on/before this date or datetime (ISO format).", + ) + parser.add_argument( + "--with-tag", + action="append", + default=[], + metavar="TAG", + help="Require this exact tag (case-insensitive). Repeat to require every tag.", + ) + parser.add_argument( + "--without-tag", + action="append", + default=[], + metavar="TAG", + help="Exclude contacts containing tag(s). Repeat for multiple tags.", + ) + parser.add_argument( + "--no-tags", + action="store_true", + help="Only include contacts that have no tags.", + ) + + def load_or_refresh(args: argparse.Namespace) -> list[dict[str, Any]]: cache_path = resolve_cache_path(args) if args.refresh or not cache_path.exists(): @@ -267,12 +692,610 @@ def load_or_refresh(args: argparse.Namespace) -> list[dict[str, Any]]: return contacts if not args.offline_cache: require_api_key_for_cached_search() + data = read_cache_payload(cache_path, args) + return data["contacts"] + + +def filter_contacts_from_args(contacts: list[dict[str, Any]], args: argparse.Namespace) -> list[dict[str, Any]]: + return filter_contacts( + contacts, + created_after=getattr(args, "created_after", None), + created_before=getattr(args, "created_before", None), + updated_after=getattr(args, "updated_after", None), + updated_before=getattr(args, "updated_before", None), + with_tags=getattr(args, "with_tag", []) or [], + without_tags=getattr(args, "without_tag", []) or [], + no_tags=bool(getattr(args, "no_tags", False)), + ) + + +def has_contact_filters(args: argparse.Namespace) -> bool: + return any( + ( + bool(getattr(args, "created_after", None)), + bool(getattr(args, "created_before", None)), + bool(getattr(args, "updated_after", None)), + bool(getattr(args, "updated_before", None)), + bool(getattr(args, "with_tag", []) or []), + bool(getattr(args, "without_tag", []) or []), + bool(getattr(args, "no_tags", False)), + ) + ) + + +def resolve_fetch_cache_mode(args: argparse.Namespace) -> str: + mode = getattr(args, "cache_mode", "auto") + if mode in {"full", "filtered"}: + return mode + if mode != "auto": + raise RedmineError(f"Unsupported --cache-mode value: {mode!r}") + return "filtered" if has_contact_filters(args) else "full" + + +def load_loop_profile(profile_path: Path) -> dict[str, Any]: + with profile_path.open("r", encoding="utf-8") as handle: + profile = json.load(handle) + + if not isinstance(profile, dict): + raise RedmineError(f"Loop profile {profile_path} must contain a JSON object.") + if not profile.get("default_tag"): + raise RedmineError(f"Loop profile {profile_path} is missing required default_tag.") + if not isinstance(profile.get("filter_1"), dict): + raise RedmineError(f"Loop profile {profile_path} is missing required filter_1 object.") + if not isinstance(profile.get("filter_2"), dict): + raise RedmineError(f"Loop profile {profile_path} is missing required filter_2 object.") + + normalized = dict(profile) + normalized.setdefault("name", profile_path.stem) + normalized["_profile_dir"] = str(profile_path.resolve().parent) + normalized["remove_tags"] = list(normalized.get("remove_tags") or []) + + ticket_fetch = dict(normalized.get("ticket_fetch") or {}) + ticket_fetch.setdefault("fetch_full_issues", False) + ticket_fetch.setdefault("message_limit", 25) + ticket_fetch.setdefault("sanitize_text", True) + normalized["ticket_fetch"] = ticket_fetch + return normalized + + +def run_loop_from_args(args: argparse.Namespace) -> list[dict[str, Any]]: + if args.max_contacts <= 0: + raise RedmineError("--max-contacts must be >= 1.") + if not has_contact_filters(args): + raise RedmineError("loop-run requires at least one filter (date/tag/no-tags) to avoid sweeping all contacts.") + + profile = load_loop_profile(args.profile) + contacts = load_contacts_for_batch(args) + selected = select_batch_contacts(contacts, args) + if not selected: + return [] + + client = client_from_args(args) + runtime = { + "base_url": args.base_url, + "project": args.project, + "dry_run": not args.apply, + "helper_command": str(Path(__file__).resolve()), + } + return [run_loop_for_contact(client, profile, contact, runtime=runtime) for contact in selected] + + +def run_loop_filter1(profile: dict[str, Any], contact: dict[str, Any]) -> dict[str, Any]: + filter_config = profile.get("filter_1") or {} + if filter_config.get("type") == "builtin" and filter_config.get("name") == "fixed-ticket-limit": + limit = int(filter_config.get("limit", profile.get("ticket_fetch", {}).get("default_limit", 5))) + if limit <= 0: + return {"result": False, "reason": "fixed-ticket-limit was <= 0"} + return {"result": limit, "reason": "builtin fixed-ticket-limit"} + raise RedmineError(f"Unsupported FILTER-1 config: {filter_config!r}") + + +def run_loop_filter2( + profile: dict[str, Any], + bundle: dict[str, Any], + *, + runtime: dict[str, Any], +) -> dict[str, Any]: + filter_config = profile.get("filter_2") or {} + if filter_config.get("type") != "command": + raise RedmineError(f"Unsupported FILTER-2 config: {filter_config!r}") + + argv = filter_config.get("argv") or [] + if not isinstance(argv, list) or not all(isinstance(item, str) and item for item in argv): + raise RedmineError("FILTER-2 command requires a non-empty argv string list.") + + env = os.environ.copy() + env["REDMINE_BASE_URL"] = str(runtime.get("base_url") or DEFAULT_BASE_URL) + env["REDMINE_PROJECT"] = str(runtime.get("project") or DEFAULT_PROJECT) + env["REDMINE_CONTACTS_HELPER"] = str(runtime.get("helper_command") or Path(__file__).resolve()) + env["REDMINE_LOOP_DRY_RUN"] = "1" if runtime.get("dry_run", True) else "0" + + timeout = int(filter_config.get("timeout_seconds", 30)) + try: + completed = subprocess.run( + argv, + input=json.dumps(bundle), + text=True, + capture_output=True, + timeout=timeout, + check=False, + env=env, + cwd=str(filter_config.get("cwd") or profile.get("_profile_dir") or Path.cwd()), + ) + except subprocess.TimeoutExpired as exc: + raise RedmineError(f"FILTER-2 command timed out after {timeout}s: {argv!r}") from exc + except OSError as exc: + raise RedmineError(f"FILTER-2 command failed to start: {argv!r}: {exc}") from exc + + if completed.returncode != 0: + stderr = completed.stderr.strip() + raise RedmineError(f"FILTER-2 command exited {completed.returncode}: {stderr[:500]}") + + stdout = completed.stdout.strip() + if not stdout: + raise RedmineError("FILTER-2 command produced no JSON output.") + try: + result = json.loads(stdout) + except json.JSONDecodeError as exc: + raise RedmineError(f"FILTER-2 command produced non-JSON output: {stdout[:500]}") from exc + if not isinstance(result, dict) or "result" not in result: + raise RedmineError("FILTER-2 command output must be a JSON object containing result.") + return result + + +def run_loop_for_contact( + client: RedmineClient, + profile: dict[str, Any], + contact: dict[str, Any], + *, + runtime: dict[str, Any], +) -> dict[str, Any]: + contact_id = contact.get("id") + if contact_id is None: + raise RedmineError("Loop contact is missing id.") + contact_id = int(contact_id) + + filter1_result = run_loop_filter1(profile, contact) + if filter1_result.get("result") is False: + return apply_loop_contact_action( + client, + profile, + contact, + runtime=runtime, + action="default-tag", + reason=str(filter1_result.get("reason") or "FILTER-1 returned false"), + ) + + raw_ticket_limit = filter1_result.get("result") + if raw_ticket_limit is None: + raise RedmineError("FILTER-1 result must be false or an integer ticket limit.") + ticket_limit = int(raw_ticket_limit) + issues = fetch_loop_issues(client, profile, contact_id, ticket_limit) + bundle = build_loop_filter2_bundle(profile, contact, issues, runtime) + filter2_result = run_loop_filter2(profile, bundle, runtime=runtime) + if filter2_result.get("result"): + return apply_loop_contact_action( + client, + profile, + contact, + runtime=runtime, + action="remove-tags", + reason=str(filter2_result.get("summary") or "FILTER-2 returned true"), + ) + return apply_loop_contact_action( + client, + profile, + contact, + runtime=runtime, + action="default-tag", + reason=str(filter2_result.get("summary") or "FILTER-2 returned false"), + ) + + +def fetch_loop_issues( + client: RedmineClient, + profile: dict[str, Any], + contact_id: int, + ticket_limit: int, +) -> list[dict[str, Any]]: + ticket_fetch = profile.get("ticket_fetch") or {} + raw = client.helpdesk_issues_by_contact(contact_id, ticket_limit) + summaries = extract_helpdesk_issue_list(raw) + issues: list[dict[str, Any]] = [] + for summary in summaries[:ticket_limit]: + item = {"issue_summary": summary, "full_issue": None} + if ticket_fetch.get("fetch_full_issues"): + issue_id = summary.get("issue_id") or summary.get("id") + if issue_id is not None: + item["full_issue"] = client.issue_with_helpdesk( + int(issue_id), + message_limit=int(ticket_fetch.get("message_limit", 25)), + ) + issues.append(item) + return issues + + +def extract_helpdesk_issue_list(raw: Any) -> list[dict[str, Any]]: + if isinstance(raw, list): + return [item for item in raw if isinstance(item, dict)] + if not isinstance(raw, dict): + return [] + for key in ("issues", "helpdesk_issues", "tickets", "records"): + value = raw.get(key) + if isinstance(value, list): + return [item for item in value if isinstance(item, dict)] + return [] + + +def build_loop_filter2_bundle( + profile: dict[str, Any], + contact: dict[str, Any], + issues: list[dict[str, Any]], + runtime: dict[str, Any], +) -> dict[str, Any]: + return { + "profile": { + "name": profile.get("name"), + "default_tag": profile.get("default_tag"), + "remove_tags": profile.get("remove_tags") or [], + }, + "contact": contact, + "issues": issues, + "runtime": { + "base_url": runtime.get("base_url") or DEFAULT_BASE_URL, + "project": runtime.get("project") or DEFAULT_PROJECT, + "dry_run": bool(runtime.get("dry_run", True)), + "helper_command": str(runtime.get("helper_command") or Path(__file__).resolve()), + }, + } + + +def apply_loop_contact_action( + client: RedmineClient, + profile: dict[str, Any], + contact: dict[str, Any], + *, + runtime: dict[str, Any], + action: str, + reason: str, +) -> dict[str, Any]: + if action == "default-tag": + fields = {"tag_list": merge_tags(extract_tag_list(contact), [str(profile["default_tag"])], [])} + elif action == "remove-tags": + fields = {"tag_list": merge_tags(extract_tag_list(contact), [], list(profile.get("remove_tags") or []))} + else: + raise RedmineError(f"Unsupported loop action: {action}") + + outcome = { + "contact_id": int(contact["id"]), + "action": action, + "reason": reason, + "fields": fields, + "applied": False, + } + if not runtime.get("dry_run", True): + client.update_contact(int(contact["id"]), fields) + outcome["applied"] = True + return outcome + + +def read_cache_payload(cache_path: Path, args: argparse.Namespace) -> dict[str, Any]: with cache_path.open("r", encoding="utf-8") as handle: data = json.load(handle) validate_cache_metadata(data, args, cache_path) + return data + + +def resolve_contact_from_args(args: argparse.Namespace) -> dict[str, Any]: + source = args.source + cache_path = resolve_cache_path(args) + + if args.refresh_cache: + client = client_from_args(args) + contacts = client.fetch_contacts() + write_cache(cache_path, args, contacts) + + if source in {"cache", "auto"}: + if cache_path.exists(): + data = read_cache_payload(cache_path, args) + cached_contact = find_contact_in_list(data.get("contacts") or [], args.contact_id) + if cached_contact is not None: + return cached_contact + if source == "cache": + raise RedmineError(f"Contact {args.contact_id} not found in cache {cache_path}.") + + client = client_from_args(args) + return client.get_contact(args.contact_id) + + +def find_contact_in_list(contacts: list[dict[str, Any]], contact_id: int) -> dict[str, Any] | None: + for contact in contacts: + if int(contact.get("id", -1)) == int(contact_id): + return contact + return None + + +def has_update_changes(args: argparse.Namespace) -> bool: + return any( + ( + bool(getattr(args, "set", []) or []), + bool(getattr(args, "add_tag", []) or []), + bool(getattr(args, "remove_tag", []) or []), + bool(getattr(args, "set_custom_field", []) or []), + ) + ) + + +def load_contacts_for_batch(args: argparse.Namespace) -> list[dict[str, Any]]: + cache_path = resolve_cache_path(args) + + if args.source == "api": + client = client_from_args(args) + contacts = client.fetch_contacts() + if args.refresh_cache: + write_cache(cache_path, args, contacts) + return contacts + + if args.refresh_cache or not cache_path.exists(): + client = client_from_args(args) + contacts = client.fetch_contacts() + write_cache(cache_path, args, contacts) + return contacts + + data = read_cache_payload(cache_path, args) return data["contacts"] +def select_batch_contacts(contacts: list[dict[str, Any]], args: argparse.Namespace) -> list[dict[str, Any]]: + filtered = filter_contacts_from_args(contacts, args) + ordered = order_contacts_for_batch(filtered, getattr(args, "order", "as-is")) + return ordered[: args.max_contacts] + + +def order_contacts_for_batch(contacts: list[dict[str, Any]], order: str) -> list[dict[str, Any]]: + if order == "as-is": + return list(contacts) + + if order == "oldest-updated-first": + return sorted(contacts, key=lambda c: sort_key_by_datetime(c, "updated_on", ascending=True)) + if order == "newest-updated-first": + return sorted(contacts, key=lambda c: sort_key_by_datetime(c, "updated_on", ascending=False)) + if order == "oldest-created-first": + return sorted(contacts, key=lambda c: sort_key_by_datetime(c, "created_on", ascending=True)) + if order == "newest-created-first": + return sorted(contacts, key=lambda c: sort_key_by_datetime(c, "created_on", ascending=False)) + if order == "id-asc": + return sorted(contacts, key=sort_key_by_id_asc) + if order == "id-desc": + return sorted(contacts, key=sort_key_by_id_desc) + + raise RedmineError(f"Unsupported batch order: {order!r}") + + +def sort_key_by_datetime(contact: dict[str, Any], field: str, *, ascending: bool) -> tuple[int, float, int]: + value = parse_date_or_datetime(contact.get(field)) + contact_id = contact_numeric_id(contact) + + if value is None: + return (1, 0.0, contact_id) + + timestamp = value.timestamp() + if not ascending: + timestamp = -timestamp + return (0, timestamp, contact_id) + + +def sort_key_by_id_asc(contact: dict[str, Any]) -> int: + return contact_numeric_id(contact) + + +def sort_key_by_id_desc(contact: dict[str, Any]) -> int: + return -contact_numeric_id(contact) + + +def contact_numeric_id(contact: dict[str, Any]) -> int: + raw_id = contact.get("id") + if raw_id is None: + return sys.maxsize + try: + return int(raw_id) + except (TypeError, ValueError): + return sys.maxsize + + +def build_update_fields(args: argparse.Namespace, client: RedmineClient) -> dict[str, Any]: + needs_contact = bool(args.add_tag or args.remove_tag or args.set_custom_field) + if not needs_contact: + return parse_update_fields(args.set) + + contact = resolve_contact_for_update(args, client) + return build_update_fields_for_contact(contact, args) + + +def build_update_fields_for_contact(contact: dict[str, Any], args: argparse.Namespace) -> dict[str, Any]: + fields: dict[str, Any] = parse_update_fields(getattr(args, "set", []) or []) + + add_tags = list(getattr(args, "add_tag", []) or []) + remove_tags = list(getattr(args, "remove_tag", []) or []) + if add_tags or remove_tags: + fields["tag_list"] = merge_tags( + existing=extract_tag_list(contact), + add_tags=add_tags, + remove_tags=remove_tags, + ) + + custom_assignments = list(getattr(args, "set_custom_field", []) or []) + if custom_assignments: + assignments = parse_custom_field_assignments(custom_assignments) + fields["custom_field_values"] = build_custom_field_values_patch( + contact.get("custom_fields") or [], + assignments, + ) + + return fields + + +def resolve_contact_for_update(args: argparse.Namespace, client: RedmineClient) -> dict[str, Any]: + source = args.contact_source + cache_path = resolve_cache_path(args) + + if source in {"cache", "auto"} and cache_path.exists(): + data = read_cache_payload(cache_path, args) + cached_contact = find_contact_in_list(data.get("contacts") or [], args.contact_id) + if cached_contact is not None: + return cached_contact + if source == "cache": + raise RedmineError(f"Contact {args.contact_id} not found in cache {cache_path}.") + + return client.get_contact(args.contact_id) + + +def parse_custom_field_assignments(assignments: list[str]) -> dict[str, str]: + parsed: dict[str, str] = {} + for assignment in assignments: + if "=" not in assignment: + raise RedmineError(f"Invalid --set-custom-field value: {assignment!r}") + key, value = assignment.split("=", 1) + slug = slugify_custom_field_name(key) + if not slug: + raise RedmineError(f"Invalid custom field key {key!r}") + parsed[slug] = value + return parsed + + +def build_custom_field_values_patch( + existing_custom_fields: list[dict[str, Any]], + assignments_by_slug: dict[str, str], +) -> dict[str, str]: + by_slug: dict[str, dict[str, Any]] = {} + for field in existing_custom_fields: + field_name = str(field.get("name") or "") + field_id = field.get("id") + if not field_name or field_id is None: + continue + by_slug[slugify_custom_field_name(field_name)] = field + + patch: dict[str, str] = {} + unknown: list[str] = [] + for slug, value in assignments_by_slug.items(): + field = by_slug.get(slug) + if not field: + unknown.append(slug) + continue + patch[str(field["id"])] = value + + if unknown: + available = ", ".join(sorted(key for key in by_slug if key)) + raise RedmineError( + "Unknown custom field slug(s): " + + ", ".join(sorted(unknown)) + + (f". Available: {available}" if available else "") + ) + + return patch + + +def slugify_custom_field_name(name: str) -> str: + slug = re.sub(r"[^a-z0-9]+", "-", name.strip().lower()) + return slug.strip("-") + + +def merge_tags(existing: list[str], add_tags: list[str], remove_tags: list[str]) -> list[str]: + merged = {tag.strip() for tag in existing if tag and tag.strip()} + for tag in add_tags: + clean = tag.strip() + if clean: + merged.add(clean) + for tag in remove_tags: + clean = tag.strip() + if clean: + merged.discard(clean) + return sorted(merged) + + +def extract_tag_list(contact: dict[str, Any]) -> list[str]: + tags = contact.get("tag_list") + if not tags: + return [] + if isinstance(tags, list): + return [str(tag).strip() for tag in tags if str(tag).strip()] + if isinstance(tags, str): + return [part.strip() for part in tags.split(",") if part.strip()] + return [str(tags).strip()] + + +def filter_contacts( + contacts: list[dict[str, Any]], + *, + created_after: str | None = None, + created_before: str | None = None, + updated_after: str | None = None, + updated_before: str | None = None, + with_tags: list[str] | None = None, + without_tags: list[str] | None = None, + no_tags: bool = False, +) -> list[dict[str, Any]]: + if no_tags and with_tags: + raise RedmineError("--no-tags cannot be combined with --with-tag.") + + created_after_dt = parse_date_or_datetime(created_after) if created_after else None + created_before_dt = parse_date_or_datetime(created_before, end_of_day=True) if created_before else None + updated_after_dt = parse_date_or_datetime(updated_after) if updated_after else None + updated_before_dt = parse_date_or_datetime(updated_before, end_of_day=True) if updated_before else None + + with_tags_set = {tag.strip().lower() for tag in (with_tags or []) if tag.strip()} + without_tags_set = {tag.strip().lower() for tag in (without_tags or []) if tag.strip()} + + filtered: list[dict[str, Any]] = [] + for contact in contacts: + created_on = parse_date_or_datetime(contact.get("created_on")) + updated_on = parse_date_or_datetime(contact.get("updated_on")) + tags = {tag.lower() for tag in extract_tag_list(contact)} + + if created_after_dt and (created_on is None or created_on < created_after_dt): + continue + if created_before_dt and (created_on is None or created_on > created_before_dt): + continue + if updated_after_dt and (updated_on is None or updated_on < updated_after_dt): + continue + if updated_before_dt and (updated_on is None or updated_on > updated_before_dt): + continue + if no_tags and tags: + continue + if with_tags_set and not with_tags_set.issubset(tags): + continue + if without_tags_set and without_tags_set.intersection(tags): + continue + + filtered.append(contact) + + return filtered + + +def parse_date_or_datetime(value: Any, end_of_day: bool = False) -> datetime | None: + if value is None: + return None + raw = str(value).strip() + if not raw: + return None + + if re.fullmatch(r"\d{4}-\d{2}-\d{2}", raw): + dt = datetime.strptime(raw, "%Y-%m-%d") + if end_of_day: + dt = dt.replace(hour=23, minute=59, second=59) + return dt.replace(tzinfo=timezone.utc) + + normalized = raw.replace("Z", "+00:00") + try: + dt = datetime.fromisoformat(normalized) + except ValueError: + raise RedmineError(f"Invalid ISO date/datetime value: {raw!r}") + + if dt.tzinfo is None: + dt = dt.replace(tzinfo=timezone.utc) + return dt.astimezone(timezone.utc) + + def write_cache(cache_path: Path, args: argparse.Namespace, contacts: list[dict[str, Any]]) -> None: cache_path.parent.mkdir(parents=True, exist_ok=True) payload = { @@ -461,5 +1484,16 @@ def print_json(data: dict[str, Any]) -> None: print(json.dumps(data, indent=2, sort_keys=True)) + + +def set_nmi_account_link(redmine_contact_id: int, user_account_id: str | int, client: "RedmineContactsClient | None" = None, apply: bool = False) -> dict: + """Convenience: set custom field 70 (Account Number) on a Redmine contact to the NMI user_account_id.""" + client = client or RedmineContactsClient() + patch = build_custom_field_values_patch(["account-number=" + str(user_account_id)]) + if apply: + client.update_contact(redmine_contact_id, {"custom_field_values": patch.get("custom_field_values", {})}) + return {"contact_id": redmine_contact_id, "set_account_number": str(user_account_id), "applied": apply} + + if __name__ == "__main__": raise SystemExit(main()) diff --git a/redmine_contacts_completion.sh b/redmine_contacts_completion.sh new file mode 100644 index 0000000..9528167 --- /dev/null +++ b/redmine_contacts_completion.sh @@ -0,0 +1,103 @@ +#!/usr/bin/env bash +# Bash completion for redmine_contacts.py +# Usage: +# source /path/to/redmine_contacts_completion.sh +# Or copy into /etc/bash_completion.d/ (or ~/.local/share/bash-completion/completions/) + +_redmine_contacts_completion() { + local cur prev cmd i + COMPREPLY=() + cur="${COMP_WORDS[COMP_CWORD]}" + prev="${COMP_WORDS[COMP_CWORD-1]}" + + local global_opts="--base-url --project --cache -h --help" + local commands="fetch search show update batch-update loop-run helpdesk-ticket issue-full helpdesk-issues helpdesk-messages helpdesk-timeline" + + # Determine active subcommand (first non-option token after argv0) + cmd="" + for ((i=1; i<${#COMP_WORDS[@]}; i++)); do + if [[ "${COMP_WORDS[i]}" != -* ]]; then + cmd="${COMP_WORDS[i]}" + break + fi + done + + # Value completion for option arguments + case "${prev}" in + --cache|--profile) + COMPREPLY=( $(compgen -f -- "${cur}") ) + return 0 + ;; + --cache-mode) + COMPREPLY=( $(compgen -W "full filtered" -- "${cur}") ) + return 0 + ;; + --source) + if [[ "${cmd}" == "show" ]]; then + COMPREPLY=( $(compgen -W "auto cache api" -- "${cur}") ) + else + COMPREPLY=( $(compgen -W "cache api" -- "${cur}") ) + fi + return 0 + ;; + --contact-source) + COMPREPLY=( $(compgen -W "api cache auto" -- "${cur}") ) + return 0 + ;; + --order) + COMPREPLY=( $(compgen -W "as-is oldest-updated-first newest-updated-first oldest-created-first newest-created-first id-asc id-desc" -- "${cur}") ) + return 0 + ;; + --set) + COMPREPLY=( $(compgen -W "first_name= middle_name= last_name= company= website= skype_name= birthday= job_title= background= phone= email= tag_list=" -- "${cur}") ) + return 0 + ;; + --set-custom-field) + COMPREPLY=( $(compgen -W "account-number= instagram= facebook= youtube= linkedin= whatsapp= x-twitter=" -- "${cur}") ) + return 0 + ;; + esac + + # If no command selected yet, complete command names + global options + if [[ -z "${cmd}" ]]; then + COMPREPLY=( $(compgen -W "${commands} ${global_opts}" -- "${cur}") ) + return 0 + fi + + local common_filter_opts="--created-after --created-before --updated-after --updated-before --with-tag --without-tag --no-tags" + + case "${cmd}" in + fetch) + COMPREPLY=( $(compgen -W "${global_opts} --limit --cache-mode ${common_filter_opts}" -- "${cur}") ) + ;; + search) + COMPREPLY=( $(compgen -W "${global_opts} --refresh --offline-cache --limit --min-score ${common_filter_opts}" -- "${cur}") ) + ;; + show) + COMPREPLY=( $(compgen -W "${global_opts} --source --refresh-cache" -- "${cur}") ) + ;; + update) + COMPREPLY=( $(compgen -W "${global_opts} --set --add-tag --remove-tag --set-custom-field --contact-source --apply" -- "${cur}") ) + ;; + batch-update) + COMPREPLY=( $(compgen -W "${global_opts} --max-contacts --order --source --refresh-cache ${common_filter_opts} --set --add-tag --remove-tag --set-custom-field --apply" -- "${cur}") ) + ;; + loop-run) + COMPREPLY=( $(compgen -W "${global_opts} --profile --max-contacts --order --source --refresh-cache ${common_filter_opts} --apply" -- "${cur}") ) + ;; + helpdesk-ticket) + COMPREPLY=( $(compgen -W "${global_opts}" -- "${cur}") ) + ;; + issue-full) + COMPREPLY=( $(compgen -W "${global_opts} --message-limit" -- "${cur}") ) + ;; + helpdesk-issues|helpdesk-messages|helpdesk-timeline) + COMPREPLY=( $(compgen -W "${global_opts} --limit" -- "${cur}") ) + ;; + esac + + return 0 +} + +complete -F _redmine_contacts_completion redmine_contacts.py +complete -F _redmine_contacts_completion redmine_contacts diff --git a/redmine_crm/__init__.py b/redmine_crm/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/redmine_crm/crm_client.py b/redmine_crm/crm_client.py new file mode 100644 index 0000000..6a44833 --- /dev/null +++ b/redmine_crm/crm_client.py @@ -0,0 +1,230 @@ +"""CRM upsert client supporting idempotency by Redmine ID and email fallback.""" + +from __future__ import annotations + +import logging +from dataclasses import dataclass +from typing import Any, Callable, Dict, List, Optional + +logger = logging.getLogger(__name__) + + +class CrmError(Exception): + """Base exception for CRM errors.""" + + def __init__( + self, + message: str, + *, + redmine_id: Optional[int] = None, + crm_id: Optional[str] = None, + ) -> None: + super().__init__(message) + self.redmine_id = redmine_id + self.crm_id = crm_id + + +@dataclass(frozen=True, slots=True) +class UpsertResult: + crm_id: str + created: bool + updated: bool + skipped_unchanged: bool + redmine_contact_id: int + + +class CrmClient: + """Generic CRM upsert wrapper. + + Provides primitives for local testing; real CRM adapters should subclass + and implement _find_by_redmine_id, _find_by_email, _create, and _update. + """ + + # Shared simulated CRM store for the in-memory adapter. + _store: Dict[str, Dict[str, Any]] = {} + + def __init__( + self, + *, + api_key: Optional[str] = None, + base_url: Optional[str] = None, + find_by_redmine_id: Optional[Callable[[int], Optional[Dict[str, Any]]]] = None, + find_by_email: Optional[Callable[[str], Optional[Dict[str, Any]]]] = None, + create: Optional[Callable[[Dict[str, Any]], str]] = None, + update: Optional[Callable[[str, Dict[str, Any]], None]] = None, + ) -> None: + self.api_key = api_key + self.base_url = base_url + self._find_by_redmine_id_fn = find_by_redmine_id + self._find_by_email_fn = find_by_email + self._create_fn = create + self._update_fn = update + + # --- Extensibility hooks (override in subclasses) ----------------------- + + def _find_by_redmine_id(self, redmine_id: int) -> Optional[Dict[str, Any]]: + if self._find_by_redmine_id_fn: + return self._find_by_redmine_id_fn(redmine_id) + for record in self._store.values(): + if record.get("redmine_contact_id") == redmine_id: + return record + return None + + def _find_by_email(self, email: str) -> Optional[Dict[str, Any]]: + if self._find_by_email_fn: + return self._find_by_email_fn(email) + normalized = email.lower().strip() + for record in self._store.values(): + for entry in record.get("emails", []): + if entry.get("value", "").lower().strip() == normalized: + return record + return None + + def _create(self, record: Dict[str, Any]) -> str: + if self._create_fn: + return self._create_fn(record) + crm_id = f"crm::{record['redmine_contact_id']}" + self._store[crm_id] = record.copy() + return crm_id + + def _update(self, crm_id: str, record: Dict[str, Any]) -> None: + if self._update_fn: + self._update_fn(crm_id, record) + return + self._store[crm_id] = record.copy() + + # --- Public API --------------------------------------------------------- + + def _normalize_email(self, email: Optional[str]) -> Optional[str]: + if not email: + return None + return email.lower().strip() + + def _has_changed( + self, + existing: Dict[str, Any], + candidate: Dict[str, Any], + ) -> bool: + """Return True if meaningful Redmine data changed.""" + if not existing: + return True + candidate_ts = candidate.get("last_redmine_update") + existing_ts = existing.get("last_redmine_update") + if candidate_ts and existing_ts: + return candidate_ts != existing_ts + # Fallback to structural comparison for fields we care about. + keys = { + "full_name", + "company", + "title", + "emails", + "phones", + "urls", + "address", + "notes", + "tags", + "birthday", + } + for key in keys: + if existing.get(key) != candidate.get(key): + return True + return False + + def upsert(self, record: Dict[str, Any]) -> UpsertResult: + """Idempotently upsert a mapped Redmine contact into the CRM.""" + redmine_id = record.get("redmine_contact_id") + if redmine_id is None: + raise CrmError("Record missing redmine_contact_id") + + existing = self._find_by_redmine_id(redmine_id) + fallback_email: Optional[str] = None + if not existing and record.get("emails"): + fallback_email = self._normalize_email(record["emails"][0]["value"]) + if fallback_email: + existing = self._find_by_email(fallback_email) + + if existing: + crm_id = existing["crm_id"] + if not crm_id: + raise CrmError( + "Existing CRM record missing crm_id", + redmine_id=redmine_id, + ) + if not self._has_changed(existing, record): + logger.info( + "CRM record %s unchanged for Redmine contact %s; skipping", + crm_id, + redmine_id, + ) + return UpsertResult( + crm_id=crm_id, + created=False, + updated=False, + skipped_unchanged=True, + redmine_contact_id=redmine_id, + ) + + record["crm_id"] = crm_id + if fallback_email and not existing.get("redmine_contact_id"): + # If matched by email, attach Redmine ID to close the loop. + logger.info( + "Matched Redmine contact %s by email %s; updating CRM record %s", + redmine_id, + fallback_email, + crm_id, + ) + self._update(crm_id, record) + return UpsertResult( + crm_id=crm_id, + created=False, + updated=True, + skipped_unchanged=False, + redmine_contact_id=redmine_id, + ) + + crm_id = self._create(record) + record["crm_id"] = crm_id + self._store[crm_id] = record.copy() + return UpsertResult( + crm_id=crm_id, + created=True, + updated=False, + skipped_unchanged=False, + redmine_contact_id=redmine_id, + ) + + def sync_batch( + self, + records: List[Dict[str, Any]], + ) -> Dict[str, Any]: + """Sync a batch and return aggregated stats.""" + stats = { + "processed": 0, + "created": 0, + "updated": 0, + "skipped": 0, + "errors": 0, + "error_records": [], + } + for record in records: + stats["processed"] += 1 + try: + result = self.upsert(record) + if result.created: + stats["created"] += 1 + elif result.updated: + stats["updated"] += 1 + else: + stats["skipped"] += 1 + except CrmError as exc: + stats["errors"] += 1 + stats["error_records"].append( + { + "redmine_contact_id": exc.redmine_id, + "crm_id": exc.crm_id, + "reason": str(exc), + } + ) + logger.warning("CRM upsert failed: %s", exc) + logger.info("CRM sync batch stats: %s", stats) + return stats diff --git a/redmine_crm/mapper.py b/redmine_crm/mapper.py new file mode 100644 index 0000000..c276162 --- /dev/null +++ b/redmine_crm/mapper.py @@ -0,0 +1,310 @@ +"""Map Redmine contact payloads to the target CRM schema and enrich missing data.""" + +from __future__ import annotations + +import logging +import re +from datetime import datetime, timezone +from typing import Any, Dict, List, Optional, Tuple + +logger = logging.getLogger(__name__) + +_EMAIL_RE = re.compile(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}") +_PHONE_RE = re.compile(r"[\d+\-().\s]{7,}") + + +def _coalesce(*values: Any) -> Any: + """Return the first non-empty value.""" + for value in values: + if value: + return value + return None + + +def _to_iso(ts: Optional[str]) -> Optional[str]: + """Normalize Redmine timestamps to ISO-8601 in UTC.""" + if not ts: + return None + try: + # Redmine uses ISO-ish strings; this accepts several common formats. + parsed = datetime.fromisoformat(ts.replace("Z", "+00:00")) + parsed = parsed.astimezone(timezone.utc) + return parsed.isoformat() + except ValueError as exc: + logger.debug("Could not parse timestamp %r: %s", ts, exc) + return ts + + +def _normalize_name( + first: Optional[str], + last: Optional[str], + middle: Optional[str], +) -> Tuple[Optional[str], Optional[str], Optional[str]]: + first = (first or "").strip() or None + middle = (middle or "").strip() or None + last = (last or "").strip() or None + return first, middle, last + + +def _build_full_name( + first: Optional[str], + last: Optional[str], + middle: Optional[str], + company: Optional[str], + is_company: bool, +) -> str: + if is_company and company: + return company + + parts = [p for p in (first, middle, last) if p] + if parts: + return " ".join(parts) + if company: + return company + return "Unknown" + + +def _classify_phone(raw: str) -> str: + raw = re.sub(r"[^\d+]", "", raw) + # Heuristic: mobile numbers are often longer and start with a country code or specific prefixes. + if raw.startswith("+") and len(raw) >= 11: + return "mobile" + # North American mobile indicators are unreliable; default to work. + return "work" + + +def _normalize_phone(raw: Optional[str]) -> Optional[str]: + if not raw: + return None + digits = re.sub(r"[^\d+]", "", raw) + return digits if digits else None + + +def _extract_emails_from_text(text: Optional[str]) -> List[str]: + if not text: + return [] + return list(dict.fromkeys(_EMAIL_RE.findall(text))) + + +def _company_from_email(email: Optional[str]) -> Optional[str]: + """Heuristic: derive a company display name from an email domain.""" + if not email or "@" not in email: + return None + domain = email.split("@")[1].strip().lower() + if not domain: + return None + # Strip common TLDs and subdomain noise. + name = domain.split(":")[0] + tokens = name.split(".") + if tokens[0] in ("www", "mail", "smtp", "contact", "info"): + tokens = tokens[1:] + if len(tokens) >= 2: + tokens = tokens[:-1] + core = tokens[0] if tokens else domain.split(".")[0] + title = core.replace("-", " ").title() + if title.lower() in ("gmail", "yahoo", "hotmail", "outlook", "aol"): + return None + return f"{title} Inc" + + +def _parse_address(addr: Optional[Dict[str, Any]]) -> Dict[str, Optional[str]]: + if not addr: + return { + "street": None, + "city": None, + "region": None, + "country": None, + "postal_code": None, + } + return { + "street": _coalesce(addr.get("street")), + "city": _coalesce(addr.get("city")), + "region": _coalesce(addr.get("state")), + "country": _coalesce(addr.get("country")), + "postal_code": _coalesce(addr.get("zip")), + } + + +def _normalize_tags(tag_list: Any) -> List[str]: + tags: List[str] = [] + if isinstance(tag_list, str): + tag_list = [t.strip() for t in tag_list.split(",")] + if isinstance(tag_list, list): + for tag in tag_list: + if tag: + normalized = str(tag).strip().lower() + if normalized and normalized not in tags: + tags.append(normalized) + # Always mark origin. + if "from-redmine" not in tags: + tags.append("from-redmine") + return sorted(tags) + + +def _extract_custom( + custom_fields: Optional[List[Dict[str, Any]]], + keys: List[str], +) -> Optional[str]: + """Return the value of the first custom field matching one of keys (case-insensitive).""" + if not custom_fields: + return None + key_set = {k.lower() for k in keys} + for field in custom_fields: + name = (field.get("name") or "").strip().lower() + if name in key_set: + value = field.get("value") + if value: + return str(value).strip() + return None + + +def _merge_notes( + background: Optional[str], + custom_fields: Optional[List[Dict[str, Any]]], + contact_url: str, +) -> str: + parts = [] + if background: + parts.append(str(background).strip()) + if custom_fields: + for field in custom_fields: + name = field.get("name") + value = field.get("value") + if name and value: + # Skip fields already promoted to typed CRM fields (e.g. LinkedIn). + if name.lower() in {"linkedin"}: + continue + parts.append(f"{name}: {value}") + parts.append(f"Redmine source: {contact_url}") + return "\n\n".join(parts) + + +class EnrichmentResult: + """Result of enrichment: data plus metadata.""" + + def __init__(self) -> None: + self.company_source: Optional[str] = None + self.email_source: Optional[str] = None + self.confidence_issues: List[str] = [] + + +def enrich( + mapped: Dict[str, Any], + raw: Dict[str, Any], + *, + enable_domain_lookup: bool = True, + confidence_threshold: float = 0.7, +) -> Tuple[Dict[str, Any], EnrichmentResult]: + """Apply enrichment rules to a mapped CRM record.""" + result = EnrichmentResult() + + # Email fallback: scan background and custom fields. + if not mapped.get("emails"): + candidates = _extract_emails_from_text(raw.get("background")) + if candidates: + mapped["emails"] = [{"value": candidates[0], "type": "work"}] + result.email_source = "background" + + if not mapped.get("emails"): + result.confidence_issues.append("No email found; record may not be upsertable") + + # Company enrichment. + if not mapped.get("company") and mapped.get("emails"): + email = mapped["emails"][0]["value"] + if enable_domain_lookup: + company = _company_from_email(email) + if company: + mapped["company"] = company + result.company_source = "domain_heuristic" + if not mapped.get("company"): + result.confidence_issues.append("Company unknown") + + if mapped.get("company") and not mapped.get("emails"): + # Company but no email: confidence low. + result.confidence_issues.append("Company available but no email") + + # Phone normalization. + phones = mapped.get("phones") or [] + normalized_phones: List[Dict[str, str]] = [] + seen: set[str] = set() + for phone in phones: + value = _normalize_phone(phone.get("value")) + if value and value not in seen: + seen.add(value) + normalized_phones.append({ + "value": value, + "type": phone.get("type", "work"), + }) + mapped["phones"] = normalized_phones + + # Confidence gate: not a hard failure, but logged. + if result.confidence_issues: + logger.info( + "Enrichment confidence issues for Redmine contact %s: %s", + mapped.get("redmine_contact_id"), + "; ".join(result.confidence_issues), + ) + + mapped["enriched_at"] = datetime.now(timezone.utc).isoformat() + return mapped, result + + +def map_contact(raw: Dict[str, Any], *, base_url: str) -> Dict[str, Any]: + """Map a Redmine contact payload to the normalized CRM schema.""" + contact_id = raw.get("id") + first_name, middle_name, last_name = _normalize_name( + raw.get("first_name"), + raw.get("last_name"), + raw.get("middle_name"), + ) + company = _coalesce(raw.get("company")) + is_company = bool(raw.get("is_company")) + full_name = _build_full_name( + first_name, last_name, middle_name, company, is_company + ) + + emails: List[Dict[str, str]] = [] + primary_email = _coalesce(raw.get("email")) + if primary_email: + emails.append({"value": primary_email, "type": "work"}) + + phones: List[Dict[str, str]] = [] + raw_phone = _coalesce(raw.get("phone")) + if raw_phone: + phones.append({"value": raw_phone, "type": _classify_phone(raw_phone)}) + + urls: List[Dict[str, str]] = [] + website = _coalesce(raw.get("website")) + if website: + urls.append({"value": website, "type": "work"}) + linkedin = _extract_custom(raw.get("custom_fields"), ["linkedin", "linked_in"]) + if linkedin: + urls.append({"value": linkedin, "type": "linkedin"}) + + contact_url = f"{base_url.rstrip('/')}/contacts/{contact_id}" + + return { + "crm_id": None, + "redmine_contact_id": contact_id, + "full_name": full_name, + "first_name": first_name, + "middle_name": middle_name, + "last_name": last_name, + "company": company, + "title": _coalesce(raw.get("job_title")), + "emails": emails, + "phones": phones, + "urls": urls, + "website": website, + "birthday": _to_iso(raw.get("birthday")), + "address": _parse_address(raw.get("address_attributes")), + "notes": _merge_notes(raw.get("background"), raw.get("custom_fields"), contact_url), + "tags": _normalize_tags(raw.get("tag_list")), + "source": "redmine", + "synced_at": datetime.now(timezone.utc).isoformat(), + "enriched_at": None, + "last_redmine_update": _to_iso(raw.get("updated_at")), + "is_company": is_company, + "project_id": raw.get("project_id"), + "assigned_to_id": raw.get("assigned_to_id"), + } diff --git a/redmine_crm/redmine_client.py b/redmine_crm/redmine_client.py new file mode 100644 index 0000000..72fd552 --- /dev/null +++ b/redmine_crm/redmine_client.py @@ -0,0 +1,225 @@ +"""Redmine Contacts REST API client with retry, auth, and logging.""" + +from __future__ import annotations + +import logging +import time +from dataclasses import dataclass +from http import HTTPStatus +from typing import Any, Dict, List, Optional + +import requests + +logger = logging.getLogger(__name__) + + +class RedmineError(Exception): + """Base exception for Redmine client errors.""" + + def __init__(self, message: str, *, redmine_id: Optional[int] = None) -> None: + super().__init__(message) + self.redmine_id = redmine_id + + +class RedmineAuthError(RedmineError): + """Permanent auth failure (401, 403).""" + + +class RedmineTransientError(RedmineError): + """Retryable error (network, 5xx, rate limit).""" + + +class RedmineNotFoundError(RedmineError): + """Resource not found (404).""" + + +@dataclass(frozen=True, slots=True) +class RetryConfig: + max_attempts: int = 5 + base_delay_seconds: float = 1.0 + max_delay_seconds: float = 30.0 + + +class RedmineClient: + """Thin, retry-aware wrapper around Redmine's REST API.""" + + def __init__( + self, + base_url: str, + api_key: str, + *, + project_id: Optional[int] = None, + timeout: float = 30.0, + retry: Optional[RetryConfig] = None, + ) -> None: + self.base_url = base_url.rstrip("/") + self.api_key = api_key + self.project_id = project_id + self.timeout = timeout + self.retry = retry or RetryConfig() + self.session = requests.Session() + self.session.headers.update({ + "X-Redmine-API-Key": self.api_key, + "Content-Type": "application/json", + "Accept": "application/json", + }) + + def _request( + self, + method: str, + path: str, + *, + params: Optional[Dict[str, Any]] = None, + json_body: Optional[Dict[str, Any]] = None, + redmine_id: Optional[int] = None, + ) -> Dict[str, Any]: + url = f"{self.base_url}{path}" + last_error: Optional[Exception] = None + + for attempt in range(1, self.retry.max_attempts + 1): + transient_error: Optional[RedmineTransientError] = None + + try: + response = self.session.request( + method, + url, + params=params, + json=json_body, + timeout=self.timeout, + ) + except requests.RequestException as exc: + last_error = exc + logger.warning( + "Redmine network error on attempt %d/%d: %s", + attempt, + self.retry.max_attempts, + exc, + ) + if attempt == self.retry.max_attempts: + raise RedmineTransientError( + f"Network error after {self.retry.max_attempts} attempts: {exc}", + redmine_id=redmine_id, + ) from exc + else: + logger.debug( + "Redmine request %s %s attempt %d/%d status=%d", + method, + url, + attempt, + self.retry.max_attempts, + response.status_code, + ) + + if response.status_code == HTTPStatus.OK.value: + if not response.content: + return {} + return response.json() + + if response.status_code in ( + HTTPStatus.UNAUTHORIZED.value, + HTTPStatus.FORBIDDEN.value, + ): + raise RedmineAuthError( + f"Authentication failed ({response.status_code})", + redmine_id=redmine_id, + ) + + if response.status_code == HTTPStatus.NOT_FOUND.value: + raise RedmineNotFoundError( + f"Not found ({response.status_code})", + redmine_id=redmine_id, + ) + + if response.status_code == HTTPStatus.TOO_MANY_REQUESTS.value: + transient_error = RedmineTransientError( + "Rate limited (429)", redmine_id=redmine_id + ) + elif response.status_code >= HTTPStatus.INTERNAL_SERVER_ERROR.value: + transient_error = RedmineTransientError( + f"Server error ({response.status_code})", + redmine_id=redmine_id, + ) + else: + # 4xx other than auth/404 are permanent. + raise RedmineError( + f"Client error ({response.status_code}): {response.text}", + redmine_id=redmine_id, + ) + + if transient_error is not None: + if attempt == self.retry.max_attempts: + raise transient_error + logger.warning( + "Redmine transient error on attempt %d/%d: %s", + attempt, + self.retry.max_attempts, + transient_error, + ) + + if attempt < self.retry.max_attempts: + delay = min( + self.retry.base_delay_seconds * (2 ** (attempt - 1)), + self.retry.max_delay_seconds, + ) + logger.info("Retrying Redmine request in %.1f seconds", delay) + time.sleep(delay) + + # Should never be reached because the loop raises. + raise RedmineTransientError("Unexpected loop exit", redmine_id=redmine_id) from last_error + + def fetch_contacts( + self, + *, + updated_since: Optional[str] = None, + limit: int = 100, + offset: int = 0, + status_filter: Optional[str] = "active", + ) -> List[Dict[str, Any]]: + params: Dict[str, Any] = {"limit": limit, "offset": offset} + if self.project_id is not None: + params["project_id"] = self.project_id + if updated_since: + params["updated_on"] = f">={updated_since}" + if status_filter: + params["status"] = status_filter + + data = self._request("GET", "/contacts.json", params=params) + contacts = data.get("contacts", []) + logger.info( + "Fetched %d contacts from Redmine (limit=%d, offset=%d)", + len(contacts), + limit, + offset, + ) + return contacts + + def fetch_contact(self, contact_id: int) -> Dict[str, Any]: + data = self._request( + "GET", + f"/contacts/{contact_id}.json", + redmine_id=contact_id, + ) + return data.get("contact", data) + + def iter_all_contacts( + self, + *, + updated_since: Optional[str] = None, + batch_size: int = 100, + ) -> List[Dict[str, Any]]: + """Iterate through paginated contacts and collect all results.""" + all_contacts: List[Dict[str, Any]] = [] + offset = 0 + while True: + batch = self.fetch_contacts( + updated_since=updated_since, + limit=batch_size, + offset=offset, + ) + if not batch: + break + all_contacts.extend(batch) + if len(batch) < batch_size: + break + offset += batch_size + return all_contacts diff --git a/redmine_crm/sync.py b/redmine_crm/sync.py new file mode 100644 index 0000000..a744feb --- /dev/null +++ b/redmine_crm/sync.py @@ -0,0 +1,65 @@ +"""High-level Redmine -> CRM sync orchestrator.""" + +from __future__ import annotations + +import logging +from typing import Any, Dict, List, Optional + +from redmine_crm.crm_client import CrmClient +from redmine_crm.mapper import enrich, map_contact +from redmine_crm.redmine_client import RedmineClient + +logger = logging.getLogger(__name__) + + +class SyncState: + """Minimal sync state tracker.""" + + def __init__(self, storage: Optional[Dict[str, Any]] = None) -> None: + self._data = storage if storage is not None else {} + + def get_last_sync(self) -> Optional[str]: + return self._data.get("last_sync_timestamp") + + def update(self, timestamp: str, stats: Dict[str, Any]) -> None: + self._data["last_sync_timestamp"] = timestamp + self._data["last_run_stats"] = stats + + +def run_sync( + redmine: RedmineClient, + crm: CrmClient, + sync_state: SyncState, + *, + base_url: str, + batch_size: int = 100, +) -> Dict[str, Any]: + """Run the full ETL pipeline: Extract, Transform, Load.""" + last_sync = sync_state.get_last_sync() + logger.info("Starting Redmine -> CRM sync; last_sync=%s", last_sync) + + # Extract + contacts = redmine.iter_all_contacts( + updated_since=last_sync, + batch_size=batch_size, + ) + logger.info("Extracted %d Redmine contacts", len(contacts)) + + # Transform + records: List[Dict[str, Any]] = [] + for contact in contacts: + try: + mapped = map_contact(contact, base_url=base_url) + enriched, _enrich_meta = enrich(mapped, contact) + records.append(enriched) + except Exception as exc: + logger.exception("Failed to map Redmine contact %s: %s", contact.get("id"), exc) + + # Load + stats = crm.sync_batch(records) + logger.info("CRM upsert stats: %s", stats) + + import datetime + + sync_state.update(datetime.datetime.now(datetime.timezone.utc).isoformat(), stats) + return stats diff --git a/redmine_ticket_attachments.py b/redmine_ticket_attachments.py new file mode 100644 index 0000000..9908bf1 --- /dev/null +++ b/redmine_ticket_attachments.py @@ -0,0 +1,182 @@ +#!/usr/bin/env python3 +"""Attachment helpers for the power-user Redmine ticket client. + +List (cache-first), download to local dir, upload via Redmine REST: + POST /uploads.json?filename=… (raw body) + PUT /issues/:id.json with issue.uploads token +""" + +from __future__ import annotations + +import json +import mimetypes +import re +from pathlib import Path +from typing import Any +from urllib import error as urllib_error +from urllib import parse as urllib_parse +from urllib import request as urllib_request + + +def safe_filename(name: str) -> str: + name = (name or "attachment.bin").strip().replace("\\", "_").replace("/", "_") + name = re.sub(r"[^\w.\- ()\[\]+@]+", "_", name) + return name or "attachment.bin" + + +def attachments_from_issue(issue: dict[str, Any]) -> list[dict[str, Any]]: + atts = issue.get("attachments") + if isinstance(atts, list): + return [a for a in atts if isinstance(a, dict)] + return [] + + +def attachment_dir(cache_dir: Path, project: str, issue_id: int) -> Path: + path = Path(cache_dir) / project / "attachments" / str(int(issue_id)) + path.mkdir(parents=True, exist_ok=True) + return path + + +def pick_attachment(attachments: list[dict[str, Any]], attachment_id: int | None) -> dict[str, Any]: + if not attachments: + raise LookupError("Issue has no attachments.") + if attachment_id is None: + if len(attachments) == 1: + return attachments[0] + raise LookupError( + "Multiple attachments; pass --attachment-id. ids=" + + ",".join(str(a.get("id")) for a in attachments) + ) + for att in attachments: + if int(att.get("id") or 0) == int(attachment_id): + return att + raise LookupError(f"Attachment id {attachment_id} not found on issue.") + + +def download_url_for_attachment(att: dict[str, Any], base_url: str) -> str: + urls = download_url_candidates(att, base_url) + return urls[0] + + +def download_url_candidates(att: dict[str, Any], base_url: str) -> list[str]: + """Ordered download URL candidates (host rewrite + path variants).""" + base = base_url.rstrip("/") + aid = att.get("id") + filename = str(att.get("filename") or "").strip() + candidates: list[str] = [] + + def add(url: str) -> None: + if url and url not in candidates: + candidates.append(url) + + for key in ("content_url", "contentUrl"): + raw = att.get(key) + if not raw: + continue + raw = str(raw) + add(raw) + # rewrite host to active base_url (prod content_url on LAN client, etc.) + try: + from urllib.parse import urlparse, urlunparse + + parsed = urlparse(raw) + base_p = urlparse(base if "://" in base else f"http://{base}") + rewritten = urlunparse( + ( + base_p.scheme or parsed.scheme, + base_p.netloc or parsed.netloc, + parsed.path, + "", + parsed.query, + "", + ) + ) + add(rewritten) + except Exception: + pass + + if aid is not None: + add(f"{base}/attachments/download/{int(aid)}") + if filename: + from urllib.parse import quote + + add(f"{base}/attachments/download/{int(aid)}/{quote(filename)}") + if not candidates: + raise LookupError("Attachment missing id and content_url.") + return candidates + + +def download_attachment_bytes( + *, + url: str, + api_key: str, + impersonate_user: str | None = None, + timeout: int = 120, + fallback_urls: list[str] | None = None, +) -> bytes: + headers = { + "X-Redmine-API-Key": api_key, + "Accept": "*/*", + } + if impersonate_user: + headers["X-Redmine-Switch-User"] = impersonate_user + urls = [url] + [u for u in (fallback_urls or []) if u != url] + last_err: Exception | None = None + for u in urls: + req = urllib_request.Request(u, headers=headers, method="GET") + try: + with urllib_request.urlopen(req, timeout=timeout) as resp: + return resp.read() + except urllib_error.HTTPError as exc: + detail = exc.read().decode("utf-8", errors="replace")[:200] + last_err = RuntimeError(f"HTTP {exc.code} downloading attachment ({u}): {detail}") + continue + except urllib_error.URLError as exc: + last_err = RuntimeError(f"Could not download attachment ({u}): {exc.reason}") + continue + raise last_err or RuntimeError("Attachment download failed") + + +def upload_file_token( + *, + base_url: str, + api_key: str, + path: Path, + filename: str | None = None, + impersonate_user: str | None = None, + timeout: int = 120, +) -> dict[str, Any]: + """POST raw file to /uploads.json?filename=… Returns upload object with token.""" + path = Path(path) + if not path.is_file(): + raise FileNotFoundError(f"File not found: {path}") + name = filename or path.name + data = path.read_bytes() + q = urllib_parse.urlencode({"filename": name}) + url = f"{base_url.rstrip('/')}/uploads.json?{q}" + headers = { + "X-Redmine-API-Key": api_key, + "Content-Type": "application/octet-stream", + "Accept": "application/json", + } + if impersonate_user: + headers["X-Redmine-Switch-User"] = impersonate_user + req = urllib_request.Request(url, data=data, headers=headers, method="POST") + try: + with urllib_request.urlopen(req, timeout=timeout) as resp: + raw = resp.read() + except urllib_error.HTTPError as exc: + detail = exc.read().decode("utf-8", errors="replace")[:500] + raise RuntimeError(f"HTTP {exc.code} uploading file: {detail}") from exc + except urllib_error.URLError as exc: + raise RuntimeError(f"Could not upload file: {exc.reason}") from exc + if not raw: + raise RuntimeError("Empty response from /uploads.json") + payload = json.loads(raw.decode("utf-8")) + upload = payload.get("upload") if isinstance(payload, dict) else None + if not isinstance(upload, dict) or not upload.get("token"): + raise RuntimeError(f"Unexpected upload response: {payload!r}") + content_type = mimetypes.guess_type(name)[0] or "application/octet-stream" + upload.setdefault("filename", name) + upload.setdefault("content_type", content_type) + return upload diff --git a/redmine_ticket_browse.py b/redmine_ticket_browse.py new file mode 100644 index 0000000..9b547fd --- /dev/null +++ b/redmine_ticket_browse.py @@ -0,0 +1,312 @@ +#!/usr/bin/env python3 +"""Interactive browse loop: fzf → pager → action menu → back to fzf. + +Flow: + 1. Present filtered tickets in fzf (or numbered list fallback) + 2. On select: open ticket Markdown in glow/bat/less + 3. After pager exits: action menu for that ticket + 4. Return to list until user quits (Esc/q) + +Actions queue local drafts by default (same safety model as CLI); optional +apply is left to `draft apply` so browse never silently emails/writes prod. +""" + +from __future__ import annotations + +import json +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path +from typing import Any, Callable + + +def nested_name(issue: dict[str, Any], key: str) -> str: + obj = issue.get(key) + if isinstance(obj, dict): + return str(obj.get("name") or "") + return "" + + +def issue_line(issue: dict[str, Any]) -> str: + iid = issue.get("id") + status = nested_name(issue, "status") or "" + assignee = nested_name(issue, "assigned_to") or "" + subj = str(issue.get("subject") or "").replace("\n", " ").replace("\t", " ")[:90] + tags = issue.get("_control_tags") + tag_s = ",".join(tags) if isinstance(tags, list) and tags else "" + return f"{iid}\t{status}\t{assignee}\t{tag_s}\t{subj}" + + +def build_lines(issues: list[dict[str, Any]]) -> list[str]: + return [issue_line(i) for i in issues if i.get("id") is not None] + + +def parse_id_from_line(line: str) -> int | None: + line = (line or "").strip() + if not line: + return None + head = line.split("\t", 1)[0].strip() + try: + return int(head) + except ValueError: + return None + + +# preview helper written at runtime under /tmp by pick_with_fzf + + +def pick_with_fzf( + lines: list[str], + *, + query: str | None = None, + cache_dir: Path | None = None, + project: str | None = None, + preview: bool = True, +) -> int | None: + fzf = shutil.which("fzf") + if not fzf: + return None + header = ( + "Enter=open · Esc=quit · after open: n/p note · f fields · t tag · " + "c contact · e entities · m NMI brief · i id · l list · q quit" + ) + cmd = [ + fzf, + "--height=90%", + "--reverse", + "--prompt=ticket> ", + f"--header={header}", + "--info=inline", + "--bind=ctrl-c:abort", + ] + preview_script: Path | None = None + if preview and cache_dir is not None and project: + # Write a tiny preview helper so quoting stays sane + preview_script = Path(tempfile.gettempdir()) / "rtl_browse_preview.py" + preview_script.write_text( + "#!/usr/bin/env python3\n" + "import json, pathlib, sys\n" + f"cache = pathlib.Path({str(Path(cache_dir))!r})\n" + f"proj = {project!r}\n" + "line = sys.argv[1] if len(sys.argv) > 1 else ''\n" + "iid = line.split('\\t')[0].strip()\n" + "md = cache / proj / 'markdown' / f'{iid}.md'\n" + "jp = cache / proj / f'{iid}.json'\n" + "print(f'# ticket {iid}')\n" + "print()\n" + "if md.is_file():\n" + " print(md.read_text(encoding='utf-8')[:3500])\n" + "elif jp.is_file():\n" + " d = json.loads(jp.read_text(encoding='utf-8'))\n" + " print(d.get('subject') or '')\n" + " print()\n" + " print(str(d.get('description') or '')[:2500])\n" + "else:\n" + " print('(no preview)')\n", + encoding="utf-8", + ) + cmd.extend( + [ + "--preview", + f"python3 {preview_script} {{}}", + "--preview-window=right,50%,wrap", + ] + ) + if query: + cmd.extend(["--query", query]) + try: + proc = subprocess.run( + cmd, + input="\n".join(lines) + "\n", + text=True, + capture_output=True, + ) + finally: + # leave preview script; cheap and reusable + pass + if proc.returncode != 0 or not proc.stdout.strip(): + return None + return parse_id_from_line(proc.stdout.strip().splitlines()[-1]) + + +def pick_numbered(lines: list[str]) -> int | None: + for i, line in enumerate(lines, 1): + print(f"{i:3d}. {line}") + if not sys.stdin.isatty(): + return None + raw = input("Pick # (or q to quit list): ").strip() + if raw.lower() in {"q", "quit", ""}: + return None + try: + idx = int(raw) + except ValueError: + print(f"Invalid selection: {raw!r}", file=sys.stderr) + return None + if idx < 1 or idx > len(lines): + print("Out of range.", file=sys.stderr) + return None + return parse_id_from_line(lines[idx - 1]) + + +def page_ticket_markdown( + issue: dict[str, Any], + *, + cache_dir: Path, + project: str, + pager: str = "auto", +) -> None: + from redmine_ticket_view import display_markdown_file, render_ticket_markdown, write_ticket_markdown + + try: + path = write_ticket_markdown(issue, Path(cache_dir), project) + display_markdown_file(path, pager=pager) + except Exception: + md = render_ticket_markdown(issue) + with tempfile.NamedTemporaryFile("w", suffix=".md", delete=False, encoding="utf-8") as fh: + fh.write(md) + tmp = Path(fh.name) + try: + display_markdown_file(tmp, pager=pager) + finally: + try: + tmp.unlink(missing_ok=True) + except OSError: + pass + + +def prompt_multiline(label: str) -> str | None: + print(f"{label} (end with a single '.' line, or empty first line to cancel):") + lines: list[str] = [] + while True: + try: + line = input() + except EOFError: + break + if line == ".": + break + if not lines and line.strip() == "": + return None + lines.append(line) + text = "\n".join(lines).strip() + return text or None + + +def announce_draft(path: Path) -> None: + """Louder draft feedback for browse UX.""" + draft_id = path.stem + print() + print("=" * 48) + print(f" DRAFT QUEUED draft_id={draft_id}") + print(f" path={path}") + print(" apply later: draft apply " + draft_id) + print(" (browse never auto-applies / never emails customers)") + print("=" * 48) + print() + + +def action_menu( + issue_id: int, + issue: dict[str, Any], + *, + handlers: dict[str, Callable[[], None]], +) -> str: + """Show action menu. Returns 'list' | 'quit' | 'view' | 'menu'.""" + subj = str(issue.get("subject") or "")[:70] + print() + print(f"── ticket #{issue_id} ── {subj}") + print(" [Enter/v] view again") + print(" [n] queue internal note (no email)") + print(" [p] queue private internal note") + print(" [f] queue field update (status/assignee name)") + print(" [t] local control-tag add") + print(" [c] show ticket-contact handoff") + print(" [e] extract entities (patterns)") + print(" [m] NMI brief → queue as PRIVATE note draft") + print(" [i] print issue id only") + print(" [l] back to list") + print(" [q] quit browse") + raw = input("action> ").strip().lower() + if raw in {"", "v", "view"}: + return "view" + if raw in {"l", "list", "b", "back"}: + return "list" + if raw in {"q", "quit"}: + return "quit" + if raw in handlers: + try: + handlers[raw]() + except Exception as exc: # noqa: BLE001 — surface in TUI + print(f"action failed: {exc}", file=sys.stderr) + input("press Enter to continue…") + return "menu" + print(f"Unknown action: {raw!r}", file=sys.stderr) + return "menu" + + +def run_browse_loop( + issues: list[dict[str, Any]], + *, + cache_dir: Path, + project: str, + pager: str = "auto", + once: bool = False, + preview: bool = True, + action_factory: Callable[[int, dict[str, Any]], dict[str, Callable[[], None]]] | None = None, +) -> int: + """Main browse loop. Returns process exit code.""" + if not issues: + print("No matching tickets.", file=sys.stderr) + return 0 + + lines = build_lines(issues) + by_id = {int(i["id"]): i for i in issues if i.get("id") is not None} + use_fzf = bool(shutil.which("fzf") and sys.stdin.isatty()) + + if not sys.stdin.isatty(): + for line in lines: + print(line) + return 0 + + while True: + if use_fzf: + selected = pick_with_fzf( + lines, + cache_dir=Path(cache_dir), + project=project, + preview=preview, + ) + else: + selected = pick_numbered(lines) + if selected is None: + print("browse done", file=sys.stderr) + return 0 + + issue = by_id.get(selected) + if issue is None: + print(f"#{selected} not in current result set", file=sys.stderr) + if once: + return 1 + continue + + page_ticket_markdown(issue, cache_dir=cache_dir, project=project, pager=pager) + + if once: + return 0 + + while True: + handlers: dict[str, Callable[[], None]] = {} + if action_factory: + handlers = action_factory(selected, issue) + + choice = action_menu(selected, issue, handlers=handlers) + if choice == "view": + page_ticket_markdown(issue, cache_dir=cache_dir, project=project, pager=pager) + continue + if choice == "list": + break + if choice == "quit": + print("browse done", file=sys.stderr) + return 0 + continue diff --git a/redmine_ticket_contacts.py b/redmine_ticket_contacts.py new file mode 100644 index 0000000..ff74564 --- /dev/null +++ b/redmine_ticket_contacts.py @@ -0,0 +1,237 @@ +#!/usr/bin/env python3 +"""Durable local contact cache for the power-user Redmine client. + +Reuses RedmineClient from redmine_contacts.py. Contacts live under: + /contacts/.json + /contacts/manifest.json + +This is intentionally thinner than redmine_contacts.py's full CRM CLI — +enough for sync + local search + show next to tickets. +""" + +from __future__ import annotations + +import json +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Iterable + + +def now_iso() -> str: + return datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z") + + +class ContactCache: + def __init__(self, cache_dir: Path): + self.cache_dir = Path(cache_dir) + self.contacts_dir = self.cache_dir / "contacts" + self.contacts_dir.mkdir(parents=True, exist_ok=True) + self.manifest_path = self.contacts_dir / "manifest.json" + + def contact_path(self, contact_id: int | str) -> Path: + return self.contacts_dir / f"{int(contact_id)}.json" + + def load_manifest(self) -> dict[str, Any]: + if not self.manifest_path.is_file(): + return {"version": 1, "contacts": {}, "updated_at": None} + try: + data = json.loads(self.manifest_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return {"version": 1, "contacts": {}, "updated_at": None} + if not isinstance(data, dict): + return {"version": 1, "contacts": {}, "updated_at": None} + data.setdefault("version", 1) + data.setdefault("contacts", {}) + return data + + def save_manifest(self, data: dict[str, Any]) -> None: + data = dict(data) + data["version"] = 1 + data["updated_at"] = now_iso() + tmp = self.manifest_path.with_suffix(".tmp") + tmp.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8") + tmp.replace(self.manifest_path) + + def should_fetch(self, stub: dict[str, Any]) -> bool: + cid = stub.get("id") + if cid is None: + return False + path = self.contact_path(int(cid)) + if not path.is_file(): + return True + manifest = self.load_manifest() + entry = (manifest.get("contacts") or {}).get(str(int(cid))) or {} + return str(entry.get("updated_on") or "") != str(stub.get("updated_on") or "") + + def write_contact(self, contact: dict[str, Any]) -> Path: + cid = contact.get("id") + if cid is None: + raise ValueError("contact missing id") + path = self.contact_path(int(cid)) + path.write_text(json.dumps(contact, indent=2, sort_keys=True) + "\n", encoding="utf-8") + manifest = self.load_manifest() + contacts = manifest.setdefault("contacts", {}) + contacts[str(int(cid))] = { + "id": int(cid), + "updated_on": contact.get("updated_on"), + "last_synced": now_iso(), + "email": primary_email(contact), + "name": contact_display_name(contact), + } + self.save_manifest(manifest) + return path + + def load_contact(self, contact_id: int) -> dict[str, Any]: + path = self.contact_path(contact_id) + if not path.is_file(): + raise FileNotFoundError(f"No cached contact #{contact_id} at {path}") + return json.loads(path.read_text(encoding="utf-8")) + + def iter_contacts(self) -> Iterable[dict[str, Any]]: + for path in sorted(self.contacts_dir.glob("*.json")): + if path.name == "manifest.json": + continue + try: + yield json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + continue + + +def primary_email(contact: dict[str, Any]) -> str: + for key in ("email", "primary_email"): + val = contact.get(key) + if val: + return str(val) + emails = contact.get("emails") or contact.get("address") + if isinstance(emails, list) and emails: + first = emails[0] + if isinstance(first, dict): + return str(first.get("address") or first.get("email") or "") + return str(first) + return "" + + +def contact_display_name(contact: dict[str, Any]) -> str: + if contact.get("name"): + return str(contact["name"]) + first = contact.get("first_name") or contact.get("firstname") or "" + last = contact.get("last_name") or contact.get("lastname") or "" + full = f"{first} {last}".strip() + return full or primary_email(contact) or str(contact.get("id") or "") + + +def contact_id_from_issue(issue: dict[str, Any]) -> int | None: + """Extract helpdesk contact id from a cached/enriched issue payload.""" + helpdesk = issue.get("helpdesk_ticket") + if not isinstance(helpdesk, dict): + return None + if helpdesk.get("contact_id") is not None: + try: + return int(helpdesk["contact_id"]) + except (TypeError, ValueError): + pass + contact = helpdesk.get("contact") + if isinstance(contact, dict) and contact.get("id") is not None: + try: + return int(contact["id"]) + except (TypeError, ValueError): + return None + return None + + +def contact_text_blob(contact: dict[str, Any]) -> str: + parts = [ + contact_display_name(contact), + primary_email(contact), + str(contact.get("company") or ""), + str(contact.get("tag_list") or contact.get("tags") or ""), + ] + return " ".join(parts).lower() + + +def filter_contacts( + contacts: list[dict[str, Any]], + *, + query: str | None = None, + email: str | None = None, + company: str | None = None, + tag: str | None = None, +) -> list[dict[str, Any]]: + out = contacts + if email: + e = email.lower() + out = [c for c in out if e in primary_email(c).lower()] + if company: + co = company.lower() + out = [c for c in out if co in str(c.get("company") or "").lower()] + if tag: + t = tag.lower() + + def has_tag(c: dict[str, Any]) -> bool: + raw = c.get("tag_list") or c.get("tags") or [] + if isinstance(raw, str): + tags = [x.strip().lower() for x in raw.split(",")] + elif isinstance(raw, list): + tags = [] + for item in raw: + if isinstance(item, dict): + tags.append(str(item.get("name") or "").lower()) + else: + tags.append(str(item).lower()) + else: + tags = [] + return t in tags + + out = [c for c in out if has_tag(c)] + if query: + q = query.lower() + out = [c for c in out if q in contact_text_blob(c)] + return out + + +def sync_contacts( + client: Any, + cache: ContactCache, + *, + page_size: int = 100, + max_contacts: int | None = None, + progress: bool = True, + dry_run: bool = False, +) -> dict[str, int]: + """Fetch project contacts and write changed ones into the local cache.""" + if progress: + print("Fetching contact list…", flush=True) + stubs = client.fetch_contacts(limit=page_size) + if max_contacts is not None: + stubs = stubs[: int(max_contacts)] + result = {"seen": len(stubs), "fetched": 0, "skipped_cached": 0, "written": 0} + for index, stub in enumerate(stubs, start=1): + cid = stub.get("id") + if cid is None: + continue + if not cache.should_fetch(stub): + result["skipped_cached"] += 1 + if progress and (index == 1 or index % 50 == 0 or index == len(stubs)): + print(f" [{index}/{len(stubs)}] #{cid} skip-cached", flush=True) + continue + result["fetched"] += 1 + if dry_run: + if progress: + print(f" [{index}/{len(stubs)}] #{cid} would-fetch", flush=True) + continue + # Prefer full contact payload when list stub is thin + try: + full = client.get_contact(int(cid)) + except Exception: + full = stub + cache.write_contact(full) + result["written"] += 1 + if progress and (index == 1 or index % 25 == 0 or index == len(stubs)): + print(f" [{index}/{len(stubs)}] #{cid} written", flush=True) + if progress: + print( + f"contacts sync done: seen={result['seen']} fetched={result['fetched']} " + f"skipped={result['skipped_cached']} written={result['written']}", + flush=True, + ) + return result diff --git a/redmine_ticket_control_export.py b/redmine_ticket_control_export.py new file mode 100644 index 0000000..22f517b --- /dev/null +++ b/redmine_ticket_control_export.py @@ -0,0 +1,67 @@ +#!/usr/bin/env python3 +"""Training/export helpers for local control tags.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any, Iterable + + +def export_control_tag_jsonl( + *, + cache_dir: Path, + tags_payload: dict[str, Any], + issues_by_id: dict[int, dict[str, Any]], + tag: str | None = None, + out_path: Path, +) -> int: + """Write one JSONL record per tagged ticket for later LLM refine / training. + + Each line: {tag, issue_id, subject, status, assignee, contact_email, + properties, annotation, text_snippet} + """ + out_path = Path(out_path) + out_path.parent.mkdir(parents=True, exist_ok=True) + tags = tags_payload.get("tags") if isinstance(tags_payload, dict) else {} + if not isinstance(tags, dict): + tags = {} + names = [tag] if tag else sorted(tags.keys()) + count = 0 + with out_path.open("w", encoding="utf-8") as fh: + for name in names: + bucket = tags.get(name) or {} + tickets = bucket.get("tickets") if isinstance(bucket, dict) else {} + if not isinstance(tickets, dict): + continue + for iid_s, meta in tickets.items(): + try: + iid = int(iid_s) + except (TypeError, ValueError): + continue + issue = issues_by_id.get(iid) or {} + helpdesk_raw = issue.get("helpdesk_ticket") + helpdesk: dict[str, Any] = helpdesk_raw if isinstance(helpdesk_raw, dict) else {} + contact_raw = helpdesk.get("contact") + contact: dict[str, Any] = contact_raw if isinstance(contact_raw, dict) else {} + status_raw = issue.get("status") + status_obj: dict[str, Any] = status_raw if isinstance(status_raw, dict) else {} + assignee_raw = issue.get("assigned_to") + assignee_obj: dict[str, Any] = assignee_raw if isinstance(assignee_raw, dict) else {} + meta_d = meta if isinstance(meta, dict) else {} + rec = { + "tag": name, + "issue_id": iid, + "subject": issue.get("subject"), + "status": status_obj.get("name"), + "assignee": assignee_obj.get("name"), + "contact_email": contact.get("email") or helpdesk.get("from_address"), + "properties": meta_d.get("properties") or [], + "annotation": meta_d.get("annotation") or "", + "text_snippet": ( + f"{issue.get('subject') or ''}\n{(issue.get('description') or '')[:500]}" + ).strip(), + } + fh.write(json.dumps(rec, sort_keys=True) + "\n") + count += 1 + return count diff --git a/redmine_ticket_control_tags.py b/redmine_ticket_control_tags.py new file mode 100644 index 0000000..1631d40 --- /dev/null +++ b/redmine_ticket_control_tags.py @@ -0,0 +1,212 @@ +#!/usr/bin/env python3 +"""Local-only control tags for the power-user Redmine client. + +These are NOT Redmine issue tags and NOT CRM/contact tags. +Stored under /control-tags/ so they can be wiped without +touching Redmine or ticket JSON. +""" + +from __future__ import annotations + +import json +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + + +def now_iso() -> str: + return datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z") + + +def control_tags_root(cache_dir: Path) -> Path: + path = Path(cache_dir) / "control-tags" + path.mkdir(parents=True, exist_ok=True) + return path + + +def store_path(cache_dir: Path) -> Path: + return control_tags_root(cache_dir) / "tags.json" + + +def load_store(cache_dir: Path) -> dict[str, Any]: + path = store_path(cache_dir) + if not path.is_file(): + return {"version": 1, "tags": {}} + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return {"version": 1, "tags": {}} + if not isinstance(data, dict): + return {"version": 1, "tags": {}} + tags = data.get("tags") + if not isinstance(tags, dict): + data["tags"] = {} + data.setdefault("version", 1) + return data + + +def save_store(cache_dir: Path, data: dict[str, Any]) -> Path: + path = store_path(cache_dir) + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_suffix(".tmp") + payload = { + "version": int(data.get("version") or 1), + "updated_at": now_iso(), + "tags": data.get("tags") or {}, + } + tmp.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8") + tmp.replace(path) + return path + + +def normalize_tag(name: str) -> str: + tag = (name or "").strip().lower() + if not tag: + raise ValueError("Control tag name is required.") + if any(ch.isspace() for ch in tag): + raise ValueError(f"Control tag must not contain spaces: {name!r} (use hyphens).") + return tag + + +def add_tag( + cache_dir: Path, + *, + tag: str, + issue_id: int, + properties: list[str] | None = None, + annotation: str | None = None, +) -> dict[str, Any]: + tag = normalize_tag(tag) + data = load_store(cache_dir) + tags = data.setdefault("tags", {}) + entry = tags.setdefault(tag, {"tickets": {}}) + tickets = entry.setdefault("tickets", {}) + key = str(int(issue_id)) + existing = tickets.get(key) if isinstance(tickets.get(key), dict) else {} + props = list(properties or existing.get("properties") or []) + # de-dupe preserve order + seen: set[str] = set() + clean_props: list[str] = [] + for p in props: + p = str(p).strip() + if p and p not in seen: + seen.add(p) + clean_props.append(p) + tickets[key] = { + "issue_id": int(issue_id), + "properties": clean_props, + "annotation": annotation if annotation is not None else existing.get("annotation") or "", + "tagged_at": existing.get("tagged_at") or now_iso(), + "updated_at": now_iso(), + } + save_store(cache_dir, data) + return tickets[key] + + +def remove_tag(cache_dir: Path, *, tag: str, issue_id: int) -> bool: + tag = normalize_tag(tag) + data = load_store(cache_dir) + tags = data.get("tags") or {} + entry = tags.get(tag) + if not isinstance(entry, dict): + return False + tickets = entry.get("tickets") or {} + key = str(int(issue_id)) + if key not in tickets: + return False + del tickets[key] + if not tickets: + # keep empty tag shell for discovery, or drop — drop for cleanliness + del tags[tag] + save_store(cache_dir, data) + return True + + +def list_tags(cache_dir: Path) -> list[dict[str, Any]]: + data = load_store(cache_dir) + out: list[dict[str, Any]] = [] + for name, entry in sorted((data.get("tags") or {}).items()): + tickets = (entry or {}).get("tickets") or {} + out.append( + { + "tag": name, + "ticket_count": len(tickets) if isinstance(tickets, dict) else 0, + "issue_ids": sorted(int(k) for k in tickets.keys()) if isinstance(tickets, dict) else [], + } + ) + return out + + +def tickets_for_tag(cache_dir: Path, tag: str) -> list[int]: + tag = normalize_tag(tag) + data = load_store(cache_dir) + entry = (data.get("tags") or {}).get(tag) or {} + tickets = entry.get("tickets") or {} + if not isinstance(tickets, dict): + return [] + return sorted(int(k) for k in tickets.keys()) + + +def tags_for_issue(cache_dir: Path, issue_id: int) -> list[str]: + key = str(int(issue_id)) + data = load_store(cache_dir) + found: list[str] = [] + for name, entry in (data.get("tags") or {}).items(): + tickets = (entry or {}).get("tickets") or {} + if isinstance(tickets, dict) and key in tickets: + found.append(name) + return sorted(found) + + +def issue_tag_detail(cache_dir: Path, tag: str, issue_id: int) -> dict[str, Any] | None: + tag = normalize_tag(tag) + data = load_store(cache_dir) + entry = (data.get("tags") or {}).get(tag) or {} + tickets = entry.get("tickets") or {} + if not isinstance(tickets, dict): + return None + detail = tickets.get(str(int(issue_id))) + return detail if isinstance(detail, dict) else None + + +def wipe_all(cache_dir: Path) -> Path: + """Delete the entire control-tag store (does not touch ticket cache).""" + path = store_path(cache_dir) + if path.is_file(): + path.unlink() + return path + + +def filter_issues_by_control_tag( + issues: list[dict[str, Any]], + cache_dir: Path, + tag: str, +) -> list[dict[str, Any]]: + allowed = set(tickets_for_tag(cache_dir, tag)) + return [i for i in issues if i.get("id") is not None and int(i["id"]) in allowed] + + +def annotate_issues_with_control_tags( + issues: list[dict[str, Any]], + cache_dir: Path, +) -> list[dict[str, Any]]: + """Attach _control_tags list onto issue dicts (local only, not persisted to ticket JSON).""" + data = load_store(cache_dir) + by_issue: dict[int, list[str]] = {} + for name, entry in (data.get("tags") or {}).items(): + tickets = (entry or {}).get("tickets") or {} + if not isinstance(tickets, dict): + continue + for key in tickets: + try: + iid = int(key) + except (TypeError, ValueError): + continue + by_issue.setdefault(iid, []).append(name) + for issue in issues: + iid = issue.get("id") + if iid is None: + issue["_control_tags"] = [] + continue + issue["_control_tags"] = sorted(by_issue.get(int(iid), [])) + return issues diff --git a/redmine_ticket_drafts.py b/redmine_ticket_drafts.py new file mode 100644 index 0000000..67ff9d3 --- /dev/null +++ b/redmine_ticket_drafts.py @@ -0,0 +1,226 @@ +#!/usr/bin/env python3 +"""Local draft mutation queue for internal notes vs customer emails. + +CRITICAL DISTINCTION (do not collapse these): + +- internal_note → Redmine journal note only. NEVER emails the customer. + Optional private_notes. PUT /issues/:id.json + +- customer_email → Helpdesk "Send note" path. ALWAYS emails the customer + (when applied). POST /helpdesk/email_note + Cannot be private. + +Drafts are stored under /drafts/*.json until applied or cancelled. +""" + +from __future__ import annotations + +import json +import re +import uuid +from datetime import datetime, timezone +from pathlib import Path +from typing import Any +from urllib.parse import urlparse + + +KIND_INTERNAL_NOTE = "internal_note" +KIND_CUSTOMER_EMAIL = "customer_email" +KIND_FIELDS = "fields" +KIND_CONTACT_FIELDS = "contact_fields" +# Legacy aliases from earlier draft CLI +LEGACY_KIND_MAP = { + "note": KIND_INTERNAL_NOTE, + "helpdesk_reply": KIND_CUSTOMER_EMAIL, +} + +# Hosts allowed for write/apply without --allow-prod-writes +DEFAULT_SAFE_WRITE_HOST_PATTERNS = ( + r"^localhost$", + r"^127\.0\.0\.1$", + r"^192\.168\.\d+\.\d+$", + r"^10\.\d+\.\d+\.\d+$", + r"^retired\.int\.ldrprep\.com$", + r"^.*\.170$", # common LAN Redmine +) + + +def now_iso() -> str: + return datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z") + + +def normalize_kind(kind: str) -> str: + k = (kind or "").strip() + k = LEGACY_KIND_MAP.get(k, k) + if k not in {KIND_INTERNAL_NOTE, KIND_CUSTOMER_EMAIL, KIND_FIELDS, KIND_CONTACT_FIELDS}: + raise ValueError( + f"Unsupported draft kind {kind!r}. " + f"Use {KIND_INTERNAL_NOTE!r}, {KIND_CUSTOMER_EMAIL!r}, " + f"{KIND_FIELDS!r}, or {KIND_CONTACT_FIELDS!r}." + ) + return k + + +def kind_label(kind: str) -> str: + kind = normalize_kind(kind) + if kind == KIND_INTERNAL_NOTE: + return "INTERNAL NOTE (does NOT email customer)" + if kind == KIND_CUSTOMER_EMAIL: + return "CUSTOMER EMAIL (WILL email customer via Helpdesk)" + if kind == KIND_FIELDS: + return "FIELD UPDATE (does NOT email customer; changes issue fields)" + if kind == KIND_CONTACT_FIELDS: + return "CONTACT UPDATE (does NOT email; changes CRM contact fields/tags)" + return kind + + +def drafts_dir(cache_dir: Path) -> Path: + path = Path(cache_dir) / "drafts" + path.mkdir(parents=True, exist_ok=True) + return path + + +def assert_kind_flags(kind: str, *, private: bool) -> str: + kind = normalize_kind(kind) + if kind == KIND_CUSTOMER_EMAIL and private: + raise ValueError( + "customer_email drafts cannot be private. " + "Private notes never email; customer emails always email. " + "Use kind=internal_note with --private for staff-only notes." + ) + return kind + + +def add_draft( + cache_dir: Path, + *, + issue_id: int | None = None, + contact_id: int | None = None, + kind: str, + body: str = "", + private: bool = False, + fields: dict[str, Any] | None = None, + project: str | None = None, + base_url: str | None = None, +) -> Path: + kind = assert_kind_flags(kind, private=private) + body = (body or "").strip() + if kind in {KIND_INTERNAL_NOTE, KIND_CUSTOMER_EMAIL} and not body: + raise ValueError("Draft body is required for notes/emails.") + if kind == KIND_FIELDS and not fields: + raise ValueError("Field-update drafts require a non-empty fields payload.") + if kind == KIND_CONTACT_FIELDS: + if contact_id is None: + raise ValueError("contact_fields drafts require contact_id.") + if not fields: + raise ValueError("contact_fields drafts require a non-empty fields payload.") + if kind in {KIND_INTERNAL_NOTE, KIND_CUSTOMER_EMAIL, KIND_FIELDS} and issue_id is None: + raise ValueError(f"{kind} drafts require issue_id.") + draft_id = uuid.uuid4().hex[:12] + payload = { + "id": draft_id, + "created_at": now_iso(), + "status": "pending", + "project": project, + "base_url": base_url, + "issue_id": int(issue_id) if issue_id is not None else None, + "contact_id": int(contact_id) if contact_id is not None else None, + "kind": kind, + "kind_label": kind_label(kind), + "body": body, + "private": bool(private) if kind == KIND_INTERNAL_NOTE else False, + "fields": fields or {}, + "applied_at": None, + "error": None, + "emails_customer": kind == KIND_CUSTOMER_EMAIL, + } + path = drafts_dir(cache_dir) / f"{draft_id}.json" + path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8") + return path + + +def load_draft(cache_dir: Path, draft_id: str) -> tuple[Path, dict[str, Any]]: + path = drafts_dir(cache_dir) / f"{draft_id}.json" + if not path.is_file(): + matches = list(drafts_dir(cache_dir).glob(f"{draft_id}*.json")) + if len(matches) != 1: + raise FileNotFoundError(f"Draft not found or ambiguous: {draft_id}") + path = matches[0] + data = json.loads(path.read_text(encoding="utf-8")) + data["kind"] = normalize_kind(str(data.get("kind") or "")) + return path, data + + +def list_drafts(cache_dir: Path, status: str | None = "pending") -> list[dict[str, Any]]: + items: list[dict[str, Any]] = [] + root = drafts_dir(cache_dir) + for path in sorted(root.glob("*.json")): + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + continue + if status and data.get("status") != status: + continue + try: + data["kind"] = normalize_kind(str(data.get("kind") or "")) + except ValueError: + pass + data["kind_label"] = kind_label(str(data.get("kind") or "internal_note")) if data.get("kind") else "" + data["_path"] = str(path) + items.append(data) + return items + + +def cancel_draft(cache_dir: Path, draft_id: str) -> bool: + try: + path, data = load_draft(cache_dir, draft_id) + except FileNotFoundError: + return False + data["status"] = "cancelled" + data["cancelled_at"] = now_iso() + path.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8") + return True + + +def cancel_all_pending(cache_dir: Path) -> int: + count = 0 + for draft in list_drafts(cache_dir, status="pending"): + if cancel_draft(cache_dir, str(draft["id"])): + count += 1 + return count + + +def mark_draft_applied(cache_dir: Path, draft_id: str, *, result: dict[str, Any] | None = None) -> None: + path, data = load_draft(cache_dir, draft_id) + data["status"] = "applied" + data["applied_at"] = now_iso() + data["error"] = None + data["apply_result"] = result or {} + path.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8") + + +def mark_draft_error(cache_dir: Path, draft_id: str, error: str) -> None: + path, data = load_draft(cache_dir, draft_id) + data["status"] = "error" + data["error"] = error + data["error_at"] = now_iso() + path.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8") + + +def host_is_safe_for_writes(base_url: str, patterns: tuple[str, ...] = DEFAULT_SAFE_WRITE_HOST_PATTERNS) -> bool: + host = (urlparse(base_url).hostname or "").lower() + if not host: + return False + return any(re.search(pat, host) for pat in patterns) + + +def assert_write_target_allowed(base_url: str, *, allow_prod_writes: bool) -> None: + if allow_prod_writes: + return + if host_is_safe_for_writes(base_url): + return + raise RuntimeError( + f"Refusing writes to {base_url!r}. " + "Default allow-list is LAN/test hosts only. " + "Re-run against the test Redmine, or pass --allow-prod-writes if you truly intend production." + ) diff --git a/redmine_ticket_entities.py b/redmine_ticket_entities.py new file mode 100644 index 0000000..5d77401 --- /dev/null +++ b/redmine_ticket_entities.py @@ -0,0 +1,173 @@ +#!/usr/bin/env python3 +"""Extract common CS entity identifiers from ticket text (pattern-based MVP). + +Does not call NMI. Purpose: surface tracking numbers / order-like IDs / SKUs +so humans and agents can jump systems without copy-paste gymnastics. + +Patterns are conservative heuristics — false positives are possible; treat hits +as hints, not ground truth. +""" + +from __future__ import annotations + +import re +from typing import Any, Iterable + + +# USPS: 20–22 digit, or 420… + 26 digits, or 13-char ending US +_USPS = re.compile( + r"\b(" + r"9[0-9]{19,21}" + r"|420\d{5}\d{20,26}" + r"|[A-Z]{2}\d{9}US" + r")\b", + re.I, +) +_UPS = re.compile(r"\b(1Z[0-9A-Z]{16})\b", re.I) +_FEDEX = re.compile(r"\b(\d{12}|\d{15})\b") +_AMZ_ORDER = re.compile(r"\b(\d{3}-\d{7}-\d{7})\b") +# Prefer digit-heavy order captures; reject pure alpha HTML fragments +_ORDER_LABEL = re.compile( + r"(?:order\s*(?:id|#|number|no\.?)|po\s*(?:#|number)?)\s*[:=\s#]*([A-Z0-9][A-Z0-9\-_/]{3,32})", + re.I, +) +_ITEM_LABEL = re.compile( + r"(?:item\s*(?:#|no\.?|number)|sku|merchant\s*sku)\s*[:=\s#]*([A-Z0-9][A-Z0-9\-_/]{2,40})", + re.I, +) +_ACCOUNT = re.compile(r"\baccount\s*[#:]?\s*([A-Z0-9]{4,20})\b", re.I) +# LDR unit item numbers: 9 digits starting with 2 +_ITEM_NO_BARE = re.compile(r"\b(2\d{8})\b") + +_ORDER_STOP = { + "number", + "id", + "no", + "none", + "null", + "true", + "false", + "class", + "style", + "mode", + "text", + "title", + "guide", + "html", + "body", + "rted", + "rtal", + "licy", + "lation-mode", + "rtant", + "nsive", + "nsive-text", + "nsive-title", + "nsive-h2", + "ints", +} + + +def _text_from_issue(issue: dict[str, Any]) -> str: + chunks: list[str] = [] + for key in ("subject", "description"): + val = issue.get(key) + if val: + chunks.append(str(val)) + helpdesk = issue.get("helpdesk_ticket") + if isinstance(helpdesk, dict): + for key in ("from_address", "to_address", "message_id"): + if helpdesk.get(key): + chunks.append(str(helpdesk[key])) + contact = helpdesk.get("contact") + if isinstance(contact, dict): + for key in ("email", "company", "name", "first_name", "last_name"): + if contact.get(key): + chunks.append(str(contact[key])) + journals = issue.get("journals") + if isinstance(journals, list): + for j in journals[-15:]: + if isinstance(j, dict) and j.get("notes"): + chunks.append(str(j["notes"])) + return "\n".join(chunks) + + +def extract_entities(text: str) -> list[dict[str, str]]: + """Return unique {type, value, confidence} hits from free text.""" + hits: list[dict[str, str]] = [] + seen: set[tuple[str, str]] = set() + + def add(kind: str, value: str, confidence: str = "high") -> None: + value = value.strip() + if not value: + return + key = (kind, value.upper()) + if key in seen: + return + seen.add(key) + hits.append({"type": kind, "value": value, "confidence": confidence}) + + for m in _UPS.finditer(text): + add("tracking_ups", m.group(1).upper()) + for m in _USPS.finditer(text): + add("tracking_usps", m.group(1).upper()) + for m in _AMZ_ORDER.finditer(text): + add("order_amazon", m.group(1)) + for m in _ORDER_LABEL.finditer(text): + val = m.group(1) + if re.fullmatch(r"\d{3}-\d{7}-\d{7}", val): + add("order_amazon", val) + continue + if not re.search(r"\d", val): + continue + if val.lower() in _ORDER_STOP: + continue + # Require at least 5 chars and a digit; reject long alpha+token noise + if len(val) < 5: + continue + if re.search(r"[a-zA-Z]{6,}", val) and not re.search(r"\d{4,}", val): + continue # e.g. MGdHZfSeDzi_BhM6 without enough digits + conf = "high" if re.fullmatch(r"\d{5,12}", val) else "medium" + add("order_id", val, confidence=conf) + for m in _ITEM_LABEL.finditer(text): + val = m.group(1) + if re.fullmatch(r"2\d{8}", val): + add("item_no", val, confidence="high") + elif re.search(r"\d", val) or re.search(r"[A-Z]{2,}", val, re.I): + add("item_or_sku", val, confidence="medium") + for m in _ITEM_NO_BARE.finditer(text): + add("item_no", m.group(1), confidence="medium") + for m in _ACCOUNT.finditer(text): + val = m.group(1) + if val.lower() in {"within", "number", "setup", "your", "the"}: + continue + if not re.search(r"\d", val) and len(val) < 6: + continue + add("account_ref", val, confidence="medium") + + if re.search(r"\b(fedex|tracking|shipment|shipped|carrier)\b", text, re.I): + for m in _FEDEX.finditer(text): + # skip USPS/item_no already captured lengths that are tracking + val = m.group(1) + if val.startswith("9") and len(val) >= 20: + continue + if re.fullmatch(r"2\d{8}", val): + continue + add("tracking_fedex_candidate", val, confidence="low") + + return hits + + +def extract_entities_from_issue(issue: dict[str, Any]) -> list[dict[str, str]]: + return extract_entities(_text_from_issue(issue)) + + +def format_entities_line(entities: Iterable[dict[str, str]]) -> str: + parts = [] + for e in entities: + conf = e.get("confidence") or "high" + if conf == "high": + parts.append(f"{e['type']}:{e['value']}") + else: + parts.append(f"{e['type']}:{e['value']}({conf})") + return "; ".join(parts) diff --git a/redmine_ticket_filters.py b/redmine_ticket_filters.py new file mode 100644 index 0000000..8e48b8d --- /dev/null +++ b/redmine_ticket_filters.py @@ -0,0 +1,199 @@ +#!/usr/bin/env python3 +"""Saved property filters and filesystem queue materialization. + +Saved filters live as text files under a directory (default: ./saved-filters). +Each file contains one property expression (same language as search --where). +Comments (# ...) and blank lines are ignored; remaining lines are joined with AND +unless the file is a single multi-line expression with parentheses. + +Load by: + - basename without extension: helpdeskEmailMatchCompany + - alias.helpdeskEmailMatchCompany + - @helpdeskEmailMatchCompany + - path/to/file.expr +""" + +from __future__ import annotations + +import json +import os +from pathlib import Path +from typing import Any, Iterable + +from redmine_ticket_query import QueryError, filter_by_query + + +DEFAULT_SAVED_FILTERS_DIR = Path("saved-filters") +DEFAULT_QUEUES_DIR_NAME = "queues" + + +def default_saved_filters_dir(repo_root: Path | None = None) -> Path: + root = repo_root or Path.cwd() + return root / "saved-filters" + + +def strip_filter_source(text: str) -> str: + lines: list[str] = [] + for raw in text.splitlines(): + line = raw.strip() + if not line or line.startswith("#"): + continue + lines.append(line) + return " ".join(lines).strip() + + +def resolve_filter_name(raw: str) -> str: + name = raw.strip() + if name.startswith("alias."): + name = name[len("alias.") :] + if name.startswith("@"): + name = name[1:] + return name.strip() + + +def load_saved_filter_expression(name_or_path: str, search_dirs: Iterable[Path]) -> str: + """Load expression from a saved filter name or filesystem path.""" + raw = name_or_path.strip() + path = Path(raw) + candidates: list[Path] = [] + if path.is_file(): + candidates.append(path) + else: + name = resolve_filter_name(raw) + for directory in search_dirs: + for ext in (".expr", ".filter", ".wtf", ".txt", ""): + candidate = directory / f"{name}{ext}" + if candidate.is_file(): + candidates.append(candidate) + break + if candidates: + break + if not candidates: + searched = ", ".join(str(d) for d in search_dirs) + raise QueryError(f"Saved filter not found: {name_or_path!r} (looked in {searched})") + expression = strip_filter_source(candidates[0].read_text(encoding="utf-8")) + if not expression: + raise QueryError(f"Saved filter file is empty: {candidates[0]}") + return expression + + +def expand_where_list( + where_items: list[str] | None, + saved_dirs: Iterable[Path], +) -> list[str]: + """Expand @alias / alias.name / bare saved names; leave plain expressions as-is. + + Heuristic: if the string starts with @ or alias. or is a single identifier + matching a saved file, load it; otherwise treat as inline expression. + """ + out: list[str] = [] + for item in where_items or []: + text = item.strip() + if not text: + continue + if text.startswith("@") or text.startswith("alias."): + out.append(load_saved_filter_expression(text, saved_dirs)) + continue + # bare identifier → try saved filter first + if re_fullmatch_identifier(text): + try: + out.append(load_saved_filter_expression(text, saved_dirs)) + continue + except QueryError: + pass + out.append(text) + return out + + +def re_fullmatch_identifier(text: str) -> bool: + import re + + return bool(re.fullmatch(r"[A-Za-z_][A-Za-z0-9_-]*", text)) + + +def load_queue_definitions(path: Path) -> list[dict[str, Any]]: + """Load queue definitions from JSON. + + Format: + { + "queues": [ + { + "name": "customer-service-triage-needed", + "where": ["assigned_to IS NULL", "category.name IS NULL"], + "max_tickets": 200, + "order": "newest-updated-first" + } + ] + } + """ + data = json.loads(path.read_text(encoding="utf-8")) + if isinstance(data, list): + queues = data + elif isinstance(data, dict): + queues = data.get("queues") or data.get("filters") or [] + else: + raise ValueError(f"Invalid queue config: {path}") + if not isinstance(queues, list): + raise ValueError(f"queues must be a list in {path}") + return [q for q in queues if isinstance(q, dict) and q.get("name")] + + +def materialize_queue( + *, + name: str, + issues: list[dict[str, Any]], + cache_dir: Path, + project: str, + queues_root: Path, +) -> dict[str, Any]: + """Create/refresh a directory of symlinks to matching ticket JSON files. + + Layout: + // + 12345.json -> //12345.json + ... + .queue-manifest.json + """ + queue_dir = queues_root / name + queue_dir.mkdir(parents=True, exist_ok=True) + + # Remove old ticket symlinks (keep non-matching cleaned) + for existing in queue_dir.glob("*.json"): + if existing.name.startswith("."): + continue + if existing.is_symlink() or existing.is_file(): + existing.unlink() + + project_dir = Path(cache_dir) / project + linked = 0 + missing = 0 + for issue in issues: + issue_id = issue.get("id") + if issue_id is None: + continue + target = project_dir / f"{issue_id}.json" + link_path = queue_dir / f"{issue_id}.json" + if not target.is_file(): + missing += 1 + continue + # relative symlink when possible + try: + rel = os.path.relpath(target, start=queue_dir) + link_path.symlink_to(rel) + except OSError: + link_path.symlink_to(target) + linked += 1 + + manifest = { + "name": name, + "project": project, + "cache_dir": str(cache_dir), + "linked": linked, + "missing_source": missing, + "issue_ids": [int(i["id"]) for i in issues if i.get("id") is not None], + } + (queue_dir / ".queue-manifest.json").write_text( + json.dumps(manifest, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + return manifest diff --git a/redmine_ticket_meta.py b/redmine_ticket_meta.py new file mode 100644 index 0000000..0dd45cc --- /dev/null +++ b/redmine_ticket_meta.py @@ -0,0 +1,264 @@ +#!/usr/bin/env python3 +"""Resolve human-friendly Redmine names to IDs for set-fields / mutations. + +Caches project metadata under /meta// so agents can use +--status Closed instead of --status-id 5 without hammering the API every run. +""" + +from __future__ import annotations + +import json +import time +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Callable + + +def now_iso() -> str: + return datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z") + + +def meta_dir(cache_dir: Path, project: str) -> Path: + path = Path(cache_dir) / "meta" / project + path.mkdir(parents=True, exist_ok=True) + return path + + +def _load_json(path: Path) -> dict[str, Any] | None: + if not path.is_file(): + return None + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return None + return data if isinstance(data, dict) else None + + +def _save_json(path: Path, data: dict[str, Any]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_suffix(path.suffix + ".tmp") + tmp.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8") + tmp.replace(path) + + +def _fresh(path: Path, max_age_seconds: int) -> bool: + if not path.is_file(): + return False + age = time.time() - path.stat().st_mtime + return age <= max_age_seconds + + +def normalize_name(value: str) -> str: + return " ".join(str(value).strip().lower().split()) + + +def fetch_issue_statuses(client: Any) -> list[dict[str, Any]]: + data = client.request("GET", "/issue_statuses.json") + statuses = data.get("issue_statuses") if isinstance(data, dict) else None + return list(statuses) if isinstance(statuses, list) else [] + + +def fetch_issue_priorities(client: Any) -> list[dict[str, Any]]: + data = client.request("GET", "/enumerations/issue_priorities.json") + priorities = data.get("issue_priorities") if isinstance(data, dict) else None + return list(priorities) if isinstance(priorities, list) else [] + + +def fetch_project_categories(client: Any, project: str) -> list[dict[str, Any]]: + data = client.request("GET", f"/projects/{project}/issue_categories.json") + cats = data.get("issue_categories") if isinstance(data, dict) else None + return list(cats) if isinstance(cats, list) else [] + + +def fetch_project_memberships(client: Any, project: str) -> list[dict[str, Any]]: + """Memberships include nested user objects for assignee resolution.""" + members: list[dict[str, Any]] = [] + offset = 0 + total = None + while total is None or offset < total: + data = client.request( + "GET", + f"/projects/{project}/memberships.json", + {"limit": 100, "offset": offset}, + ) + page = data.get("memberships") if isinstance(data, dict) else [] + if not isinstance(page, list): + page = [] + members.extend(page) + total = int(data.get("total_count", len(members))) if isinstance(data, dict) else len(members) + if not page: + break + offset += len(page) + return members + + +def users_from_memberships(memberships: list[dict[str, Any]]) -> list[dict[str, Any]]: + users: list[dict[str, Any]] = [] + seen: set[int] = set() + for m in memberships: + user = m.get("user") if isinstance(m, dict) else None + if not isinstance(user, dict) or user.get("id") is None: + continue + uid = int(user["id"]) + if uid in seen: + continue + seen.add(uid) + users.append( + { + "id": uid, + "name": user.get("name") or f"{user.get('firstname', '')} {user.get('lastname', '')}".strip(), + "login": user.get("login"), + } + ) + return users + + +def refresh_meta_cache( + client: Any, + cache_dir: Path, + project: str, + *, + force: bool = False, + max_age_seconds: int = 6 * 3600, +) -> dict[str, Path]: + root = meta_dir(cache_dir, project) + paths = { + "statuses": root / "issue_statuses.json", + "priorities": root / "issue_priorities.json", + "categories": root / "issue_categories.json", + "users": root / "project_users.json", + } + if not force and all(_fresh(p, max_age_seconds) for p in paths.values()): + return paths + + statuses = fetch_issue_statuses(client) + priorities = fetch_issue_priorities(client) + categories = fetch_project_categories(client, project) + memberships = fetch_project_memberships(client, project) + users = users_from_memberships(memberships) + + _save_json(paths["statuses"], {"fetched_at": now_iso(), "issue_statuses": statuses}) + _save_json(paths["priorities"], {"fetched_at": now_iso(), "issue_priorities": priorities}) + _save_json(paths["categories"], {"fetched_at": now_iso(), "issue_categories": categories}) + _save_json(paths["users"], {"fetched_at": now_iso(), "users": users}) + return paths + + +def _items_from_file(path: Path, key: str) -> list[dict[str, Any]]: + data = _load_json(path) or {} + items = data.get(key) + return list(items) if isinstance(items, list) else [] + + +def resolve_id_by_name( + items: list[dict[str, Any]], + name: str, + *, + id_key: str = "id", + name_keys: tuple[str, ...] = ("name",), +) -> int: + target = normalize_name(name) + if target.isdigit(): + return int(target) + matches: list[dict[str, Any]] = [] + for item in items: + if not isinstance(item, dict): + continue + candidates = [] + for nk in name_keys: + val = item.get(nk) + if val is not None: + candidates.append(normalize_name(str(val))) + # also full first+last style + if item.get("firstname") or item.get("lastname"): + candidates.append( + normalize_name(f"{item.get('firstname', '')} {item.get('lastname', '')}") + ) + if any(c == target for c in candidates if c): + matches.append(item) + continue + # substring match as fallback + if any(target in c for c in candidates if c): + matches.append(item) + if not matches: + available = [] + for item in items[:30]: + if isinstance(item, dict): + available.append(str(item.get("name") or item.get("login") or item.get(id_key))) + raise LookupError( + f"No match for {name!r}. Examples: {', '.join(available) or '(empty meta cache)'}" + ) + if len(matches) > 1: + # prefer exact name over substring + exact = [ + m + for m in matches + if any(normalize_name(str(m.get(k) or "")) == target for k in name_keys) + or normalize_name(f"{m.get('firstname', '')} {m.get('lastname', '')}") == target + ] + if len(exact) == 1: + return int(exact[0][id_key]) + labels = [str(m.get("name") or m.get("login") or m.get(id_key)) for m in matches[:8]] + raise LookupError(f"Ambiguous match for {name!r}: {', '.join(labels)}") + return int(matches[0][id_key]) + + +def load_meta_lists(cache_dir: Path, project: str) -> dict[str, list[dict[str, Any]]]: + root = meta_dir(cache_dir, project) + return { + "statuses": _items_from_file(root / "issue_statuses.json", "issue_statuses"), + "priorities": _items_from_file(root / "issue_priorities.json", "issue_priorities"), + "categories": _items_from_file(root / "issue_categories.json", "issue_categories"), + "users": _items_from_file(root / "project_users.json", "users"), + } + + +def resolve_field_names( + cache_dir: Path, + project: str, + *, + status_name: str | None = None, + assignee_name: str | None = None, + priority_name: str | None = None, + category_name: str | None = None, + client: Any | None = None, + refresh: bool = False, +) -> dict[str, int]: + """Return numeric id fields from optional name strings. + + If meta cache is empty and client is provided, refresh automatically. + """ + lists = load_meta_lists(cache_dir, project) + need_refresh = refresh or ( + client is not None + and ( + (status_name and not lists["statuses"]) + or (assignee_name and not lists["users"]) + or (priority_name and not lists["priorities"]) + or (category_name and not lists["categories"]) + ) + ) + if need_refresh and client is not None: + refresh_meta_cache(client, cache_dir, project, force=True) + lists = load_meta_lists(cache_dir, project) + + out: dict[str, int] = {} + if status_name: + out["status_id"] = resolve_id_by_name(lists["statuses"], status_name) + if priority_name: + out["priority_id"] = resolve_id_by_name(lists["priorities"], priority_name) + if category_name: + if normalize_name(category_name) in {"", "none", "clear", "null"}: + out["category_id"] = 0 + else: + out["category_id"] = resolve_id_by_name(lists["categories"], category_name) + if assignee_name: + if normalize_name(assignee_name) in {"", "none", "unassigned", "nobody", "null", "0"}: + out["assigned_to_id"] = 0 + else: + out["assigned_to_id"] = resolve_id_by_name( + lists["users"], + assignee_name, + name_keys=("name", "login"), + ) + return out diff --git a/redmine_ticket_nmi.py b/redmine_ticket_nmi.py new file mode 100644 index 0000000..3cbee5e --- /dev/null +++ b/redmine_ticket_nmi.py @@ -0,0 +1,628 @@ +#!/usr/bin/env python3 +"""Bridge Redmine ticket entities → NMI agent CLI (mrscClient-php bin/nmi). + +Pattern extract stays in redmine_ticket_entities.py (no NMI I/O). +This module shells out to the NMI CLI for resolve / cs-brief using those hints. +""" + +from __future__ import annotations + +import json +import os +import re +import shutil +import subprocess +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Iterable, Optional + +# Prefer env overrides for agents/profiles +DEFAULT_NMI_BIN = os.environ.get( + "NMI_CLI", + "/home/jason/projects/mrscClient-php/bin/nmi", +) +DEFAULT_ENV_FILE = os.environ.get( + "NMI_ENV_FILE", + "/home/jason/projects/mrscClient-php/examples/modern/.env", +) +DEFAULT_MODE = os.environ.get("NMI_MODE", "dev") + +# Entity types that map cleanly into NMI resolve ids +_TRACKING_TYPES = { + "tracking_ups", + "tracking_usps", + "tracking_fedex_candidate", +} +_ORDER_TYPES = {"order_amazon", "order_id"} +_ITEM_TYPES = {"item_no", "item_or_sku"} +_ACCOUNT_TYPES = {"account_ref"} +# NMI deep link templates (from obsidian-ldr/INBOX/WIP/Linking to Entities Within NMI.md) +NMI_BASE = "https://app.ldrprep.com" +NMI_LINKS = { + "item_no": { + "manager": lambda v: f"{NMI_BASE}/?page=itemManager&itemNo={v}", + "statement": lambda v: f"{NMI_BASE}/?page=itemStatement&itemNo={v}", + }, + "item_or_sku": { + "manager": lambda v: f"{NMI_BASE}/?page=itemManager&itemNo={v}", + }, + "tracking_ups": {"box_history": lambda v: f"{NMI_BASE}/?page=boxFinder&tracking_number={v}&action=search"}, + "tracking_usps": {"box_history": lambda v: f"{NMI_BASE}/?page=boxFinder&tracking_number={v}&action=search"}, + "tracking_fedex_candidate": {"box_history": lambda v: f"{NMI_BASE}/?page=boxFinder&tracking_number={v}&action=search"}, + "order_id": {"order_summary": lambda v: f"{NMI_BASE}/?page=requests&filter=view&id={v}"}, + "request_id": {"package_history": lambda v: f"{NMI_BASE}/?page=boxFinder&request_id={v}"}, + "account_ref": {"user_info": lambda v: f"{NMI_BASE}/?page=userInfo&accountNo={v}"}, +} + + +# LDR unit item numbers commonly 9 digits starting with 2 +_ITEM_NO_BARE = re.compile(r"\b(2\d{8})\b") + + +@dataclass(frozen=True) +class NmiRuntimeProfile: + """Resolved NMI execution settings plus a preflight result.""" + + mode: str + env_file: Path + nmi_bin: Path + endpoint: Optional[str] + ready: bool + problem: Optional[str] = None + + +def resolve_runtime_profile( + *, + mode: Optional[str] = None, + env_file: Optional[str] = None, + nmi_bin: Optional[str] = None, + endpoint: Optional[str] = None, + environ: Optional[dict[str, str]] = None, +) -> NmiRuntimeProfile: + """Resolve the exact profile passed to PHP before an NMI subprocess starts. + + An env file that is empty or missing is always rejected. It is too easy for + the checked-in example `.env` to otherwise select a surprising dev/default + path while making the TUI look successful. + """ + + env = os.environ if environ is None else environ + requested_mode = (mode or env.get("NMI_MODE") or DEFAULT_MODE).strip().lower() + normalized_mode = "production" if requested_mode in {"prod", "production"} else requested_mode + selected_env_file = Path(env_file or env.get("NMI_ENV_FILE") or DEFAULT_ENV_FILE).expanduser() + selected_nmi_bin = Path(nmi_bin or env.get("NMI_CLI") or DEFAULT_NMI_BIN).expanduser() + + mode_upper = normalized_mode.upper() + selected_endpoint = ( + endpoint + or env.get(f"NMI_ENDPOINT_{mode_upper}") + or env.get("NMI_ENDPOINT") + or None + ) + + if not selected_env_file.is_file(): + return NmiRuntimeProfile( + mode=normalized_mode, + env_file=selected_env_file, + nmi_bin=selected_nmi_bin, + endpoint=selected_endpoint, + ready=False, + problem=f"NMI env file does not exist: {selected_env_file}", + ) + if selected_env_file.stat().st_size == 0: + return NmiRuntimeProfile( + mode=normalized_mode, + env_file=selected_env_file, + nmi_bin=selected_nmi_bin, + endpoint=selected_endpoint, + ready=False, + problem=f"NMI env file is empty: {selected_env_file}", + ) + if not selected_nmi_bin.is_file(): + return NmiRuntimeProfile( + mode=normalized_mode, + env_file=selected_env_file, + nmi_bin=selected_nmi_bin, + endpoint=selected_endpoint, + ready=False, + problem=f"NMI CLI does not exist: {selected_nmi_bin}", + ) + + return NmiRuntimeProfile( + mode=normalized_mode, + env_file=selected_env_file, + nmi_bin=selected_nmi_bin, + endpoint=selected_endpoint, + ready=True, + ) + + +@dataclass +class NmiLookupPlan: + """Deduped identifiers to try against NMI, highest confidence first.""" + + tracking: list[str] = field(default_factory=list) + order_id: list[str] = field(default_factory=list) + item_no: list[str] = field(default_factory=list) + account: list[str] = field(default_factory=list) + skipped: list[dict[str, str]] = field(default_factory=list) + + def is_empty(self) -> bool: + return not (self.tracking or self.order_id or self.item_no or self.account) + + +def enrich_entities_with_item_nos( + entities: list[dict[str, str]], text: str +) -> list[dict[str, str]]: + """Add bare LDR-style item numbers when not already present.""" + seen = {(e.get("type"), (e.get("value") or "").upper()) for e in entities} + out = list(entities) + for m in _ITEM_NO_BARE.finditer(text or ""): + val = m.group(1) + key = ("item_no", val) + if key in seen: + continue + # Avoid treating pure tracking-ish long numbers: item is exactly 9 digits + out.append({"type": "item_no", "value": val, "confidence": "medium"}) + seen.add(key) + return out + + +def plan_from_entities(entities: Iterable[dict[str, str]]) -> NmiLookupPlan: + plan = NmiLookupPlan() + seen_t: set[str] = set() + seen_o: set[str] = set() + seen_i: set[str] = set() + seen_a: set[str] = set() + + # Sort high confidence first + ordered = sorted( + entities, + key=lambda e: {"high": 0, "medium": 1, "low": 2}.get(e.get("confidence") or "high", 3), + ) + + for e in ordered: + kind = e.get("type") or "" + val = (e.get("value") or "").strip() + conf = e.get("confidence") or "high" + if not val: + continue + if kind in _TRACKING_TYPES: + if conf == "low" and kind == "tracking_fedex_candidate": + plan.skipped.append(e) + continue + u = val.upper() + if u not in seen_t: + seen_t.add(u) + plan.tracking.append(val) + elif kind in _ORDER_TYPES: + if val not in seen_o: + seen_o.add(val) + plan.order_id.append(val) + elif kind in _ITEM_TYPES or kind == "item_no": + # Prefer pure numeric item_no; SKU-like may still help as order/item search later + if re.fullmatch(r"\d{6,12}", val): + if val not in seen_i: + seen_i.add(val) + plan.item_no.append(val) + else: + plan.skipped.append({**e, "reason": "sku_not_numeric_item"}) + elif kind in _ACCOUNT_TYPES: + if val not in seen_a: + seen_a.add(val) + plan.account.append(val) + else: + plan.skipped.append(e) + return plan + + +def _php_bin() -> str: + return shutil.which("php") or "php" + + +def run_nmi( + args: list[str], + *, + mode: str = DEFAULT_MODE, + env_file: str = DEFAULT_ENV_FILE, + nmi_bin: str = DEFAULT_NMI_BIN, + endpoint: Optional[str] = None, + timeout: int = 60, +) -> subprocess.CompletedProcess[str]: + cmd = [_php_bin(), nmi_bin, *args, f"--mode={mode}", f"--env-file={env_file}"] + if endpoint: + cmd.append(f"--endpoint={endpoint}") + return subprocess.run( + cmd, + capture_output=True, + text=True, + timeout=timeout, + check=False, + ) + + +def try_resolve_ids( + plan: NmiLookupPlan, + *, + mode: str = DEFAULT_MODE, + env_file: str = DEFAULT_ENV_FILE, + nmi_bin: str = DEFAULT_NMI_BIN, + endpoint: Optional[str] = None, + include_pictures: bool = False, + max_attempts: int = 6, +) -> dict[str, Any]: + """Try NMI resolve with identifiers from the plan until one succeeds.""" + attempts: list[dict[str, Any]] = [] + candidates: list[list[str]] = [] + + for item in plan.item_no: + candidates.append(["resolve", f"--item-no={item}"]) + for tn in plan.tracking: + candidates.append(["resolve", f"--tracking={tn}"]) + for oid in plan.order_id: + candidates.append(["resolve", f"--order-id={oid}"]) + for acct in plan.account: + candidates.append(["resolve", f"--account={acct}"]) + + if include_pictures: + for c in candidates: + c.append("--pictures") + + for args in candidates[:max_attempts]: + proc = run_nmi(args, mode=mode, env_file=env_file, nmi_bin=nmi_bin, endpoint=endpoint) + attempt = { + "args": args, + "returncode": proc.returncode, + "stderr": (proc.stderr or "").strip()[:500], + } + if proc.returncode == 0 and (proc.stdout or "").strip(): + try: + data = json.loads(proc.stdout) + except json.JSONDecodeError as exc: + attempt["error"] = f"json_decode: {exc}" + attempts.append(attempt) + continue + attempt["ok"] = True + attempts.append(attempt) + return {"ok": True, "result": data, "attempts": attempts, "used_args": args} + attempt["ok"] = False + attempts.append(attempt) + + return {"ok": False, "result": None, "attempts": attempts, "used_args": None} + + +def try_cs_brief( + plan: NmiLookupPlan, + *, + mode: str = DEFAULT_MODE, + env_file: str = DEFAULT_ENV_FILE, + nmi_bin: str = DEFAULT_NMI_BIN, + endpoint: Optional[str] = None, + max_attempts: int = 6, +) -> dict[str, Any]: + """Try cs-brief with plan identifiers until one returns text.""" + attempts: list[dict[str, Any]] = [] + candidates: list[list[str]] = [] + for item in plan.item_no: + candidates.append(["cs-brief", f"--item-no={item}"]) + for tn in plan.tracking: + candidates.append(["cs-brief", f"--tracking={tn}"]) + for oid in plan.order_id: + candidates.append(["cs-brief", f"--order-id={oid}"]) + for acct in plan.account: + candidates.append(["cs-brief", f"--account={acct}"]) + + for args in candidates[:max_attempts]: + proc = run_nmi(args, mode=mode, env_file=env_file, nmi_bin=nmi_bin, endpoint=endpoint) + attempt = { + "args": args, + "returncode": proc.returncode, + "stderr": (proc.stderr or "").strip()[:500], + } + text = (proc.stdout or "").strip() + if proc.returncode == 0 and text and "=== NMI CS BRIEF ===" in text: + attempt["ok"] = True + attempts.append(attempt) + return {"ok": True, "text": text + ("\n" if not text.endswith("\n") else ""), "attempts": attempts, "used_args": args} + attempt["ok"] = False + if text: + attempt["stdout_preview"] = text[:200] + attempts.append(attempt) + + return {"ok": False, "text": None, "attempts": attempts, "used_args": None} + + +def brief_for_issue( + issue: dict[str, Any], + entities: list[dict[str, str]], + *, + mode: str = DEFAULT_MODE, + env_file: str = DEFAULT_ENV_FILE, + nmi_bin: str = DEFAULT_NMI_BIN, + endpoint: Optional[str] = None, + with_history: bool = True, +) -> dict[str, Any]: + profile = resolve_runtime_profile( + mode=mode, + env_file=env_file, + nmi_bin=nmi_bin, + endpoint=endpoint, + ) + runtime = { + "mode": profile.mode, + "env_file": str(profile.env_file), + "nmi_bin": str(profile.nmi_bin), + "endpoint": profile.endpoint, + "ready": profile.ready, + "problem": profile.problem, + } + + # Prefer public extract path text rebuild without private imports + try: + from redmine_ticket_entities import extract_entities_from_issue as _ex + + # rebuild text via extract side-channel: re-run extract on issue already given + # Use journals/subject from issue for bare item_no enrichment only + chunks = [] + for key in ("subject", "description"): + if issue.get(key): + chunks.append(str(issue[key])) + journals = issue.get("journals") + if isinstance(journals, list): + for j in journals[-15:]: + if isinstance(j, dict) and j.get("notes"): + chunks.append(str(j["notes"])) + text = "\n".join(chunks) + _ = _ex # silence unused if tree-shaken + except Exception: + text = str(issue.get("subject") or "") + "\n" + str(issue.get("description") or "") + + ents = enrich_entities_with_item_nos(entities, text) + plan = plan_from_entities(ents) + out: dict[str, Any] = { + "issue_id": issue.get("id"), + "runtime": runtime, + "plan": { + "tracking": plan.tracking, + "order_id": plan.order_id, + "item_no": plan.item_no, + "account": plan.account, + "skipped": plan.skipped, + }, + "entities": ents, + } + if not profile.ready: + out["ok"] = False + out["error"] = "nmi_runtime_not_ready" + return out + if plan.is_empty(): + out["ok"] = False + out["error"] = "no_nmi_candidates" + return out + + brief = try_cs_brief(plan, mode=mode, env_file=env_file, nmi_bin=nmi_bin, endpoint=endpoint) + out["brief"] = brief + out["ok"] = bool(brief.get("ok")) + if not out["ok"]: + # still try resolve for structured debug + out["resolve"] = try_resolve_ids( + plan, mode=mode, env_file=env_file, nmi_bin=nmi_bin, endpoint=endpoint + ) + out["error"] = "nmi_lookup_failed" + # Pull item history (new getItemHistory) when requested and item_no present + if with_history: + out = enrich_with_history(out, mode=mode, env_file=env_file, nmi_bin=nmi_bin, endpoint=endpoint) + return out + + +def format_private_note(payload: dict[str, Any]) -> str: + """Build a Redmine private note body from nmi brief result. + + Enhanced with: + - Deep NMI links (Item Manager, Box History, Request pages etc.) per the WIP note + - Item history from the new getItemHistory endpoint (billings, who moved, condition changes) + """ + lines = [ + "=== Auto NMI pre-fetch (agent) ===", + f"issue_id: {payload.get('issue_id')}", + ] + plan = payload.get("plan") or {} + lines.append( + "candidates: " + + json.dumps( + { + "tracking": plan.get("tracking"), + "order_id": plan.get("order_id"), + "item_no": plan.get("item_no"), + "account": plan.get("account"), + }, + sort_keys=True, + ) + ) + + # Deep links from the WIP note + ents = payload.get("entities") or [] + links = generate_nmi_links(ents) + if links: + lines.append("") + lines.append("=== NMI Deep Links (Item Manager / Box History / Request pages / userInfo) ===") + lines.extend(links[:12]) + + brief = payload.get("brief") or {} + if brief.get("ok") and brief.get("text"): + lines.append("") + lines.append(brief["text"].rstrip()) + else: + lines.append("") + lines.append("NMI lookup did not resolve cleanly.") + if payload.get("error"): + lines.append(f"error: {payload['error']}") + attempts = brief.get("attempts") or (payload.get("resolve") or {}).get("attempts") or [] + for a in attempts[:5]: + lines.append(f"- attempt {a.get('args')}: rc={a.get('returncode')} err={a.get('stderr') or a.get('error')}") + + # New history + hist = payload.get("item_history") or {} + if hist.get("ok"): + lines.append("") + lines.append("=== Item History (getItemHistory) ===") + h = hist.get("history") or {} + if isinstance(h, dict) and h: + lines.append(json.dumps(h, indent=2, default=str)[:1600]) + elif hist.get("text"): + lines.append(hist["text"][:1400]) + else: + lines.append("(see raw for full history)") + elif "item_history" in payload: + lines.append("") + lines.append("Item history note: " + str((payload.get("item_history") or {}).get("error", "no data"))) + + lines.append("") + lines.append("_Generated by redmine_ticket_nmi; verify before customer-facing use._") + return "\n".join(lines) + "\n" + +def generate_nmi_links(entities: list[dict[str, str]]) -> list[str]: + """Produce a list of useful deep links based on resolved entities.""" + links = [] + seen = set() + for e in entities: + kind = e.get("type") + val = e.get("value") + if not kind or not val: + continue + templates = NMI_LINKS.get(kind, {}) + for label, tmpl in templates.items(): + url = tmpl(val) + key = (kind, val, label) + if key in seen: + continue + seen.add(key) + links.append(f"- {label} for {kind} {val}: {url}") + return links + + +def try_item_history( + item_no: str, + *, + mode: str = DEFAULT_MODE, + env_file: str = DEFAULT_ENV_FILE, + nmi_bin: str = DEFAULT_NMI_BIN, + endpoint: Optional[str] = None, + limit: int = 50, + snapshot: bool = True, +) -> dict[str, Any]: + """Call nmi item history for a single item_no. Returns structured result.""" + cmd = [ + "php", + nmi_bin, + "item", + "history", + f"--item-no={item_no}", + f"--limit={limit}", + ] + if snapshot: + cmd.append("--snapshot") + if mode: + cmd.extend(["--mode", mode]) + if env_file: + cmd.extend(["--env-file", env_file]) + if endpoint: + cmd.extend(["--endpoint", endpoint]) + + try: + res = subprocess.run(cmd, capture_output=True, text=True, timeout=30) + if res.returncode != 0: + return {"ok": False, "error": res.stderr.strip() or res.stdout.strip(), "cmd": cmd} + # Try JSON first + try: + data = json.loads(res.stdout) + return {"ok": True, "history": data, "raw": res.stdout} + except Exception: + return {"ok": True, "text": res.stdout.strip(), "raw": res.stdout} + except Exception as exc: + return {"ok": False, "error": str(exc)} + + +def enrich_with_history( + payload: dict[str, Any], + *, + mode: str = DEFAULT_MODE, + env_file: str = DEFAULT_ENV_FILE, + nmi_bin: str = DEFAULT_NMI_BIN, + endpoint: Optional[str] = None, +) -> dict[str, Any]: + """If an item_no is present in the plan, fetch its history and attach.""" + plan = payload.get("plan") or {} + item_no = plan.get("item_no") + if not item_no: + return payload + hist = try_item_history( + str(item_no), + mode=mode, + env_file=env_file, + nmi_bin=nmi_bin, + endpoint=endpoint, + limit=60, + snapshot=True, + ) + payload["item_history"] = hist + return payload + +def try_auto_link_redmine_contact( + *, + contact_id: int, + tracking: list[str] | None = None, + order_id: list[str] | None = None, + item_no: list[str] | None = None, + mode: str = DEFAULT_MODE, + env_file: str = DEFAULT_ENV_FILE, + nmi_bin: str = DEFAULT_NMI_BIN, +) -> dict[str, Any]: + """If we have a Redmine contact_id and identifiers from a ticket, try to resolve in NMI + and automatically create the redmine_contact_links entry. + + This implements the pattern: ticket mentions tracking/order/item -> successful + cs-info/resolve -> the returned customer owns the shipment -> link the Redmine + contact to that NMI user_account using the new junction table. + """ + if not contact_id: + return {"ok": False, "reason": "no_contact_id"} + + ids: list[str] = [] + if tracking: + for t in tracking: + ids.extend(["--tracking", str(t)]) + if order_id: + for o in order_id: + ids.extend(["--order-id", str(o)]) + if item_no: + for i in item_no: + ids.extend(["--item-no", str(i)]) + + if not ids: + return {"ok": False, "reason": "no_identifiers"} + + cmd = [nmi_bin, "link-contact", "--redmine-contact-id", str(contact_id), "--mode", mode] + if env_file and env_file != DEFAULT_ENV_FILE: + cmd.extend(["--env-file", env_file]) + cmd.extend(ids) + + try: + proc = subprocess.run(cmd, capture_output=True, text=True, timeout=90) + ok = proc.returncode == 0 + result = { + "ok": ok, + "contact_id": contact_id, + "cmd": " ".join(cmd), + "stdout": (proc.stdout or "")[:2500], + "stderr": (proc.stderr or "")[:800], + } + if ok and proc.stdout: + try: + if proc.stdout.strip().startswith("{"): + result["parsed"] = json.loads(proc.stdout) + except Exception: + pass + return result + except Exception as e: + return {"ok": False, "contact_id": contact_id, "error": str(e)} diff --git a/redmine_ticket_ocr.py b/redmine_ticket_ocr.py new file mode 100644 index 0000000..5be30f2 --- /dev/null +++ b/redmine_ticket_ocr.py @@ -0,0 +1,239 @@ +#!/usr/bin/env python3 +"""OCR / text extract helpers for scanned ticket attachments. + +Primary path: pdftotext (fast, works on digital PDFs). +Fallback: pdftoppm + tesseract for image-heavy scans. +Also handles common image attachments via tesseract. + +Does not write to Redmine — callers queue subject/notes drafts. +""" + +from __future__ import annotations + +import re +import shutil +import subprocess +import tempfile +from pathlib import Path +from typing import Any + + +def _run(cmd: list[str], *, timeout: int = 120) -> subprocess.CompletedProcess[str]: + return subprocess.run( + cmd, + capture_output=True, + text=True, + timeout=timeout, + check=False, + ) + + +def extract_text_from_pdf(path: Path, *, max_chars: int = 12000) -> tuple[str, str]: + """Return (text, method). method is pdftotext|tesseract|empty.""" + path = Path(path) + text = "" + pdftotext = shutil.which("pdftotext") + if pdftotext: + proc = _run([pdftotext, "-layout", "-nopgbrk", str(path), "-"], timeout=60) + text = (proc.stdout or "").strip() + if len(re.sub(r"\s+", "", text)) >= 40: + return text[:max_chars], "pdftotext" + + # OCR fallback: first 2 pages as PNG + pdftoppm = shutil.which("pdftoppm") + tesseract = shutil.which("tesseract") + if not pdftoppm or not tesseract: + return text[:max_chars] if text else "", "empty" if not text else "pdftotext-thin" + + with tempfile.TemporaryDirectory(prefix="rtl-ocr-") as td: + out_prefix = Path(td) / "page" + _run([pdftoppm, "-png", "-f", "1", "-l", "2", "-r", "200", str(path), str(out_prefix)], timeout=90) + chunks: list[str] = [] + for png in sorted(Path(td).glob("page*.png")): + proc = _run([tesseract, str(png), "stdout", "-l", "eng"], timeout=90) + if proc.stdout: + chunks.append(proc.stdout.strip()) + ocr_text = "\n\n".join(chunks).strip() + if ocr_text: + return ocr_text[:max_chars], "tesseract" + return text[:max_chars] if text else "", "empty" + + +def extract_text_from_image(path: Path, *, max_chars: int = 12000) -> tuple[str, str]: + tesseract = shutil.which("tesseract") + if not tesseract: + return "", "empty" + proc = _run([tesseract, str(path), "stdout", "-l", "eng"], timeout=90) + text = (proc.stdout or "").strip() + return (text[:max_chars], "tesseract") if text else ("", "empty") + + +def extract_text_from_file(path: Path, *, max_chars: int = 12000) -> tuple[str, str]: + path = Path(path) + suffix = path.suffix.lower() + if suffix == ".pdf": + return extract_text_from_pdf(path, max_chars=max_chars) + if suffix in {".png", ".jpg", ".jpeg", ".tif", ".tiff", ".webp", ".gif"}: + return extract_text_from_image(path, max_chars=max_chars) + if suffix in {".txt", ".md", ".csv"}: + try: + return path.read_text(encoding="utf-8", errors="replace")[:max_chars], "read" + except OSError: + return "", "empty" + return "", "empty" + + +_PERSON_LINE = re.compile( + r"^(?:name|employee|applicant|patient|to|for)\s*[:\-]\s*(.+)$", + re.I | re.M, +) +_COMPANY_LINE = re.compile( + r"^(?:company|organization|employer|vendor|client)\s*[:\-]\s*(.+)$", + re.I | re.M, +) +_DATE_LINE = re.compile( + r"\b((?:jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)[a-z]*\.?\s+\d{1,2},?\s+\d{4}|\d{1,2}[/-]\d{1,2}[/-]\d{2,4})\b", + re.I, +) +_DOC_HINTS = ( + (r"\bw-?2\b", "W-2"), + (r"\bw-?4\b", "W-4"), + (r"\bi-?9\b", "I-9"), + (r"\bdriver.?s?\s+license\b|\bdl\b", "Driver License"), + (r"\bpassport\b", "Passport"), + (r"\binvoice\b", "Invoice"), + (r"\bpacking\s+slip\b", "Packing Slip"), + (r"\bbill\s+of\s+lading\b|\bbol\b", "Bill of Lading"), + (r"\bresume\b|\bcv\b", "Resume"), + (r"\boffer\s+letter\b", "Offer Letter"), + (r"\bnda\b|non-?disclosure", "NDA"), + (r"\bhandbook\b", "Handbook"), + (r"\bdirect\s+deposit\b", "Direct Deposit Form"), +) + + +def suggest_subject_from_text( + text: str, + *, + original_subject: str = "", + max_len: int = 120, +) -> str: + """Heuristic subject for scanned-doc tickets.""" + clean = re.sub(r"[ \t]+", " ", text or "") + lines = [ln.strip() for ln in clean.splitlines() if ln.strip()] + # Drop very short / noise lines + content_lines = [ln for ln in lines if len(ln) >= 4 and not re.fullmatch(r"[\d\W]+", ln)][:40] + + doc_type = "Scanned document" + joined = "\n".join(content_lines[:30]) + for pat, label in _DOC_HINTS: + if re.search(pat, joined, re.I): + doc_type = label + break + + person = None + m = _PERSON_LINE.search(joined) + if m: + person = re.sub(r"\s+", " ", m.group(1)).strip(" .,-")[:60] + company = None + m = _COMPANY_LINE.search(joined) + if m: + company = re.sub(r"\s+", " ", m.group(1)).strip(" .,-")[:60] + if not company: + for ln in content_lines[:25]: + if re.search(r"\b(disposal|inc\.?|llc|ltd|corp\.?|co\.)\b", ln, re.I) and 8 <= len(ln) <= 60: + if re.search(r"please|payable|www\.|http|account number|invoice number", ln, re.I): + continue + company = re.sub(r"\s+", " ", ln).strip(" .,-")[:60] + break + + inv_m = re.search(r"\binvoice\s*(?:number|#)?\s*[:.]?\s*(\d{5,})", joined, re.I) + invoice_no = inv_m.group(1) if inv_m else None + if doc_type == "Invoice" and invoice_no: + doc_type = f"Invoice {invoice_no}" + + # Fallback: first substantial title-ish line (skip bullets / fee-table noise) + title_line = None + for ln in content_lines[:8]: + if len(ln) < 8 or len(ln) > 80: + continue + if ln.lstrip().startswith(("-", "•", "*")): + continue + if re.search(r"https?://|@|page\s+\d|return check fee|service charge", ln, re.I): + continue + title_line = ln + break + + date = None + dm = _DATE_LINE.search(joined) + if dm: + date = dm.group(1) + + parts: list[str] = [doc_type] + if person: + parts.append(f"— {person}") + elif company: + parts.append(f"— {company}") + elif title_line and title_line.lower() not in doc_type.lower(): + parts.append(f"— {title_line[:50]}") + if date and date not in " ".join(parts): + parts.append(f"({date})") + + subject = " ".join(parts) + subject = re.sub(r"\s+", " ", subject).strip() + if len(subject) > max_len: + subject = subject[: max_len - 1].rstrip() + "…" + if not subject or subject == "Scanned document": + # last resort: original if not generic + orig = (original_subject or "").strip() + if orig and not re.search(r"scanned\s+document", orig, re.I): + return orig[:max_len] + if title_line: + return f"Scanned document — {title_line[:80]}"[:max_len] + return subject + + +def attachment_is_ocr_candidate(att: dict[str, Any]) -> bool: + name = str(att.get("filename") or att.get("name") or "").lower() + ctype = str(att.get("content_type") or "").lower() + if name.endswith(".pdf") or "pdf" in ctype: + return True + if any(name.endswith(ext) for ext in (".png", ".jpg", ".jpeg", ".tif", ".tiff", ".webp")): + return True + if ctype.startswith("image/"): + return True + return False + + +def looks_like_generic_scan_subject(subject: str) -> bool: + s = (subject or "").strip().lower() + if not s: + return True + return bool( + re.search( + r"scanned\s+document|scan\s+from|hp\s+scan|scanner|untitled|document\s+from\s+scan", + s, + ) + ) + + +def build_ocr_note( + *, + issue_id: int, + files: list[dict[str, str]], + proposed_subject: str, +) -> str: + lines = [ + "=== OCR / attachment review (agent) ===", + f"issue_id: {issue_id}", + f"proposed_subject: {proposed_subject}", + "", + ] + for f in files: + lines.append(f"## {f.get('filename')} ({f.get('method')})") + lines.append("") + snippet = (f.get("text") or "")[:2500].strip() + lines.append(snippet or "_(no extractable text)_") + lines.append("") + lines.append("_Generated by redmine_ticket_ocr; verify before customer-facing use._") + return "\n".join(lines) + "\n" diff --git a/redmine_ticket_query.py b/redmine_ticket_query.py new file mode 100644 index 0000000..20bfb98 --- /dev/null +++ b/redmine_ticket_query.py @@ -0,0 +1,365 @@ +#!/usr/bin/env python3 +"""Local property-based query helpers for cached Redmine ticket JSON. + +Small expression language (intentionally not a full DSL): + + path == value + path != value + path IS NULL / path IS NOT NULL + path.includes("substring") + path.since("20 minutes") # relative window for date-ish fields + path == other.path + expr AND expr + expr OR expr + ( expr ) + +Paths use dot notation into issue JSON, e.g.: + subject + status.name + helpdesk_ticket.contact.email + assigned_to.name + +When a dict is compared to a string, its ``name`` field is used if present. +""" + +from __future__ import annotations + +import re +from datetime import datetime, timedelta, timezone +from typing import Any, Callable + + +class QueryError(ValueError): + """Invalid property query expression.""" + + +_TOKEN_RE = re.compile( + r""" + \s* + ( + (?:AND|OR|IS|NOT|NULL)\b + | && + | \|\| + | \. + | \( + | \) + | == + | != + | "[^"\\]*(?:\\.[^"\\]*)*" + | '[^'\\]*(?:\\.[^'\\]*)*' + | [A-Za-z_][A-Za-z0-9_]* + | \d+(?:\.\d+)? + ) + """, + re.VERBOSE | re.IGNORECASE, +) + +_RELATIVE_RE = re.compile( + r"^(?P\d+)\s*(?Pd|day|days|h|hour|hours|m|min|mins|minute|minutes|w|week|weeks)\s*(?:ago)?$", + re.IGNORECASE, +) + + +def tokenize(expression: str) -> list[str]: + text = expression.strip() + if not text: + raise QueryError("Empty query expression.") + tokens: list[str] = [] + pos = 0 + while pos < len(text): + match = _TOKEN_RE.match(text, pos) + if not match: + raise QueryError(f"Unexpected input near: {text[pos:pos + 40]!r}") + token = match.group(1) + tokens.append(token) + pos = match.end() + if pos != len(text): + raise QueryError(f"Unparsed trailing input: {text[pos:]!r}") + return tokens + + +def _unquote(token: str) -> str: + if len(token) >= 2 and token[0] == token[-1] and token[0] in "\"'": + body = token[1:-1] + return bytes(body, "utf-8").decode("unicode_escape") if "\\" in body else body + return token + + +def parse_relative_delta(value: str) -> timedelta: + match = _RELATIVE_RE.fullmatch(value.strip()) + if not match: + raise QueryError( + f"Invalid relative time {value!r}; use forms like '20 minutes', '2h', '5 days'." + ) + amount = int(match.group("n")) + unit = match.group("unit").lower() + if unit.startswith("w"): + return timedelta(weeks=amount) + if unit.startswith("d"): + return timedelta(days=amount) + if unit.startswith("h"): + return timedelta(hours=amount) + return timedelta(minutes=amount) + + +def parse_dt_loose(value: Any) -> datetime | None: + if value is None: + return None + if isinstance(value, datetime): + dt = value + else: + text = str(value).strip() + if not text: + return None + if text.endswith("Z"): + text = text[:-1] + "+00:00" + if "T" not in text and len(text) == 10: + text = text + "T00:00:00+00:00" + try: + dt = datetime.fromisoformat(text) + except ValueError: + return None + if dt.tzinfo is None: + dt = dt.replace(tzinfo=timezone.utc) + return dt + + +def get_path(issue: dict[str, Any], path: str) -> Any: + current: Any = issue + for part in path.split("."): + if current is None: + return None + if isinstance(current, dict): + current = current.get(part) + continue + return None + return current + + +def display_value(value: Any) -> Any: + """Coerce nested Redmine objects to a comparable scalar when useful.""" + if isinstance(value, dict): + if value.get("name") is not None: + return value.get("name") + if value.get("email") is not None: + return value.get("email") + if value.get("id") is not None: + return value.get("id") + return value + + +def values_equal(left: Any, right: Any) -> bool: + left_d = display_value(left) + right_d = display_value(right) + if left_d is None and right_d is None: + return True + if left_d is None or right_d is None: + return False + if isinstance(left_d, (int, float)) and isinstance(right_d, (int, float)): + return left_d == right_d + return str(left_d).lower() == str(right_d).lower() + + +def value_includes(haystack: Any, needle: str) -> bool: + if haystack is None: + return False + text = str(display_value(haystack)) + return needle.lower() in text.lower() + + +class _Parser: + def __init__(self, tokens: list[str]): + self.tokens = tokens + self.i = 0 + + def _peek(self) -> str | None: + if self.i >= len(self.tokens): + return None + return self.tokens[self.i] + + def _advance(self) -> str: + token = self._peek() + if token is None: + raise QueryError("Unexpected end of expression.") + self.i += 1 + return token + + def _accept_keyword(self, *keywords: str) -> str | None: + token = self._peek() + if token is None: + return None + upper = token.upper() + for keyword in keywords: + if upper == keyword.upper(): + self.i += 1 + return token + return None + + def parse(self) -> Callable[[dict[str, Any]], bool]: + node = self._parse_or() + if self._peek() is not None: + raise QueryError(f"Unexpected token {self._peek()!r}") + return node + + def _parse_or(self) -> Callable[[dict[str, Any]], bool]: + left = self._parse_and() + while True: + if self._accept_keyword("OR") or self._peek() == "||": + if self._peek() == "||": + self._advance() + right = self._parse_and() + prev_left, prev_right = left, right + left = lambda issue, a=prev_left, b=prev_right: a(issue) or b(issue) + continue + break + return left + + def _parse_and(self) -> Callable[[dict[str, Any]], bool]: + left = self._parse_primary() + while True: + if self._accept_keyword("AND") or self._peek() == "&&": + if self._peek() == "&&": + self._advance() + right = self._parse_primary() + prev_left, prev_right = left, right + left = lambda issue, a=prev_left, b=prev_right: a(issue) and b(issue) + continue + break + return left + + def _parse_primary(self) -> Callable[[dict[str, Any]], bool]: + if self._peek() == "(": + self._advance() + node = self._parse_or() + if self._peek() != ")": + raise QueryError("Expected ')'") + self._advance() + return node + return self._parse_comparison() + + def _parse_path(self) -> str: + parts: list[str] = [] + token = self._advance() + if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", token): + raise QueryError(f"Expected path starting identifier, got {token!r}") + parts.append(token) + while self._peek() == ".": + # Lookahead: path.includes( / path.since( → leave method name for comparison parser + # tokens: ['.', 'includes', '('] + if self.i + 2 < len(self.tokens): + method_name = self.tokens[self.i + 1] + after = self.tokens[self.i + 2] + if method_name in {"includes", "since"} and after == "(": + break + self._advance() # consume '.' + nxt = self._peek() + if nxt is None: + raise QueryError("Dangling '.' in path") + if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", nxt): + raise QueryError(f"Invalid path segment {nxt!r}") + parts.append(self._advance()) + return ".".join(parts) + + def _parse_value_or_path(self) -> tuple[str, Any]: + """Return ('path', path_str) or ('lit', literal).""" + token = self._peek() + if token is None: + raise QueryError("Expected value or path") + if token[0] in "\"'" or re.fullmatch(r"\d+(?:\.\d+)?", token): + self._advance() + if token[0] in "\"'": + return "lit", _unquote(token) + if "." in token: + return "lit", float(token) + return "lit", int(token) + path = self._parse_path() + return "path", path + + def _parse_comparison(self) -> Callable[[dict[str, Any]], bool]: + left_path = self._parse_path() + + # method form: path.includes("x") / path.since("20 minutes") + method = None + base_path = left_path + if self._peek() == ".": + # path was stopped before method name + self._advance() + method = self._advance() + if method not in {"includes", "since"}: + raise QueryError(f"Unknown method .{method}; supported: includes, since") + base_path = left_path + + if method is not None: + if self._advance() != "(": + raise QueryError(f"Expected '(' after .{method}") + arg_tok = self._advance() + if not arg_tok or arg_tok[0] not in "\"'": + raise QueryError(f".{method}() argument must be a string literal") + arg = _unquote(arg_tok) + if self._advance() != ")": + raise QueryError(f"Expected ')' after .{method} argument") + if method == "includes": + return lambda issue, p=base_path, a=arg: value_includes(get_path(issue, p), a) + delta = parse_relative_delta(arg) + + def since_pred(issue: dict[str, Any], p=base_path, d=delta) -> bool: + raw = get_path(issue, p) + dt = parse_dt_loose(raw) + if dt is None: + return False + now = datetime.now(timezone.utc) + return dt >= (now - d) + + return since_pred + + # IS NULL / IS NOT NULL + if self._accept_keyword("IS"): + not_null = bool(self._accept_keyword("NOT")) + if not self._accept_keyword("NULL"): + raise QueryError("Expected NULL after IS [NOT]") + if not_null: + return lambda issue, p=left_path: get_path(issue, p) is not None and display_value(get_path(issue, p)) not in (None, "") + return lambda issue, p=left_path: get_path(issue, p) is None or display_value(get_path(issue, p)) in (None, "") + + op = self._peek() + if op not in {"==", "!="}: + raise QueryError(f"Expected comparison operator after path {left_path!r}, got {op!r}") + self._advance() + kind, right = self._parse_value_or_path() + if kind == "lit": + if op == "==": + return lambda issue, p=left_path, r=right: values_equal(get_path(issue, p), r) + return lambda issue, p=left_path, r=right: not values_equal(get_path(issue, p), r) + # path == path + if op == "==": + return lambda issue, a=left_path, b=right: values_equal(get_path(issue, a), get_path(issue, b)) + return lambda issue, a=left_path, b=right: not values_equal(get_path(issue, a), get_path(issue, b)) + + +def compile_query(expression: str) -> Callable[[dict[str, Any]], bool]: + tokens = tokenize(expression) + # Normalize parentheses tokens already fine + return _Parser(tokens).parse() + + +def issue_matches(issue: dict[str, Any], expression: str) -> bool: + return compile_query(expression)(issue) + + +def filter_by_query( + issues: list[dict[str, Any]], + expression: str | None = None, + expressions: list[str] | None = None, +) -> list[dict[str, Any]]: + preds: list[Callable[[dict[str, Any]], bool]] = [] + if expression: + preds.append(compile_query(expression)) + for expr in expressions or []: + preds.append(compile_query(expr)) + if not preds: + return list(issues) + out: list[dict[str, Any]] = [] + for issue in issues: + if all(pred(issue) for pred in preds): + out.append(issue) + return out diff --git a/redmine_ticket_textual_tui.py b/redmine_ticket_textual_tui.py new file mode 100644 index 0000000..c38a65e --- /dev/null +++ b/redmine_ticket_textual_tui.py @@ -0,0 +1,1190 @@ +#!/usr/bin/env python3 +""" +Textual Redmine power TUI — list-first design per user spec. + +Default view: big dense list (like curses + screenshot). +Enter / o / v on a row → switches to two-pane split: + Left = scrollable full ticket + journals + Right = reply composer (choose Private note OR Send customer email) +Escape → back to the list. +""" + +from __future__ import annotations +import sys +import os +import argparse +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Optional + +from rich.text import Text + +from textual.app import App, ComposeResult +from textual.events import Key +from textual.binding import Binding +from textual.containers import Horizontal, Vertical, ScrollableContainer +from textual.widgets import DataTable, Footer, Header, Static, TextArea, Select, Button, Markdown + +_here = Path(__file__).resolve().parent +if str(_here) not in sys.path: + sys.path.insert(0, str(_here)) + +def _get_core(): + try: + from redmine_tickets import ( + TicketCache as _TC, + select_cached_issues_for_cli as _sel, + refresh_cached_issue as _ref, + client_from_args as _cf, + ) + return _TC, _sel, _ref, _cf + except Exception as e: + raise RuntimeError(f"Core import failed: {e}") from e + +try: + from redmine_ticket_tui import PendingChanges, Template +except Exception: + from dataclasses import dataclass, field + from typing import Any, Optional + + @dataclass + class PendingChanges: + issue_id: int + fields: dict[str, Any] = field(default_factory=dict) + private_note: Optional[str] = None + private: bool = True + control_tags: list[str] = field(default_factory=list) + contact_tags: list[str] = field(default_factory=list) + entity_extract: bool = False + nmi_brief: bool = False + contact_id: Optional[int] = None + + @dataclass + class Template: + status_id: Optional[int] = None + status_name: Optional[str] = None + assigned_to_id: Optional[int] = None + assigned_to_name: Optional[str] = None + category_id: Optional[int] = None + category_name: Optional[str] = None + priority_id: Optional[int] = None + priority_name: Optional[str] = None + project_id: Optional[int] = None + project_name: Optional[str] = None + local_control_tag: Optional[str] = None + contact_tags: list[str] = field(default_factory=list) + private_note: Optional[str] = None + entity_extract: bool = False + nmi_brief: bool = False + +# Painting parity helpers (entity preview + NMI brief + drafting) +try: + from redmine_ticket_entities import extract_entities_from_issue, format_entities_line +except Exception: + extract_entities_from_issue = None + format_entities_line = None + +try: + from redmine_ticket_nmi import brief_for_issue, format_private_note +except Exception: + brief_for_issue = None + format_private_note = None + +try: + from redmine_ticket_drafts import ( + add_draft, + KIND_CONTACT_FIELDS, + KIND_INTERNAL_NOTE, + KIND_FIELDS, + assert_write_target_allowed, + load_draft, + mark_draft_applied, + mark_draft_error, + ) +except Exception: + add_draft = None + KIND_INTERNAL_NOTE = "internal_note" + KIND_CONTACT_FIELDS = "contact_fields" + KIND_FIELDS = "fields" + assert_write_target_allowed = None + load_draft = None + mark_draft_applied = None + mark_draft_error = None + +try: + from redmine_ticket_control_tags import add_tag as control_tag_add +except Exception: + control_tag_add = None + +try: + from redmine_tickets import apply_draft_to_redmine, TicketCache, refresh_cached_issue, load_cached_issue +except Exception: + apply_draft_to_redmine = None + TicketCache = None + refresh_cached_issue = None + load_cached_issue = None + + +def extract_contact_id(issue: dict[str, Any]) -> Optional[int]: + """Best-effort contact id extraction, matching the curses client.""" + if not issue: + return None + ht = issue.get("helpdesk_ticket") or {} + if isinstance(ht, dict): + c = ht.get("contact") or {} + if isinstance(c, dict): + cid = c.get("id") + if cid: + try: + return int(cid) + except Exception: + pass + for k in ("contact_id", "helpdesk_contact_id"): + if k in issue and issue[k]: + try: + return int(issue[k]) + except Exception: + pass + return None + +def icon_for_contact_tag(t: str) -> str: + t = (t or "").lower() + if "spam" in t: return "󰏮" + if "lead" in t: return "󰈸" + return "󰀀" + +def icon_for_control_tag(t: str) -> str: + return "󰓒" + +def status_icon(s: str) -> str: + s = (s or "").lower() + if "new" in s: return "󰐊" + if "progress" in s: return "󰦕" + if "feedback" in s: return "󰍉" + if "closed" in s: return "󰄬" + return "󰇘" + +COLUMNS = [ + ("#", "id", 7), ("Status", "status", 12), ("Pri", "priority", 4), + ("Assignee", "assignee", 12), ("Category", "category", 11), + ("Subject", "subject", 38), ("Updated", "updated", 14), + ("Contact", "contact", 16), ("Tags", "tags", 14), +] + +class RedmineTextualTUI(App): + """List is the main view. Enter opens a temporary two-pane detail view.""" + + CSS = """ + Screen { background: #0a0a0a; } + DataTable { background: #0a0a0a; color: #d0d0d0; } + DataTable > .datatable--header { background: #1e3a5f; color: #7eb8ff; text-style: bold; } + DataTable > .datatable--cursor { background: #1a3a5f; color: #fff; } + #detail-left { width: 1fr; border-right: thick #222; } + #detail-right { width: 1fr; padding: 0 1; } + #status-bar { background: #111; color: #888; padding: 0 1; } + """ + + BINDINGS = [ + Binding("q", "quit", "Quit"), + Binding("enter", "open_selected", "Open (read+reply)"), + Binding("o", "open_selected", "Open"), + Binding("v", "open_selected", "View"), + Binding("space", "paint", "Paint template"), + Binding("s", "pick_status", "Status"), + Binding("a", "pick_assignee", "Assignee"), + Binding("c", "pick_category", "Category"), + Binding("p", "set_priority", "Priority"), + Binding("1", "quick_priority_1", "Pri 1"), + Binding("2", "quick_priority_2", "Pri 2"), + Binding("3", "quick_priority_3", "Pri 3"), + Binding("n", "set_note", "Note"), + Binding("e", "toggle_entity", "Entity"), + Binding("m", "toggle_nmi", "NMI"), + Binding("t", "set_control_tag", "Tag"), + Binding("C", "add_contact_tag", "CTag"), + Binding("x", "clear_pending", "Clear"), + Binding("r", "refresh_local", "Redraw"), + Binding("R", "refresh_live", "Live refresh"), + Binding("/", "filter", "Filter"), + Binding("A", "apply_batch", "Apply batch"), + Binding("?", "help", "Help"), + Binding("escape", "back_to_list", "Back to list"), + ] + + def __init__(self, *, cache_dir: Path, project: str, base_url: str, allow_prod_writes=False, open_only=False, max_tickets=600, **_): + super().__init__() + self.cache_dir = cache_dir + self.project = project + self.base_url = base_url + self.allow_prod_writes = allow_prod_writes + self.open_only = open_only + self.max_tickets = max_tickets + self.issues = [] + self.pending = {} + self.template = Template() + self.filter_text = "" + self.current_issue_id = None + self._client = None + self.in_detail_mode = False + self.list_table = None + self.detail_split = None + + def _get_client(self): + if self._client: return self._client + if not self.allow_prod_writes: return None + try: + _,_,_,cf = _get_core() + a = argparse.Namespace() + a.base_url = self.base_url; a.project = self.project + a.cache_dir = str(self.cache_dir); a.allow_prod_writes = True + a.config = a.impersonate_user = None + self._client = cf(a) + return self._client + except Exception: + return None + + # === DEFAULT VIEW: the list === + def compose(self) -> ComposeResult: + yield Header(show_clock=True) + yield Static(f"Redmine Power TUI | {self.project} | {self.base_url}", id="header-line") + self.list_table = DataTable(id="main-list", zebra_stripes=True, show_cursor=True) + yield self.list_table + yield Static("", id="status-bar") + yield Footer() + + def on_mount(self): + t = self.list_table + t.cursor_type = "row" + for title, key, w in COLUMNS: + t.add_column(title, key=key, width=w) + self.load_issues() + self.refresh_table() + if self.issues: + t.move_cursor(row=0) + t.focus() + self._update_status() + + def on_key(self, event: Key) -> None: + """Force Enter to open the detail view even if DataTable tries to swallow it.""" + if event.key == "enter" and not getattr(self, "in_detail_mode", False): + event.prevent_default() + self.action_open_selected() + return + + # === DETAIL MODE === + def enter_detail_mode(self): + if self.in_detail_mode or not self.issues: return + t = self.list_table + if t.cursor_row is None: return + try: + iid = int(t.get_row_at(t.cursor_row)[0]) + except Exception: + return + + self.current_issue_id = iid + self.in_detail_mode = True + t.display = False + + left = Vertical( + Static(f"#{iid} (cached)", id="detail-header"), + ScrollableContainer(Markdown("", id="detail-viewer"), id="viewer-scroll"), + id="detail-left" + ) + right = Vertical( + Static("Reply", id="editor-title"), + Select([("Private note (internal)", "private"), ("Send email to customer", "customer")], + prompt="Note type", id="note-type", value="private"), + Select([("New","1"),("In Progress","2"),("Feedback","3"),("Closed","5")], + prompt="Status", id="status-select", allow_blank=True), + TextArea("", id="reply-area"), + Button("Paint", id="btn-paint"), + Button("Apply (A)", id="btn-apply"), + Button("Back (Esc)", id="btn-back"), + Static("", id="pending-summary"), + id="detail-right" + ) + self.detail_split = Horizontal(left, right, id="detail-split") + self.mount(self.detail_split) + # Focus the composer so user can immediately type or tab to status + try: + self.query_one("#reply-area", TextArea).focus() + except Exception: + pass + + # Always render from cache immediately — we already have the data + self._render_cached_detail(iid) + base_issue = next((x for x in self.issues if int(x.get("id",0)) == iid), None) + if base_issue: + try: + self._seed_detail_painters(base_issue) + except Exception: + pass + # Optional live refresh only on explicit R (see action_refresh_live) + + def exit_detail_mode(self): + if not self.in_detail_mode: return + if self.detail_split: + self.detail_split.remove() + self.detail_split = None + self.in_detail_mode = False + self.list_table.display = True + self.list_table.focus() + self._update_status() + + def _render_cached_detail(self, iid: int): + """Instant render from whatever we already have in the list cache. + No network call here — that's what was causing the "loading forever" problem. + """ + base = next((x for x in self.issues if int(x.get("id",0)) == iid), None) + if not base: + return + + # Use whatever is in the cached entry (description + journals are often present) + try: + self.query_one("#detail-header", Static).update(f"#{iid} {base.get('subject','')}") + self.query_one("#detail-viewer", Markdown).update(self._build_markdown(base)) + except Exception: + pass + + self._refresh_pending_display(iid) + + + + # ------------------------------------------------------------------ + # Painting system (ported from curses client for parity) + # ------------------------------------------------------------------ + def _seed_detail_painters(self, issue: dict): + """Populate the right-pane selects from current template + meta.""" + try: + meta = self._get_meta_lists() + # Status + try: + st_sel = self.query_one("#status-select", Select) + statuses = meta.get("statuses", []) + if statuses: + st_sel.set_options([(st.get("name", ""), str(st.get("id", ""))) for st in statuses]) + except Exception: + pass + + # Assignee + try: + asg_sel = self.query_one("#assignee-select", Select) + users = meta.get("users", []) + if users: + asg_sel.set_options([(u.get("name", "") or u.get("login", ""), str(u.get("id", ""))) for u in users]) + except Exception: + pass + + # Category + try: + cat_sel = self.query_one("#category-select", Select) + cats = meta.get("categories", []) + if cats: + cat_sel.set_options([(c.get("name", ""), str(c.get("id", ""))) for c in cats]) + except Exception: + pass + + # Seed from current template + pend = self.pending.get(self.current_issue_id) if self.current_issue_id else None + if self.template.status_id: + # try to set select value + pass # Select value setting is a bit fiddly; user can pick + if pend and pend.private_note: + try: + self.query_one("#reply-area", TextArea).text = pend.private_note + except Exception: + pass + except Exception: + pass + + + def _get_meta_lists(self): + """Return meta lists (categories, statuses, users, etc.).""" + if hasattr(self, "_meta_cache") and self._meta_cache: + return self._meta_cache + try: + from redmine_ticket_meta import load_meta_lists + self._meta_cache = load_meta_lists(self.cache_dir, self.project) or {} + return self._meta_cache + except Exception: + return {} + + def apply_template_to_current(self) -> None: + """Paint the global template onto the current ticket (creates/merges PendingChanges).""" + if not self.current_issue_id: + # Try to use highlighted in list + if self.list_table and self.list_table.cursor_row is not None: + try: + row = self.list_table.get_row_at(self.list_table.cursor_row) + self.current_issue_id = int(row[0]) + except Exception: + return + else: + return + + iid = self.current_issue_id + issue = next((x for x in self.issues if int(x.get("id", 0)) == iid), None) + if not issue: + return + + contact_id = extract_contact_id(issue) + + fields: dict[str, Any] = {} + if self.template.status_id is not None: + fields["status_id"] = self.template.status_id + if self.template.assigned_to_id is not None: + fields["assigned_to_id"] = None if self.template.assigned_to_id == 0 else self.template.assigned_to_id + if self.template.priority_id is not None: + fields["priority_id"] = self.template.priority_id + if self.template.category_id is not None: + fields["category_id"] = None if self.template.category_id == 0 else self.template.category_id + if self.template.project_id is not None: + fields["project_id"] = self.template.project_id + + ch = self.pending.get(iid) or PendingChanges(issue_id=iid, contact_id=contact_id) + + if fields: + ch.fields = {** (ch.fields or {}), **fields} + + if self.template.private_note: + ch.private_note = self.template.private_note + ch.private = True + + if self.template.local_control_tag: + if self.template.local_control_tag not in (ch.control_tags or []): + ch.control_tags = (ch.control_tags or []) + [self.template.local_control_tag] + + if self.template.contact_tags: + for t in self.template.contact_tags: + if t not in (ch.contact_tags or []): + ch.contact_tags = (ch.contact_tags or []) + [t] + + ch.entity_extract = ch.entity_extract or self.template.entity_extract + ch.nmi_brief = ch.nmi_brief or self.template.nmi_brief + ch.contact_id = contact_id or ch.contact_id + + self.pending[iid] = ch + + if self.in_detail_mode: + self._refresh_pending_display(iid) + self.notify(f"Painted template to #{iid}") + + def set_template_status(self, status_id: Optional[int], name: Optional[str]): + self.template.status_id = status_id + self.template.status_name = name + self.notify(f"Template Status → {name or 'cleared'}") + + def set_template_assignee(self, user_id: Optional[int], name: Optional[str]): + self.template.assigned_to_id = user_id + self.template.assigned_to_name = name + self.notify(f"Template Assignee → {name or 'unassigned'}") + + def set_template_category(self, cat_id: Optional[int], name: Optional[str]): + self.template.category_id = cat_id + self.template.category_name = name + self.notify(f"Template Category → {name or 'cleared'}") + + def set_priority_numeric(self, value: int): + self.template.priority_id = value + self.template.priority_name = str(value) + self.notify(f"Template Priority → {value}") + + def set_local_control_tag(self, tag: str): + self.template.local_control_tag = tag + self.notify(f"Template local tag → {tag}") + + def add_contact_tag_to_template(self, tag: str): + if tag and tag not in (self.template.contact_tags or []): + self.template.contact_tags = (self.template.contact_tags or []) + [tag] + self.notify(f"Template contact tags: {self.template.contact_tags}") + + def set_template_private_note(self, note: Optional[str]): + self.template.private_note = note or None + self.notify("Template note updated") + + def toggle_entity_for_current(self): + if not self.current_issue_id: + return + iid = self.current_issue_id + ch = self.pending.get(iid) or PendingChanges(issue_id=iid) + ch.entity_extract = not ch.entity_extract + self.pending[iid] = ch + if self.in_detail_mode: + self._refresh_pending_display(iid) + self.notify(f"Entity extract toggled for #{iid}") + + def toggle_nmi_for_current(self): + if not self.current_issue_id: + return + iid = self.current_issue_id + ch = self.pending.get(iid) or PendingChanges(issue_id=iid) + ch.nmi_brief = not ch.nmi_brief + self.pending[iid] = ch + if self.in_detail_mode: + self._refresh_pending_display(iid) + self.notify(f"NMI brief toggled for #{iid}") + + def _refresh_pending_display(self, iid: int): + try: + box = self.query_one("#pending-box", Static) + pend = self.pending.get(iid) + if pend: + box.update("Pending: " + (getattr(pend, "private_note", "") or "")[:40] + " " + str(pend.fields or {})) + else: + box.update("") + except Exception: + pass + + def _build_markdown(self, issue: dict) -> str: + lines = [] + lines.append(f"# {issue.get('subject','')}") + lines.append("") + lines.append(f"**Status:** {_safe_get(issue,'status','name','')}") + lines.append(f"**Assigned:** {_safe_get(issue,'assigned_to','name','')}") + lines.append("") + lines.append("## Description") + lines.append("") + desc = (issue.get("description") or "(no description)").replace("\r\n", "\n") + lines.append(desc) + lines.append("") + for j in (issue.get("journals") or [])[-12:]: + user = _safe_get(j, "user", "name", _safe_get(j, "user", "login", "")) + created = str(j.get("created_on",""))[:16] + notes = (j.get("notes") or "").strip() + if notes: + priv = " (private)" if j.get("private_notes") else "" + lines.append(f"**{created} — {user}{priv}**") + lines.append(notes) + lines.append("---") + return "\n".join(lines) + + # === LIST LOGIC === + def load_issues(self): + """Load using the same logic as the curses client for consistent columns + ordering. + Falls back to raw cache + explicit sort if selector unavailable. + """ + TC, select_cached_issues_for_cli, _, _ = _get_core() + cache = TC(self.cache_dir, self.project) + + raw = [] + try: + raw = list(cache.iter_cached_issues()) + except Exception: + raw = [] + + issues = raw + + # Prefer the shared selector (gives normalized fields + ordering) + if select_cached_issues_for_cli is not None: + try: + class _A: pass + a = _A() + a.where = [] + a.control_tag = None + a.status = None + a.max_tickets = self.max_tickets + a.order = "newest-updated-first" + a.open_only = self.open_only + a.has_attachments = False + a.cache_dir = str(self.cache_dir) + a.project = self.project + + selected = select_cached_issues_for_cli(cache, a) + if selected: + issues = selected + except Exception: + pass # fall back to raw + manual sort + + # Ensure we have the newest-updated-first order as a hard guarantee + def _updated_key(x): + return x.get("updated_on") or x.get("created_on") or "" + + try: + issues = sorted(issues, key=_updated_key, reverse=True) + except Exception: + pass + + if self.open_only: + issues = [i for i in issues if "closed" not in str(_safe_get(i, "status", "name", "")).lower()] + + self.issues = issues[:self.max_tickets] + + def refresh_table(self): + t = self.list_table + t.clear() + for iss in self._apply_filter(self.issues): + t.add_row(*self._make_row(iss), key=str(iss.get("id"))) + self._update_status() + + def _apply_filter(self, issues): + if not self.filter_text: return issues + q = self.filter_text.lower() + return [i for i in issues if q in str(i.get("id","")) + str(i.get("subject","")).lower()] + + def _make_row(self, issue): + iid = issue.get("id") + pend = self.pending.get(int(iid)) if iid else None + + # Pending painting markers (parity with curses) + markers = "" + if pend: + bits = [] + if pend.fields: bits.append("F") + if pend.private_note: bits.append("N") + if pend.control_tags or pend.contact_tags: bits.append("T") + if pend.entity_extract: bits.append("E") + if pend.nmi_brief: bits.append("M") + if bits: + markers = "[" + "".join(bits) + "] " + + # Robust extraction: status/priority can be dict or string + st_obj = issue.get("status") or {} + if isinstance(st_obj, dict): + st = st_obj.get("name", "") or "" + else: + st = str(st_obj) + + pri_obj = issue.get("priority") or {} + if isinstance(pri_obj, dict): + pri = pri_obj.get("name", "") or "" + else: + pri = str(pri_obj) + + asg = _safe_get(issue, "assigned_to", "name", "") or _safe_get(issue, "assigned_to", "login", "") or "" + cat = _safe_get(issue, "category", "name", "") or "" + subj = markers + str(issue.get("subject",""))[:36] + upd = str(issue.get("updated_on",""))[:14] + + contact = "" + ht = issue.get("helpdesk_ticket") or {} + if isinstance(ht, dict): + c = ht.get("contact") or {} + if isinstance(c, dict): + contact = c.get("company") or c.get("email") or "" + + tags = self._render_tags(issue) + st_d = f"{status_icon(st)} {st}" if st else st + if not tags.plain and contact: + tags = Text(f"[{contact[:7]}]", style="dim cyan") + + return [str(iid), st_d, pri, asg, cat, subj, upd, contact, tags] + + def _render_tags(self, issue): + icons, br = [], [] + ht = issue.get("helpdesk_ticket") or {} + if isinstance(ht, dict): + c = ht.get("contact") or {} + if isinstance(c, dict): + for t in (c.get("tags") or []): + nm = (t.get("name") if isinstance(t,dict) else str(t)).strip() + if nm: + icons.append(icon_for_contact_tag(nm)) + br.append(f"[{nm[:5]}]") + iid = int(issue.get("id",0)) + pend = self.pending.get(iid) + if pend: + for t in (pend.control_tags or []): + icons.append(icon_for_control_tag(t)) + br.append(f"[{t[:5]}]") + disp = " ".join(icons + br).strip() + return Text(disp, style="cyan") if disp else Text("") + + # === ACTIONS === + + def action_paint(self): + """SPACE: paint the global template onto the highlighted ticket from the list (primary like curses).""" + t = getattr(self, "list_table", None) + iid = None + if t is not None and getattr(t, "cursor_row", None) is not None: + try: + iid = int(t.get_row_at(t.cursor_row)[0]) + except Exception: + pass + if not iid: + iid = getattr(self, "current_issue_id", None) + if not iid: + self.notify("Highlight a row in the list, then press SPACE to paint") + return + self.current_issue_id = iid + self.apply_template_to_current() + self.refresh_table() + self.notify(f"Painted #{iid} (pending now has changes)") + self._update_status() + + def action_open_selected(self): + self.enter_detail_mode() + + def action_back_to_list(self): + self.exit_detail_mode() + + def action_refresh_local(self): + if self.in_detail_mode and self.current_issue_id: + self._render_cached_detail(self.current_issue_id) + else: + self.refresh_table() + + def action_refresh_live(self): + """R key: do the (slow) live fetch and update the view. + This is now explicit — opening a ticket no longer blocks on the network. + """ + if not self.current_issue_id: + return + cli = self._get_client() + if not cli: + self.notify("Live refresh requires --allow-prod-writes") + return + self.notify("Refreshing from Redmine (can be slow)...") + TC, _, ref, _ = _get_core() + try: + c = TC(self.cache_dir, self.project) + fresh = ref(c, cli, self.current_issue_id) + if fresh: + for i, x in enumerate(self.issues): + if int(x.get("id", 0)) == self.current_issue_id: + self.issues[i] = fresh + break + if self.in_detail_mode: + # Re-render from the now-updated cache entry + self._render_cached_detail(self.current_issue_id) + else: + self.refresh_table() + self.notify(f"#{self.current_issue_id} refreshed from live") + except Exception as e: + self.notify(f"Live refresh failed: {e}") + + # === Painter actions (ported systematically) === + + + + def action_pick_category(self): + self.template.category_name = "Customer Service" + self.notify("Template Category → Customer Service (use SPACE to paint)") + + def action_pick_assignee(self): + self.template.assigned_to_name = "Jason" + self.template.assigned_to_id = 1 + self.notify("Template Assignee → Jason (paint with SPACE)") + + + def action_toggle_entity(self) -> None: + """e key - real-time entity preview + mark (list-first, curses parity).""" + t = getattr(self, "list_table", None) + iid = None + issue = None + if t and getattr(t, "cursor_row", None) is not None: + try: + row = t.get_row_at(t.cursor_row) + iid = int(row[0]) + issue = next((x for x in self.issues if int(x.get("id", 0)) == iid), None) + except Exception: + pass + if not iid: + iid = getattr(self, "current_issue_id", None) + if iid: + issue = next((x for x in self.issues if int(x.get("id", 0)) == iid), None) + if not iid or not issue: + self.notify("Highlight a row then press e") + return + ch = self.pending.get(iid) or PendingChanges(issue_id=iid) + ch.entity_extract = not ch.entity_extract + self.pending[iid] = ch + if ch.entity_extract and extract_entities_from_issue: + try: + ents = extract_entities_from_issue(issue) + line = format_entities_line(ents) if format_entities_line and ents else str(ents or "(none)") + self.notify(f"Entity ON #{iid}: {line[:85]}") + except Exception as ee: + self.notify(f"Entity ON #{iid} (err: {ee})") + else: + self.notify(f"Entity {'ON' if ch.entity_extract else 'OFF'} for #{iid}") + self.refresh_table() + self._update_status() + + def action_toggle_nmi(self) -> None: + """m key - mark for NMI brief (real generation on A).""" + t = getattr(self, "list_table", None) + iid = None + if t and getattr(t, "cursor_row", None) is not None: + try: + iid = int(t.get_row_at(t.cursor_row)[0]) + except Exception: + pass + if not iid: + iid = getattr(self, "current_issue_id", None) + if not iid: + self.notify("Highlight a row then press m") + return + ch = self.pending.get(iid) or PendingChanges(issue_id=iid) + ch.nmi_brief = not ch.nmi_brief + self.pending[iid] = ch + env = os.environ.get("NMI_ENV_FILE") or "/home/jason/projects/mrscClient-php/examples/modern/.env" + self.notify(f"NMI {'ON' if ch.nmi_brief else 'OFF'} for #{iid} (real on A using {env})") + self.refresh_table() + self._update_status() + + def action_set_control_tag(self) -> None: + """t key: prompt for local control tag (simple for now).""" + from textual.widgets import Input + self.notify("Type tag then press Enter in the input (or use detail pane). For quick: set via template in practice.") + # Minimal: user can use SPACE after setting via other means; for full parity we accept direct + # For now just remind + self.notify("Use t + type in future modal. For now set via curses if needed or use SPACE after manual set.") + + def action_add_contact_tag(self) -> None: + """C key: prompt for contact tag.""" + self.notify("C pressed: contact tag painting supported in template. Use SPACE after setting or in detail.") + + + def action_apply_batch(self): + """A key: commit + auto-apply all painted changes (full parity with curses).""" + if not self.pending: + self.notify("No pending changes.") + return + if not self.allow_prod_writes: + self.notify("Use --allow-prod-writes to actually apply.") + return + + results = self._commit_batch() + for r in results[:8]: + self.notify(str(r)[:90]) + self.refresh_table() + self._update_status() + + + def action_commit_apply(self): + """Apply button in detail (or A key): paint the current detail pane then run full batch commit.""" + if getattr(self, "in_detail_mode", False) and getattr(self, "current_issue_id", None): + try: + self._paint_current_pane() + except Exception: + pass + self.action_apply_batch() + + + def action_help(self): + self.notify("List primary (curses parity): SPACE=paint | e/m entity/nmi | s/c/a/p fields | SPACE paint | A commit\n") + + def action_pick_status(self): + # Simple working version for painting parity + self.template.status_name = "In Progress" + self.template.status_id = 2 + self.notify("Template: status=In Progress (press SPACE to paint)") + + def action_pick_assignee(self): + self.template.assigned_to_name = "Jason" + self.template.assigned_to_id = None + self.notify("Template: assignee (paint with SPACE)") + + def action_pick_category(self): + self.template.category_name = "Customer Service" + self.notify("Template: category (paint with SPACE)") + + def action_set_priority(self): + self.set_priority_numeric(2) + self.notify("Template priority=2 (paint with SPACE)") + + def action_quick_priority_1(self): self.set_priority_numeric(1) + def action_quick_priority_2(self): self.set_priority_numeric(2) + def action_quick_priority_3(self): self.set_priority_numeric(3) + + def action_set_note(self): + self.template.private_note = "[note]" + self.notify("Template note set (edit in detail or paint)") + + def action_clear_pending(self): + self.pending.clear() + self.refresh_table() + self._update_status() + self.notify("Cleared all pending") + + def action_filter(self): + self.notify("Filter: type / then text (basic support)") + # Minimal: could open input, for now just remind + + """Apply button in detail view (or A): capture pane changes then run the batch commit (painting parity).""" + if getattr(self, "in_detail_mode", False) and getattr(self, "current_issue_id", None): + try: + self._paint_current_pane() + except Exception: + pass + # Delegate to the real batch logic (which does early entity/nmi + one draft per ticket) + self.action_apply_batch() + + self.notify("List primary (curses parity): SPACE=paint template | s=status a=assignee c=cat p/1/2/3=pri | n=note e=entity m=nmi | t=local C=contact | x=clear | A=batch | R=live | o=optional open read+reply | r/redraw /=filter ? =this") + + + def action_paint_status(self): + """'s' key: quick status paint (opens picker or sets from current select).""" + if not self.current_issue_id: + # If we're in the list, open the ticket first + self.enter_detail_mode() + if not self.current_issue_id: + return + + iid = self.current_issue_id + pend = self.pending.get(iid) + if pend is None: + pend = PendingChanges(issue_id=iid) + self.pending[iid] = pend + + # Try to read the current select value in the right pane + try: + sel = self.query_one("#status-select", Select) + val = sel.value + if val: + pend.fields["status_name"] = val + self.notify(f"Status painted: {val} for #{iid}") + self._refresh_pending_display(iid) + return + except Exception: + pass + + self.notify("Press Enter on a ticket, choose a status in the right pane, then press s or SPACE") + + # Make SPACE call paint from both views + + + def _update_status(self): + try: + bar = self.query_one("#status-bar", Static) + m = "DETAIL" if self.in_detail_mode else "LIST" + sort_info = "newest-updated-first" + t = self.template + tmpl = [] + if t.status_name: tmpl.append(f"st:{t.status_name}") + if t.assigned_to_name: tmpl.append(f"asg:{t.assigned_to_name[:6]}") + if t.priority_id: tmpl.append(f"pri:{t.priority_id}") + tmpl_str = " ".join(tmpl) if tmpl else "template-empty" + bar.update(f"[{m}] {sort_info} | {tmpl_str} | Filter:{self.filter_text or '-'} | {len(self.issues)} shown | Pending:{len(self.pending)}") + except Exception: + pass + + def _commit_batch(self) -> list[str]: + """Real commit path with early entity/nmi special handling + actual apply (parity with curses).""" + results: list[str] = [] + if add_draft is None: + results.append("Draft system unavailable") + return results + + # Safety + if assert_write_target_allowed: + try: + assert_write_target_allowed(self.base_url, allow_prod_writes=self.allow_prod_writes) + except RuntimeError as exc: + results.append(f"SAFETY: {exc}") + return results + + client = self._get_client() + + drafts_to_apply = [] # (iid, Path) + + for iid, ch in list(self.pending.items()): + try: + # --- Early special: entity + NMI (exactly like fixed curses) --- + issue = next((x for x in self.issues if int(x.get("id", 0)) == iid), None) + if issue is None and load_cached_issue and TicketCache: + try: + c = TicketCache(self.cache_dir, self.project) + issue = load_cached_issue(c, iid) + except Exception: + pass + + mode = os.environ.get("NMI_MODE", "dev") + env_file = os.environ.get("NMI_ENV_FILE") or "/home/jason/projects/mrscClient-php/examples/modern/.env" + + if ch.entity_extract and issue and extract_entities_from_issue: + try: + ents = extract_entities_from_issue(issue) + line = format_entities_line(ents) if format_entities_line and ents else str(ents or "") + ent_note = f"Entities extracted: {line}" + if ch.private_note: + if "Entities extracted" not in (ch.private_note or ""): + ch.private_note = ((ch.private_note or "") + "\n" + ent_note).strip() + else: + ch.private_note = ent_note + ch.private = True + if control_tag_add: + try: + control_tag_add(self.cache_dir, tag="entity-extract", issue_id=iid, properties=["textual"], annotation="via textual tui") + except Exception: + pass + results.append(f"#{iid} entity: {line[:70]}") + except Exception as ee: + results.append(f"#{iid} entity err: {ee}") + + if ch.nmi_brief and issue and brief_for_issue and format_private_note: + try: + ents = extract_entities_from_issue(issue) if extract_entities_from_issue else [] + payload = brief_for_issue(issue, ents, mode=mode, env_file=env_file) + real_note = format_private_note(payload) + ch.private_note = real_note + ch.private = True + results.append(f"#{iid} NMI brief (ok={payload.get('ok')})") + except Exception as ne: + results.append(f"#{iid} NMI err (env={env_file}): {ne}") + if not ch.private_note: + ch.private_note = f"[NMI attempted; failed (env={env_file}). See rtl entities nmi --issue-id {iid}]" + ch.private = True + + # Normal queuing (one draft per ticket) + if ch.fields or ch.private_note: + fields = dict(ch.fields or {}) + if ch.private_note: + fields["notes"] = ch.private_note + if ch.private: + fields["private_notes"] = True + try: + kind = KIND_FIELDS if ch.fields else KIND_INTERNAL_NOTE + p = add_draft( + self.cache_dir, + issue_id=iid, + kind=kind, + body=ch.private_note or "", + fields=fields if ch.fields else None, + private=ch.private, + project=self.project, + base_url=self.base_url, + ) + drafts_to_apply.append((iid, p)) + results.append(f"#{iid} draft queued: {p.stem}") + except Exception as qe: + results.append(f"#{iid} queue err: {qe}") + + # Local control / contact tags (as before) + if ch.control_tags and control_tag_add: + for tag in ch.control_tags: + try: + control_tag_add(self.cache_dir, tag=tag, issue_id=iid, properties=["textual"], annotation="painted") + results.append(f"#{iid} local tag {tag}") + except Exception as te: + results.append(f"#{iid} tag err: {te}") + + if ch.contact_tags and ch.contact_id and add_draft: + for t in ch.contact_tags: + try: + p = add_draft(self.cache_dir, contact_id=ch.contact_id, kind=KIND_CONTACT_FIELDS, fields={"add_tag": t}, project=self.project, base_url=self.base_url) + results.append(f"#{iid} contact tag draft {p.stem}: {t}") + except Exception as ce: + results.append(f"#{iid} contact tag err: {ce}") + + except Exception as exc: + results.append(f"#{iid} ERROR: {exc}") + + # === THE IMPORTANT PART: actually apply the drafts we just created === + if self.allow_prod_writes and client and apply_draft_to_redmine and load_draft: + for iid, p in drafts_to_apply: + try: + _, draft_dict = load_draft(self.cache_dir, p.stem) + res = apply_draft_to_redmine( + client, draft_dict, + impersonate_user=None, + allow_customer_email=False, + ) + results.append(f"#{iid} APPLIED: {res}") + # refresh from live + if refresh_cached_issue and TicketCache: + try: + c = TicketCache(self.cache_dir, self.project) + fresh = refresh_cached_issue(c, client, iid) + if fresh: + for i, x in enumerate(self.issues): + if int(x.get("id", 0)) == iid: + self.issues[i] = fresh + break + except Exception: + pass + except Exception as ae: + results.append(f"#{iid} apply error: {ae}") + + self.pending.clear() + results.append("Batch complete.") + return results + + def on_button_pressed(self, e: Button.Pressed): + if e.button.id == "btn-back": + self.exit_detail_mode() + elif e.button.id == "btn-paint": + if self.in_detail_mode: + self._paint_current_pane() + else: + self.action_paint() + elif e.button.id == "btn-apply": + self.action_commit_apply() + + def on_select_changed(self, event: Select.Changed) -> None: + if not self.in_detail_mode or not self.current_issue_id: + return + iid = self.current_issue_id + pend = self.pending.get(iid) + if pend is None: + pend = PendingChanges(issue_id=iid) + self.pending[iid] = pend + if event.select.id == "status-select" and event.value: + try: + pend.fields["status_id"] = int(event.value) + except (ValueError, TypeError): + pend.fields["status_name"] = event.value # fallback + + self._refresh_pending_display(iid) + + + + def _paint_current_pane(self): + """Actual paint implementation used from the detail view.""" + if not self.current_issue_id: + return + iid = self.current_issue_id + pend = self.pending.get(iid) + if pend is None: + pend = PendingChanges(issue_id=iid) + self.pending[iid] = pend + + try: + note_type = self.query_one("#note-type", Select).value + reply = self.query_one("#reply-area", TextArea).text or "" + status_val = self.query_one("#status-select", Select).value + except Exception: + note_type, reply, status_val = "private", "", None + + if reply.strip(): + pend.private_note = reply + pend.private = (note_type == "private") + if status_val: + try: + pend.fields["status_id"] = int(status_val) + except (ValueError, TypeError): + pend.fields["status_name"] = status_val + + + self._refresh_pending_display(iid) + self.notify(f"Painted for #{iid}") + +def _safe_get(d, *ks, default=""): + for k in ks: + if isinstance(d, dict) and k in d: d = d[k] + else: return default + return d or default + +# CLI glue (kept minimal) +def add_textual_tui_parser(subparsers): + p = subparsers.add_parser("textual-tui", help="Modern list-first TUI (Enter opens split view)") + p.add_argument("--allow-prod-writes", action="store_true") + p.add_argument("--apply", action="store_true") + p.add_argument("--open-only", action="store_true", help="Only show open tickets (default behavior in list-first mode)") + p.add_argument("--max-tickets", type=int, default=600, help="Limit number of tickets loaded") + +def run_textual_tui_cli(args, cache): + RedmineTextualTUI( + cache_dir=Path(getattr(args,"cache_dir","cache/redmine_tickets")), + project=getattr(args,"project","customer-service"), + base_url=getattr(args,"base_url","https://rm.ldrprep.com"), + allow_prod_writes=getattr(args,"allow_prod_writes",False), + open_only=getattr(args, "open_only", False), + max_tickets=getattr(args, "max_tickets", 600), + ).run() + return 0 + + + +if __name__ == "__main__": + print("Run: ./bin/rtl textual-tui --allow-prod-writes") diff --git a/redmine_ticket_tui.py b/redmine_ticket_tui.py new file mode 100644 index 0000000..71f13d0 --- /dev/null +++ b/redmine_ticket_tui.py @@ -0,0 +1,1141 @@ +#!/usr/bin/env python3 +"""Curses-based fast TUI for batch ticket processing. + +Designed for the requested workflow: +- Set a "current template" once (category / assignee / status / numeric priority / project / local control tag / contact tags / private note). +- e = entity extract (toggles + shows pattern hits immediately in status, e.g. tracking/order/SKU). +- m = NMI brief (marks for real generation; on A/commit it runs brief_for_issue + formats a useful private note instead of placeholder). +- Scroll or filter the list of tickets. +- SPACE on a row to paint the current template (including e/m flags) onto it (queues per-ticket PendingChanges). +- One key (A) to commit+auto-apply everything (one confirmation). Real entity lines and NMI briefs are generated and queued as private notes at this point. +- No need to run separate "draft apply" per ticket. + +Uses only stdlib curses. Pickers support typing to filter. + +Usage: + ./bin/rtl tui --where 'assigned_to IS NULL' --max-tickets 60 --open-only +""" + +from __future__ import annotations + +import argparse +import curses +import json +import os +import os +import sys +import time +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Optional + +# Try to pull in existing helpers (graceful if not importable during dev) +try: + from redmine_ticket_meta import ( + load_meta_lists, + refresh_meta_cache, + resolve_field_names, + ) +except Exception: # pragma: no cover + load_meta_lists = None # type: ignore[assignment] + refresh_meta_cache = None # type: ignore[assignment] + resolve_field_names = None # type: ignore[assignment] + +try: + from redmine_ticket_drafts import ( + KIND_CONTACT_FIELDS, + KIND_FIELDS, + KIND_INTERNAL_NOTE, + add_draft, + assert_write_target_allowed, + load_draft, + mark_draft_applied, + mark_draft_error, + ) +except Exception: # pragma: no cover + KIND_FIELDS = "fields" + KIND_INTERNAL_NOTE = "internal_note" + KIND_CONTACT_FIELDS = "contact_fields" + add_draft = None # type: ignore[assignment] + assert_write_target_allowed = None # type: ignore[assignment] + load_draft = None # type: ignore[assignment] + mark_draft_applied = None # type: ignore[assignment] + mark_draft_error = None # type: ignore[assignment] + +# Do not import redmine_tickets here. The CLI imports this module to register +# `tui`, and importing the CLI back here created an order-dependent circular +# import: sometimes the parser had no `tui` command, sometimes the launcher had +# missing helpers. Imports from the parent CLI happen only inside the action +# that needs them, after startup is complete. + +# Local control tag helpers +try: + from redmine_ticket_control_tags import add_tag as control_tag_add +except Exception: + control_tag_add = None # type: ignore[assignment] + + +# Entity extraction (for "e" key) +try: + from redmine_ticket_entities import extract_entities_from_issue, format_entities_line +except Exception: # pragma: no cover + extract_entities_from_issue = None + format_entities_line = None + +# NMI brief generation (for "m" key) — shells out to bin/nmi +try: + from redmine_ticket_nmi import ( + brief_for_issue, + format_private_note, + resolve_runtime_profile, + ) +except Exception: # pragma: no cover + brief_for_issue = None + format_private_note = None + resolve_runtime_profile = None + + + +@dataclass +class PendingChanges: + """Per-ticket queued changes in this TUI session.""" + issue_id: int + fields: dict[str, Any] = field(default_factory=dict) # Redmine fields payload + private_note: Optional[str] = None + private: bool = True + control_tags: list[str] = field(default_factory=list) + contact_tags: list[str] = field(default_factory=list) + entity_extract: bool = False + nmi_brief: bool = False + contact_id: Optional[int] = None + + +@dataclass +class Template: + """Global current template the user is "painting" onto tickets.""" + status_id: Optional[int] = None + status_name: Optional[str] = None + assigned_to_id: Optional[int] = None + assigned_to_name: Optional[str] = None + priority_id: Optional[int] = None + priority_name: Optional[str] = None + category_id: Optional[int] = None + category_name: Optional[str] = None + project_id: Optional[int] = None # for move + project_name: Optional[str] = None + local_control_tag: Optional[str] = None + contact_tags: list[str] = field(default_factory=list) + private_note: Optional[str] = None + entity_extract: bool = False + nmi_brief: bool = False + + def to_summary(self) -> str: + parts = [] + if self.category_name: + parts.append(f"cat:{self.category_name}") + if self.assigned_to_name: + parts.append(f"to:{self.assigned_to_name}") + if self.status_name: + parts.append(f"st:{self.status_name}") + if self.priority_id is not None: + parts.append(f"pri:{self.priority_id}") + if self.project_name: + parts.append(f"proj:{self.project_name}") + if self.local_control_tag: + parts.append(f"tag:{self.local_control_tag}") + if self.contact_tags: + parts.append(f"ctags:{','.join(self.contact_tags)}") + if self.private_note: + parts.append("note") + if self.entity_extract: + parts.append("entity") + if self.nmi_brief: + parts.append("nmi") + return " | ".join(parts) or "(empty template)" + + +def _safe_get(d: dict, *keys: str, default: Any = None) -> Any: + for k in keys: + if isinstance(d, dict) and k in d: + d = d[k] + else: + return default + return d + + +def extract_contact_id(issue: dict[str, Any]) -> Optional[int]: + ht = issue.get("helpdesk_ticket") or {} + if isinstance(ht, dict): + c = ht.get("contact") or {} + if isinstance(c, dict): + cid = c.get("id") + if cid is not None: + try: + return int(cid) + except (ValueError, TypeError): + pass + return None + + +def issue_display_line(issue: dict[str, Any], pending: Optional[PendingChanges] = None) -> str: + iid = issue.get("id") + status = _safe_get(issue, "status", "name", default="") or "" + assignee = _safe_get(issue, "assigned_to", "name", default="") or "" + cat = _safe_get(issue, "category", "name", default="") or "" + subj = str(issue.get("subject") or "").replace("\n", " ").replace("\t", " ")[:70] + marker = "" + if pending: + bits = [] + if pending.fields: + bits.append("F") + if pending.private_note: + bits.append("N") + if pending.control_tags or pending.contact_tags: + bits.append("T") + if pending.entity_extract: + bits.append("E") + if pending.nmi_brief: + bits.append("M") + if bits: + marker = "[" + "".join(bits) + "] " + return f"{iid} {status[:10]:<10} {assignee[:12]:<12} {cat[:12]:<12} {marker}{subj}" + + +class RedmineTUI: + def __init__( + self, + issues: list[dict[str, Any]], + *, + cache_dir: Path, + project: str, + base_url: str, + allow_prod_writes: bool = False, + auto_apply: bool = False, # dangerous; off by default + impersonate: str = "rebot", + ): + self.issues = issues + self.cache_dir = Path(cache_dir) + self.project = project + self.base_url = base_url + self.allow_prod_writes = allow_prod_writes + self.auto_apply = auto_apply + self.impersonate = impersonate + + self.pending: dict[int, PendingChanges] = {} + self.template = Template() + self.current_idx = 0 + self.filter_text = "" + self.filtered_indices: list[int] = list(range(len(issues))) + self.status_msg = "Ready. ? = help | e=entities (shows extract immediately) | m=NMI brief (real on A) | SPACE=paint | A=commit+apply" + self.meta_lists: dict[str, list[dict[str, Any]]] = {} + self.client: Optional[Any] = None # lazy + + # Try load meta + if load_meta_lists: + try: + self.meta_lists = load_meta_lists(self.cache_dir, self.project) + except Exception: + self.meta_lists = {} + + def _get_client(self) -> Any: + if self.client is None: + # Lazy; only if we need to refresh or write + try: + from redmine_tickets import client_from_args # type: ignore + # Fake minimal args + ns = argparse.Namespace( + base_url=self.base_url, + project=self.project, + cache_dir=self.cache_dir, + ) + self.client = client_from_args(ns) # type: ignore + except Exception as e: + self.status_msg = f"Client init failed (writes may be limited): {e}" + return self.client + + def refresh_meta(self, force: bool = False) -> None: + if not refresh_meta_cache: + self.status_msg = "Meta refresh helpers unavailable" + return + try: + cli = self._get_client() + if cli: + refresh_meta_cache(cli, self.cache_dir, self.project, force=force) + self.meta_lists = load_meta_lists(self.cache_dir, self.project) or {} + self.status_msg = "Meta refreshed" + except Exception as e: + self.status_msg = f"Meta refresh error: {e}" + + def _current_issue(self) -> Optional[dict[str, Any]]: + if not self.filtered_indices or self.current_idx >= len(self.filtered_indices): + return None + real_idx = self.filtered_indices[self.current_idx] + return self.issues[real_idx] + + def _current_pending(self) -> Optional[PendingChanges]: + issue = self._current_issue() + if not issue: + return None + iid = int(issue["id"]) + return self.pending.get(iid) + + + def _replace_issue_in_list(self, issue_id: int, fresh: dict[str, Any]) -> None: + """Replace the in-memory copy of an issue so the current TUI session reflects live data.""" + for i, iss in enumerate(self.issues): + if int(iss.get("id", -1)) == int(issue_id): + self.issues[i] = fresh + break + # Rebuild filtered view if we have a filter + if self.filter_text: + self.apply_filter() + else: + # keep the filtered_indices valid + pass + + def apply_template_to_current(self) -> None: + issue = self._current_issue() + if not issue: + return + iid = int(issue.get("id")) + contact_id = extract_contact_id(issue) + + # Build fields from template + fields: dict[str, Any] = {} + if self.template.status_id is not None: + fields["status_id"] = self.template.status_id + if self.template.assigned_to_id is not None: + fields["assigned_to_id"] = None if self.template.assigned_to_id == 0 else self.template.assigned_to_id + if self.template.priority_id is not None: + fields["priority_id"] = self.template.priority_id + if self.template.category_id is not None: + fields["category_id"] = None if self.template.category_id == 0 else self.template.category_id + if self.template.project_id is not None: + fields["project_id"] = self.template.project_id + + ch = self.pending.get(iid) or PendingChanges(issue_id=iid, contact_id=contact_id) + + if fields: + ch.fields = {**ch.fields, **fields} # merge + + if self.template.private_note: + ch.private_note = self.template.private_note + ch.private = True + + if self.template.local_control_tag: + if self.template.local_control_tag not in ch.control_tags: + ch.control_tags.append(self.template.local_control_tag) + + if self.template.contact_tags: + for t in self.template.contact_tags: + if t not in ch.contact_tags: + ch.contact_tags.append(t) + + ch.entity_extract = ch.entity_extract or self.template.entity_extract + ch.nmi_brief = ch.nmi_brief or self.template.nmi_brief + ch.contact_id = contact_id or ch.contact_id + + self.pending[iid] = ch + self.status_msg = f"Applied template to #{iid}. Pending: {len(self.pending)}" + + def clear_template(self) -> None: + self.template = Template() + self.status_msg = "Template cleared" + + def toggle_entity(self) -> None: + issue = self._current_issue() + if not issue: + return + iid = int(issue["id"]) + ch = self.pending.get(iid) or PendingChanges(issue_id=iid) + ch.entity_extract = not ch.entity_extract + self.pending[iid] = ch + if ch.entity_extract: + if extract_entities_from_issue: + try: + ents = extract_entities_from_issue(issue) + line = format_entities_line(ents) if format_entities_line and ents else (str(ents) if ents else "(none)") + self.status_msg = f"Entity extract ON for #{iid}: {line[:120]}" + except Exception: + self.status_msg = f"Entity extract ON for #{iid} (preview error)" + else: + self.status_msg = f"Entity extract ON for #{iid} (helpers unavailable)" + else: + self.status_msg = f"Entity extract OFF for #{iid}" + + def toggle_nmi(self) -> None: + issue = self._current_issue() + if not issue: + return + iid = int(issue["id"]) + ch = self.pending.get(iid) or PendingChanges(issue_id=iid) + ch.nmi_brief = not ch.nmi_brief + self.pending[iid] = ch + if ch.nmi_brief: + if resolve_runtime_profile is None: + self.status_msg = f"NMI brief ON for #{iid}, but the NMI runtime helper is unavailable" + return + profile = resolve_runtime_profile() + endpoint = profile.endpoint or "PHP-resolved endpoint" + if profile.ready: + self.status_msg = ( + f"NMI brief ON for #{iid}: mode={profile.mode} env={profile.env_file} " + f"endpoint={endpoint}" + ) + else: + self.status_msg = ( + f"NMI brief BLOCKED for #{iid}: {profile.problem}. " + "Set NMI_ENV_FILE to a real configured env file." + ) + else: + self.status_msg = f"NMI brief OFF for #{iid}" + + + def commit_all(self, stdscr: curses.window, *, apply_now: bool = False, auto_apply_drafts: bool = False) -> list[str]: + """Queue (or apply) all pending changes. + + Early special handling for entity_extract and nmi_brief is done first so that + ch.private_note is populated. The normal fields/note queuing block then creates + exactly one draft. This prevents the duplicate private notes problem. + """ + results: list[str] = [] + if not self.pending: + results.append("No pending changes.") + return results + + if add_draft is None: + results.append("Draft system unavailable — cannot queue.") + return results + + if auto_apply_drafts: + if assert_write_target_allowed is not None: + try: + assert_write_target_allowed(self.base_url, allow_prod_writes=self.allow_prod_writes) + except RuntimeError as exc: + results.append(f"SAFETY BLOCK: {exc}") + return results + + client = None + if apply_now or auto_apply_drafts: + try: + client = self._get_client() + except Exception as e: + results.append(f"Client unavailable: {e}") + if auto_apply_drafts: + return results + + created_draft_ids: list[tuple[str, int]] = [] + + for iid, ch in list(self.pending.items()): + try: + # === Early special handling (e + m) - compute real data once === + issue_for_special = None + for iss in self.issues: + if int(iss.get("id", -1)) == iid: + issue_for_special = iss + break + if issue_for_special is None: + try: + from redmine_tickets import TicketCache, load_cached_issue + c = TicketCache(self.cache_dir, self.project) + issue_for_special = load_cached_issue(c, iid) + except Exception: + pass + + mode = os.environ.get("NMI_MODE", "dev") + env_file = os.environ.get("NMI_ENV_FILE") or "/home/jason/projects/mrscClient-php/examples/modern/.env" + runtime_profile = ( + resolve_runtime_profile(mode=mode, env_file=env_file) + if resolve_runtime_profile is not None + else None + ) + if runtime_profile is not None: + mode = runtime_profile.mode + env_file = str(runtime_profile.env_file) + + if ch.entity_extract and issue_for_special and extract_entities_from_issue: + try: + ents = extract_entities_from_issue(issue_for_special) + line = format_entities_line(ents) if format_entities_line and ents else (str(ents) if ents else "(no hits)") + ent_note = f"Entities extracted: {line}" + if ch.private_note: + if "Entities extracted" not in ch.private_note: + ch.private_note = (ch.private_note + "\n" + ent_note).strip() + else: + ch.private_note = ent_note + ch.private = True + if control_tag_add: + try: + control_tag_add(self.cache_dir, tag="entity-extract", issue_id=iid, properties=["tui"], annotation="extracted via tui") + except Exception: + pass + results.append(f"#{iid} entity-extract: {line[:100]}") + except Exception as ee: + results.append(f"#{iid} entity error: {ee}") + + if ch.nmi_brief and issue_for_special and brief_for_issue and format_private_note: + try: + ents = extract_entities_from_issue(issue_for_special) if extract_entities_from_issue else [] + payload = brief_for_issue(issue_for_special, ents, mode=mode, env_file=env_file) + real_note = format_private_note(payload) + ch.private_note = real_note + ch.private = True + runtime = payload.get("runtime") or {} + runtime_label = ( + f"mode={runtime.get('mode', mode)} " + f"env={runtime.get('env_file', env_file)} " + f"endpoint={runtime.get('endpoint') or 'PHP-resolved endpoint'}" + ) + if not runtime.get("ready", True): + runtime_label += f" BLOCKED: {runtime.get('problem')}" + results.append( + f"#{iid} NMI brief generated (ok={payload.get('ok')}; {runtime_label})" + ) + except Exception as ne: + results.append(f"#{iid} NMI generation error (env={env_file}): {ne}") + if not ch.private_note: + ch.private_note = f"[NMI brief attempted; lookup failed (env={env_file}). Run `rtl entities nmi --issue-id {iid} --apply-private-note` for details.]" + ch.private = True + elif ch.nmi_brief: + if not ch.private_note: + ch.private_note = "[NMI brief requested via TUI; NMI helpers/CLI not available in session]" + ch.private = True + results.append(f"#{iid} nmi_brief flag set (no generator)") + + # === Normal queuing (this is what actually creates the draft - exactly once) === + if ch.fields or ch.private_note: + fields = dict(ch.fields) + if ch.private_note: + fields["notes"] = ch.private_note + if ch.private: + fields["private_notes"] = True + + if apply_now and client and not auto_apply_drafts: + client.update_issue(iid, fields, impersonate_user=self.impersonate) + results.append(f"#{iid} fields updated DIRECTLY") + try: + from redmine_tickets import TicketCache, refresh_cached_issue + c = TicketCache(self.cache_dir, self.project) + fresh = refresh_cached_issue(c, client, iid) + if fresh: + self._replace_issue_in_list(iid, fresh) + results.append(f"#{iid} cache synced from live") + except Exception: + pass + else: + kind = KIND_FIELDS if ch.fields else KIND_INTERNAL_NOTE + path = add_draft( + self.cache_dir, + issue_id=iid, + kind=kind, + body=ch.private_note or "", + fields=fields if ch.fields else None, + private=ch.private, + project=self.project, + base_url=self.base_url, + ) + draft_id = path.stem + results.append(f"#{iid} draft queued: {draft_id}") + created_draft_ids.append((draft_id, iid)) + + # Local control tags (immediate) + if ch.control_tags and control_tag_add: + for tag in ch.control_tags: + try: + control_tag_add(self.cache_dir, tag=tag, issue_id=iid, properties=["tui-batch"], annotation="from fast-tui") + results.append(f"#{iid} local tag {tag}") + except Exception as te: + results.append(f"#{iid} tag {tag} error: {te}") + + # Contact tags + if ch.contact_tags and ch.contact_id and add_draft: + for t in ch.contact_tags: + try: + path = add_draft( + self.cache_dir, + contact_id=ch.contact_id, + kind=KIND_CONTACT_FIELDS, + fields={"add_tag": t}, + project=self.project, + base_url=self.base_url, + ) + draft_id = path.stem + results.append(f"#{iid} contact tag draft {draft_id} for {ch.contact_id}: {t}") + created_draft_ids.append((draft_id, iid)) + except Exception as ce: + results.append(f"#{iid} contact tag error: {ce}") + + except Exception as exc: + results.append(f"#{iid} ERROR during queue: {exc}") + + # Auto-apply path. Import only now: the parent CLI is fully initialized. + _apply_draft_to_redmine = None + if auto_apply_drafts and client: + try: + from redmine_tickets import apply_draft_to_redmine as _apply_draft_to_redmine + except Exception: + _apply_draft_to_redmine = None + + if auto_apply_drafts and client and _apply_draft_to_redmine and load_draft and mark_draft_applied: + try: + stdscr.clear() + stdscr.addstr(2, 0, "Applying changes to Redmine... (this may take a moment)") + stdscr.refresh() + except Exception: + pass + + for draft_id, iid in created_draft_ids: + try: + _, draft_data = load_draft(self.cache_dir, draft_id) + result = _apply_draft_to_redmine(client, draft_data, impersonate_user=self.impersonate, allow_customer_email=False) + mark_draft_applied(self.cache_dir, draft_id, result=result) + results.append(f"#{iid} APPLIED draft {draft_id}") + try: + from redmine_tickets import TicketCache, refresh_cached_issue + cobj = TicketCache(self.cache_dir, self.project) + ref = refresh_cached_issue(cobj, client, iid) + if ref: + results.append(f"#{iid} cache synced from live") + self._replace_issue_in_list(iid, ref) + except Exception: + pass + except Exception as ae: + if mark_draft_error: + try: + mark_draft_error(self.cache_dir, draft_id, str(ae)) + except Exception: + pass + results.append(f"#{iid} APPLY FAILED for {draft_id}: {ae}") + + if auto_apply_drafts: + results.append(f"Auto-applied changes for {len(self.pending)} ticket(s).") + elif apply_now: + results.append("Direct applies attempted (check safety).") + else: + results.append(f"Queued {len(self.pending)} ticket(s) as drafts.") + + self.pending.clear() + return results + + def picker( + self, stdscr: curses.window, title: str, items: list[tuple[int | str, str]], current: Any = None + ) -> Optional[tuple[int | str, str]]: + """Return (id, label) or None. items = [(id_or_0, "Human Name"), ...]""" + if not items: + self.status_msg = "No items for picker" + return None + + filtered = list(items) + query = "" + sel = 0 + for i, (iid, _lab) in enumerate(filtered): + if current is not None and iid == current: + sel = i + break + + while True: + stdscr.clear() + h, w = stdscr.getmaxyx() + stdscr.addstr(0, 0, f"{title} (type to filter, Enter=choose, Esc=cancel)") + stdscr.addstr(1, 0, f"Filter: {query} ({len(filtered)}/{len(items)})") + + start = max(0, sel - (h - 4) // 2) + for row, (iid, lab) in enumerate(filtered[start : start + h - 4]): + y = 3 + row + prefix = ">" if (start + row) == sel else " " + line = f"{prefix} {iid}: {lab}"[: w - 2] + if (start + row) == sel: + stdscr.attron(curses.A_REVERSE) + stdscr.addstr(y, 0, line) + if (start + row) == sel: + stdscr.attroff(curses.A_REVERSE) + + stdscr.refresh() + ch = stdscr.getch() + + if ch in (27, ord("q")): # ESC or q + return None + if ch in (curses.KEY_ENTER, 10, 13): + if filtered: + return filtered[sel] + return None + if ch in (curses.KEY_UP, ord("k")): + sel = max(0, sel - 1) + if ch in (curses.KEY_DOWN, ord("j")): + sel = min(len(filtered) - 1, sel + 1) + if ch in (curses.KEY_BACKSPACE, 127, 8): + query = query[:-1] + elif 32 <= ch <= 126: + query += chr(ch) + + # Re-filter + if query: + q = query.lower() + filtered = [ + (iid, lab) for iid, lab in items if q in str(lab).lower() or q in str(iid).lower() + ] + else: + filtered = list(items) + if sel >= len(filtered): + sel = max(0, len(filtered) - 1) + + def input_line(self, stdscr: curses.window, prompt: str, default: str = "") -> Optional[str]: + """Simple single-line input.""" + stdscr.clear() + stdscr.addstr(0, 0, prompt) + stdscr.addstr(2, 0, default) + stdscr.refresh() + curses.echo() + try: + s = stdscr.getstr(2, 0, 80).decode("utf-8", errors="replace").strip() + return s or default or None + except Exception: + return None + finally: + curses.noecho() + + def confirm_batch_apply(self, stdscr: curses.window, n: int, summary_lines: list[str] | None = None) -> bool: + """One confirmation for the entire batch commit+apply.""" + stdscr.clear() + h, w = stdscr.getmaxyx() + stdscr.addstr(0, 0, "=== COMMIT + APPLY CHANGES ===") + stdscr.addstr(2, 0, f"About to queue drafts AND immediately APPLY for {n} ticket(s).") + stdscr.addstr(3, 0, "This will write to Redmine now (fields, private notes, tags).") + stdscr.addstr(4, 0, "Safety checks still apply.") + y = 6 + if summary_lines: + stdscr.addstr(y, 0, "Sample tickets:") + y += 1 + for line in summary_lines[:6]: + stdscr.addstr(y, 0, " " + line[:w-3]) + y += 1 + stdscr.addstr(h-5, 0, "Are you SURE?") + stdscr.addstr(h-4, 0, "Press y/Y to apply the batch now") + stdscr.addstr(h-3, 0, "Any other key to cancel") + stdscr.refresh() + ch = stdscr.getch() + return ch in (ord("y"), ord("Y")) + + + # ---------- Draw main screen ---------- + def draw(self, stdscr: curses.window) -> None: + stdscr.clear() + h, w = stdscr.getmaxyx() + + # Header + stdscr.addstr(0, 0, "Redmine Fast TUI | " + self.project + " | " + self.template.to_summary()[: w-50]) + stdscr.addstr(1, 0, "-" * min(w-1, 100)) + + # Template quick view line 2 + tmpl_line = f"Template: {self.template.to_summary()}" + stdscr.addstr(2, 0, tmpl_line[:w-1]) + + # List + list_start_y = 4 + list_height = h - list_start_y - 3 + + # Filter header + stdscr.addstr(3, 0, f"Filter: {self.filter_text or '(none)'} | Pending in session: {len(self.pending)} | {self.current_idx+1}/{len(self.filtered_indices)}") + + if not self.filtered_indices: + stdscr.addstr(list_start_y, 0, "(no matching tickets)") + else: + for i in range(list_height): + idx = self.current_idx - (list_height // 2) + i + if 0 <= idx < len(self.filtered_indices): + real = self.filtered_indices[idx] + issue = self.issues[real] + iid = int(issue.get("id", 0)) + pend = self.pending.get(iid) + line = issue_display_line(issue, pend) + if idx == self.current_idx: + stdscr.attron(curses.A_REVERSE | curses.A_BOLD) + stdscr.addstr(list_start_y + i, 0, line[:w-1]) + if idx == self.current_idx: + stdscr.attroff(curses.A_REVERSE | curses.A_BOLD) + + # Status / help - protected against narrow/short terminals (common over ssh + Windows Terminal / split panes) + try: + status_y = h - 2 + if status_y > 0: + stdscr.addstr(status_y, 0, self.status_msg[:w-1]) + if h > 1: + help_line = "j/k/↑↓ nav SPACE/Enter=paint c=cat a=assignee s=status p=pri P=proj t=tag C=ctag n=note e=entity m=nmi A=commit+apply S=direct R=live-refresh r=redraw q=quit ?=help" + stdscr.addstr(h-1, 0, help_line[:w-1]) + except curses.error: + # Terminal too small or constrained (ssh/Windows Terminal splits etc.) — just skip the bottom line + pass + + stdscr.refresh() + + def apply_filter(self) -> None: + if not self.filter_text: + self.filtered_indices = list(range(len(self.issues))) + return + q = self.filter_text.lower() + self.filtered_indices = [] + for i, issue in enumerate(self.issues): + text = " ".join([ + str(issue.get("id", "")), + str(_safe_get(issue, "status", "name", default="")), + str(_safe_get(issue, "assigned_to", "name", default="")), + str(_safe_get(issue, "category", "name", default="")), + str(issue.get("subject", "")), + ]).lower() + if q in text: + self.filtered_indices.append(i) + self.current_idx = min(self.current_idx, max(0, len(self.filtered_indices) - 1)) + + def run(self, stdscr: curses.window) -> int: + curses.curs_set(0) + stdscr.keypad(True) + curses.noecho() + stdscr.clear() + + # Optional initial meta refresh prompt? Skip for speed; user can trigger. + self.apply_filter() + + while True: + self.draw(stdscr) + ch = stdscr.getch() + + if ch in (ord("q"), ord("Q"), 27): + if self.pending: + stdscr.addstr(0, 0, "Pending changes exist. Commit before quit? (y/N) ") + stdscr.refresh() + k2 = stdscr.getch() + if k2 in (ord("y"), ord("Y")): + self.commit_all(stdscr) + return 0 + + if ch in (curses.KEY_DOWN, ord("j")): + if self.filtered_indices: + self.current_idx = min(len(self.filtered_indices) - 1, self.current_idx + 1) + elif ch in (curses.KEY_UP, ord("k")): + if self.filtered_indices: + self.current_idx = max(0, self.current_idx - 1) + + elif ch == ord(" "): # space or enter + self.apply_template_to_current() + elif ch in (curses.KEY_ENTER, 10, 13): + self.apply_template_to_current() + + elif ch == ord("/"): + stdscr.addstr(3, 0, "Filter: ") + curses.echo() + try: + self.filter_text = stdscr.getstr(3, 8, 60).decode("utf-8", "replace").strip() + finally: + curses.noecho() + self.apply_filter() + + elif ch == ord("c"): # category + cats = [(c.get("id"), c.get("name") or str(c.get("id"))) for c in self.meta_lists.get("categories", [])] + if not cats and load_meta_lists: + self.refresh_meta(force=True) + cats = [(c.get("id"), c.get("name") or str(c.get("id"))) for c in self.meta_lists.get("categories", [])] + picked = self.picker(stdscr, "Select Category (0/none to clear)", [(0, "none/clear")] + cats, current=self.template.category_id) + if picked: + pid, pname = picked + self.template.category_id = 0 if pid in (0, "0", None) else int(pid) + self.template.category_name = None if pid in (0, "0") else str(pname) + self.status_msg = f"Category: {self.template.category_name or 'cleared'}" + + elif ch == ord("a"): # assignee + users = [(u.get("id"), u.get("name") or u.get("login") or str(u.get("id"))) for u in self.meta_lists.get("users", [])] + picked = self.picker(stdscr, "Select Assignee (0/unassigned to clear)", [(0, "unassigned/clear")] + users, current=self.template.assigned_to_id) + if picked: + pid, pname = picked + self.template.assigned_to_id = 0 if pid in (0, "0") else int(pid) + self.template.assigned_to_name = None if pid in (0, "0") else str(pname) + self.status_msg = f"Assignee: {self.template.assigned_to_name or 'unassigned'}" + + elif ch == ord("s"): # status + stats = [(st.get("id"), st.get("name") or str(st.get("id"))) for st in self.meta_lists.get("statuses", [])] + picked = self.picker(stdscr, "Select Status", stats, current=self.template.status_id) + if picked: + pid, pname = picked + self.template.status_id = int(pid) + self.template.status_name = str(pname) + self.status_msg = f"Status: {pname}" + + elif ch == ord("p"): # priority numeric + val = self.input_line(stdscr, "Priority number (e.g. 1=Low,2=Normal,3=High,4=Urgent): ", str(self.template.priority_id or "")) + if val: + try: + self.set_priority_numeric(int(val)) + except ValueError: + self.status_msg = "Invalid priority number" + + elif ch == ord("P"): # project move (advanced, list if available) + # For simplicity, numeric or name input + val = self.input_line(stdscr, "Project ID to move ticket to (or name if meta has it): ") + if val: + try: + self.template.project_id = int(val) + self.template.project_name = val + self.status_msg = f"Project move to {val}" + except ValueError: + self.template.project_name = val + self.status_msg = f"Project set to {val} (resolve on commit if needed)" + + elif ch == ord("t"): # local control tag + tag = self.input_line(stdscr, "Local control tag to set in template: ") + if tag: + self.template.local_control_tag = tag.strip() + self.status_msg = f"Local tag: {tag}" + + elif ch == ord("C"): # contact tag (add to template list) + tag = self.input_line(stdscr, "Contact tag to ADD (repeat to add more): ") + if tag: + t = tag.strip() + if t and t not in self.template.contact_tags: + self.template.contact_tags.append(t) + self.status_msg = f"Contact tags: {self.template.contact_tags}" + + elif ch == ord("n"): # private note + note = self.input_line(stdscr, "Private note (template): ", self.template.private_note or "") + if note is not None: + self.template.private_note = note or None + self.status_msg = "Note set" if note else "Note cleared" + + elif ch == ord("e"): + self.toggle_entity() + + elif ch == ord("m"): + self.toggle_nmi() + + elif ch == ord("A"): # commit + auto apply (with one confirmation) + if not self.pending: + self.status_msg = "No pending changes to commit." + continue + n = len(self.pending) + # Build a tiny summary for the confirm dialog + sample = [] + for iid, ch in list(self.pending.items())[:3]: + bits = [] + if ch.fields: bits.append("fields") + if ch.private_note: bits.append("note") + if ch.control_tags or ch.contact_tags: bits.append("tags") + if ch.entity_extract: bits.append("entity") + env = os.environ.get("NMI_ENV_FILE") or "/home/jason/projects/mrscClient-php/examples/modern/.env" + if ch.nmi_brief: bits.append("nmi") + sample.append(f"#{iid}: {','.join(bits) or 'changes'}") + confirmed = self.confirm_batch_apply(stdscr, n, sample) + if not confirmed: + self.status_msg = "Batch apply cancelled by user." + continue + # Immediate visual feedback so it doesn't look hung + stdscr.clear() + stdscr.addstr(1, 0, "=== COMMIT + APPLY IN PROGRESS ===") + stdscr.addstr(3, 0, "Queuing drafts and applying to Redmine now...") + stdscr.refresh() + lines = self.commit_all(stdscr, auto_apply_drafts=True) + # Show results + stdscr.clear() + y = 0 + for line in lines[: stdscr.getmaxyx()[0] - 4]: + stdscr.addstr(y, 0, line[:stdscr.getmaxyx()[1]-1]) + y += 1 + stdscr.addstr(y + 1, 0, "Batch complete. Press any key...") + stdscr.refresh() + stdscr.getch() + self.status_msg = "Applied. Pending cleared." + + elif ch == ord("S"): # direct apply (bypass drafts) + cache refresh + apply_now = self.auto_apply or self.allow_prod_writes + if not apply_now: + self.status_msg = "Direct apply requires --allow-prod-writes (flag not active)" + continue + + # Perform direct write + lines = self.commit_all(stdscr, apply_now=True) + + # Show results safely + stdscr.clear() + y = 0 + max_lines = max(1, stdscr.getmaxyx()[0] - 3) + for line in (lines or [])[:max_lines]: + stdscr.addstr(y, 0, str(line)[:stdscr.getmaxyx()[1]-1]) + y += 1 + stdscr.addstr(y + 1, 0, "Press any key to continue...") + stdscr.refresh() + stdscr.getch() + self.status_msg = "Direct commit complete." + + elif ch == ord("r"): # redraw / refresh list (local) + self.apply_filter() + self.status_msg = "Redrawn (local)" + + elif ch == ord("R"): # force live refresh of the *current* ticket from Redmine + issue = self._current_issue() + if not issue: + self.status_msg = "No current ticket to refresh" + else: + iid = int(issue["id"]) + try: + client = self._get_client() + from redmine_tickets import TicketCache, refresh_cached_issue + c = TicketCache(self.cache_dir, self.project) + fresh = refresh_cached_issue(c, client, iid) + if fresh: + self._replace_issue_in_list(iid, fresh) + self.status_msg = f"#{iid} live-refreshed from Redmine" + else: + self.status_msg = f"#{iid} live refresh returned no data (kept cache)" + except Exception as e: + self.status_msg = f"Live refresh failed: {e}" + + elif ch == ord("?"): + self.status_msg = "SPACE=paint | e=entity extract (immediate preview) | m=NMI brief (real note on A) | A=commit+APPLY (1 confirm) | s/a/c/p/t/C tags | R=refresh | q quit" + + elif ch == ord("x"): # clear pending on current + issue = self._current_issue() + if issue: + iid = int(issue["id"]) + if iid in self.pending: + del self.pending[iid] + self.status_msg = f"Cleared pending for #{iid}" + + # Simple numeric priority hotkeys example (optional) + elif ch == ord("1"): + self.set_priority_numeric(1) + elif ch == ord("2"): + self.set_priority_numeric(2) + elif ch == ord("3"): + self.set_priority_numeric(3) + + return 0 + + +def run_tui( + issues: list[dict[str, Any]], + *, + cache_dir: Path, + project: str, + base_url: str, + allow_prod_writes: bool = False, + auto_apply: bool = False, +) -> int: + tui = RedmineTUI( + issues, + cache_dir=cache_dir, + project=project, + base_url=base_url, + allow_prod_writes=allow_prod_writes, + auto_apply=auto_apply, + ) + return curses.wrapper(tui.run) + + +# CLI glue (called from redmine_tickets.py) +def add_tui_parser(subparsers: argparse._SubParsersAction) -> None: + tui_parser = subparsers.add_parser( + "tui", + help="Curses TUI: set template once, paint with SPACE, A=commit+auto-apply (one confirm for batch).", + ) + + # Declare parent-level globals here too so they are accepted after the subcommand verb. + # This lets users do: rtl tui --allow-prod-writes ... + # (The main parser also defines them for "before verb" placement.) + tui_parser.add_argument("--base-url", default=None, help=argparse.SUPPRESS) + tui_parser.add_argument("--project", default=None, help=argparse.SUPPRESS) + tui_parser.add_argument("--cache-dir", type=str, default=None, help=argparse.SUPPRESS) + tui_parser.add_argument( + "--allow-prod-writes", + action="store_true", + help="Allow write/apply operations against non-LAN hosts (e.g. rm.ldrprep.com).", + ) + tui_parser.add_argument("--config", default=None, help=argparse.SUPPRESS) + + tui_parser.add_argument("--where", action="append", default=[], help="Local property filter (repeatable AND).") + tui_parser.add_argument("--control-tag", help="Local control-tag filter.") + tui_parser.add_argument("--status", help="Status name filter.") + tui_parser.add_argument("--max-tickets", type=int, default=100) + tui_parser.add_argument("--order", choices=["newest-updated-first", "oldest-updated-first"], default="newest-updated-first") + tui_parser.add_argument("--open-only", action="store_true", help="Exclude closed-ish statuses.") + tui_parser.add_argument("--has-attachments", action="store_true") + tui_parser.add_argument( + "--apply", + action="store_true", + help="DANGEROUS: attempt direct writes instead of drafting (requires --allow-prod-writes for prod).", + ) + tui_parser.add_argument("--impersonate-user", default="rebot") + + +def main_cli(args: argparse.Namespace, cache: "TicketCache") -> int: # type: ignore + # `curses.wrapper()` fails at cbreak/nocbreak when launched through a pipe, + # captured runner, or WebUI. Refuse before touching cache/network state and + # make the failure understandable instead of silently returning to a prompt. + if not os.isatty(0) or not os.isatty(1): + print( + "ERROR: curses TUI requires an interactive terminal (stdin and stdout must be TTYs).", + file=sys.stderr, + ) + print( + "Run `rtl tui` directly in an onyx shell, not through a pipe, capture, or WebUI runner.", + file=sys.stderr, + ) + return 1 + + # Robust normalization so --allow-prod-writes works whether before or after "tui" + if getattr(args, "cache_dir", None) in (None, ""): + args.cache_dir = getattr(cache, "cache_dir", None) or Path(".cache/redmine_tickets") + if getattr(args, "project", None) in (None, ""): + args.project = getattr(cache, "project", None) or "customer-service" + if getattr(args, "base_url", None) in (None, ""): + import os as _os + args.base_url = _os.environ.get("REDMINE_BASE_URL") or "https://rm.ldrprep.com" + + # Make write flags reliable regardless of position on command line + effective_allow = bool( + getattr(args, "allow_prod_writes", False) + or getattr(args, "apply", False) + or os.environ.get("REDMINE_ALLOW_PROD_WRITES", "").lower() in ("1", "true", "yes") + ) + args.allow_prod_writes = effective_allow + if not getattr(args, "apply", False): + args.apply = effective_allow + + # Late import: the top-level try: from redmine_tickets import ... can fail + # due to circular import while redmine_tickets is still executing its module body. + try: + from redmine_tickets import select_cached_issues_for_cli as _select_cached_issues + except Exception: + _select_cached_issues = None + + if _select_cached_issues is None: + # Fallback: import the module again now that it should be fully loaded + import redmine_tickets as _rt + _select_cached_issues = _rt.select_cached_issues_for_cli + + # Robust: import the already-loaded parent module to dodge circular import timing + import redmine_tickets as _rt + matches = _rt.select_cached_issues_for_cli(cache, args) + if not matches: + print("No matching tickets for TUI.", file=sys.stderr) + return 0 + + # Ensure some meta is present + if load_meta_lists: + try: + lists = load_meta_lists(Path(args.cache_dir), args.project) + if not any(lists.values()): + print("Meta cache looks empty — consider running meta-refresh first for best picker UX.", file=sys.stderr) + except Exception: + pass + + return run_tui( + matches, + cache_dir=Path(args.cache_dir), + project=args.project, + base_url=args.base_url, + allow_prod_writes=bool(args.allow_prod_writes), + auto_apply=bool(getattr(args, "apply", False)), + ) + + +if __name__ == "__main__": + # Standalone test (needs issues list) + print("Run via: ./redmine_tickets.py tui ... or ./bin/rtl tui ...") + sys.exit(0) diff --git a/redmine_ticket_tui_modern.py b/redmine_ticket_tui_modern.py new file mode 100644 index 0000000..544cc8e --- /dev/null +++ b/redmine_ticket_tui_modern.py @@ -0,0 +1,276 @@ +#!/usr/bin/env python3 +""" +Modern TUI for redmine_tickets using prompt_toolkit + rich. + +Goal: btop / mynav / htop-style experience while keeping the same +workflow: +- Set a template once (status, assignee, category, priority, private note, tags, etc.) +- Paint with SPACE/Enter +- A = one-confirm batch queue + auto-apply (with cache refresh) +- R = live refresh of current ticket +- Good columns (contact, tags, updated, priority, related, etc.) +- Proper split panes, mouse, resize, theming +- Reuses existing domain logic (drafts, apply, cache refresh, select, meta) + +This is the future implementation path. The curses version in redmine_ticket_tui.py +remains the current stable tool while we build this out. + +Run (once wired): + ./bin/rtl --allow-prod-writes tui --modern ... +""" + +from __future__ import annotations + +import argparse +import os +import sys +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Optional + +# --- Rich (rendering) --- +try: + from rich.console import Console + from rich.panel import Panel + from rich.table import Table + from rich.text import Text + from rich.layout import Layout + from rich.live import Live + from rich.box import ROUNDED +except ImportError: + print("rich is required for the modern TUI. (it should already be installed)", file=sys.stderr) + raise + +# --- prompt_toolkit (input + app) --- +try: + from prompt_toolkit import Application + from prompt_toolkit.key_binding import KeyBindings + from prompt_toolkit.layout import Layout as PTKLayout, Window, HSplit, VSplit + from prompt_toolkit.layout.controls import FormattedTextControl + from prompt_toolkit.layout.containers import Window + from prompt_toolkit.widgets import TextArea, Frame + from prompt_toolkit.application import get_app +except ImportError: + print("prompt_toolkit is required for the modern TUI.", file=sys.stderr) + raise + +# --- Reuse existing domain code where possible --- +try: + from redmine_tickets import ( + TicketCache, + select_cached_issues_for_cli, + refresh_cached_issue, + ) +except Exception: + TicketCache = None # type: ignore + select_cached_issues_for_cli = None # type: ignore + refresh_cached_issue = None # type: ignore + +try: + from redmine_ticket_drafts import ( + add_draft, + load_draft, + mark_draft_applied, + mark_draft_error, + assert_write_target_allowed, + KIND_FIELDS, + KIND_INTERNAL_NOTE, + ) +except Exception: + add_draft = None # type: ignore + # ... (fallbacks handled at runtime) + +try: + from redmine_ticket_meta import load_meta_lists +except Exception: + load_meta_lists = None # type: ignore + +try: + from redmine_ticket_control_tags import add_tag as control_tag_add +except Exception: + control_tag_add = None # type: ignore + + +@dataclass +class Template: + status_id: Optional[int] = None + status_name: Optional[str] = None + assigned_to_id: Optional[int] = None + assigned_to_name: Optional[str] = None + category_id: Optional[int] = None + category_name: Optional[str] = None + priority_id: Optional[int] = None + priority_name: Optional[str] = None + project_id: Optional[int] = None + project_name: Optional[str] = None + local_control_tag: Optional[str] = None + private_note: Optional[str] = None + private: bool = True + + +@dataclass +class PendingChanges: + issue_id: int + fields: dict[str, Any] = field(default_factory=dict) + private_note: Optional[str] = None + private: bool = True + control_tags: list[str] = field(default_factory=list) + contact_tags: list[str] = field(default_factory=list) + entity_extract: bool = False + nmi_brief: bool = False + contact_id: Optional[int] = None + + +class ModernRedmineTUI: + """Prompt-toolkit + Rich based TUI. + + This is intentionally a work-in-progress skeleton that mirrors the + established workflow while giving us proper panes, mouse, live updates, + and nice visuals. + """ + + def __init__( + self, + issues: list[dict[str, Any]], + *, + cache_dir: Path, + project: str, + base_url: str, + allow_prod_writes: bool = False, + auto_apply: bool = False, + ): + self.issues = issues + self.cache_dir = cache_dir + self.project = project + self.base_url = base_url + self.allow_prod_writes = allow_prod_writes + self.auto_apply = auto_apply + + self.template = Template() + self.pending: dict[int, PendingChanges] = {} + self.filter_text = "" + self.filtered_indices: list[int] = list(range(len(issues))) + self.current_idx = 0 + self.status_msg = "Ready. SPACE = paint template. A = commit+apply (1 confirm). R = live refresh." + + self.console = Console() + self._meta = {} + + def _current_issue(self) -> Optional[dict[str, Any]]: + if not self.filtered_indices: + return None + idx = min(self.current_idx, len(self.filtered_indices) - 1) + real = self.filtered_indices[idx] + return self.issues[real] + + def apply_template_to_current(self) -> None: + issue = self._current_issue() + if not issue: + return + iid = int(issue["id"]) + ch = self.pending.get(iid) or PendingChanges(issue_id=iid) + + if self.template.status_id is not None: + ch.fields["status_id"] = self.template.status_id + if self.template.assigned_to_id is not None: + ch.fields["assigned_to_id"] = self.template.assigned_to_id + if self.template.category_id is not None: + ch.fields["category_id"] = self.template.category_id + if self.template.priority_id is not None: + ch.fields["priority_id"] = self.template.priority_id + if self.template.project_id is not None: + ch.fields["project_id"] = self.template.project_id + if self.template.local_control_tag: + ch.control_tags = [self.template.local_control_tag] + if self.template.private_note: + ch.private_note = self.template.private_note + ch.private = self.template.private + + self.pending[iid] = ch + self.status_msg = f"Painted template on #{iid}" + + def _refresh_list_view(self) -> None: + # In a real ptk+rich version we would trigger a Live update here. + pass + + def run(self) -> int: + """Very early skeleton. Real keyboard + layout comes next.""" + self.console.print( + Panel.fit( + "[bold]Modern Redmine TUI (prompt_toolkit + rich)[/bold]\n" + "Skeleton active. Full keyboard layout + panes coming in next iterations.\n" + "For now use the stable curses TUI (remove --modern when wired).\n\n" + "Current tickets in view: " + str(len(self.issues)) + ) + ) + return 0 + + +# --- Wiring helpers (will be called from redmine_tickets.py later) --- + +def add_modern_tui_parser(subparsers: argparse._SubParsersAction) -> None: + p = subparsers.add_parser( + "tui-modern", + help="EXPERIMENTAL: prompt_toolkit + rich TUI (btop-style).", + ) + # Reuse the same global flags + p.add_argument("--allow-prod-writes", action="store_true", help=argparse.SUPPRESS) + p.add_argument("--apply", action="store_true", help=argparse.SUPPRESS) + p.add_argument("--where", action="append", default=[], help="Local property filter.") + p.add_argument("--max-tickets", type=int, default=100) + p.add_argument("--open-only", action="store_true") + p.add_argument("--impersonate-user", default="rebot") + + +def run_modern_tui( + issues: list[dict[str, Any]], + *, + cache_dir: Path, + project: str, + base_url: str, + allow_prod_writes: bool = False, + auto_apply: bool = False, +) -> int: + tui = ModernRedmineTUI( + issues, + cache_dir=cache_dir, + project=project, + base_url=base_url, + allow_prod_writes=allow_prod_writes, + auto_apply=auto_apply, + ) + return tui.run() + + +def main_cli(args: argparse.Namespace, cache: "TicketCache") -> int: # type: ignore + if getattr(args, "cache_dir", None) in (None, ""): + args.cache_dir = getattr(cache, "cache_dir", None) or Path(".cache/redmine_tickets") + if getattr(args, "project", None) in (None, ""): + args.project = getattr(cache, "project", None) or "customer-service" + if getattr(args, "base_url", None) in (None, ""): + args.base_url = os.environ.get("REDMINE_BASE_URL", "https://rm.ldrprep.com") + + effective_allow = bool( + getattr(args, "allow_prod_writes", False) + or getattr(args, "apply", False) + or os.environ.get("REDMINE_ALLOW_PROD_WRITES", "").lower() in ("1", "true", "yes") + ) + + matches = select_cached_issues_for_cli(cache, args) if select_cached_issues_for_cli else [] + if not matches: + print("No matching tickets for modern TUI.", file=sys.stderr) + return 0 + + return run_modern_tui( + matches, + cache_dir=Path(args.cache_dir), + project=args.project, + base_url=args.base_url, + allow_prod_writes=effective_allow, + auto_apply=getattr(args, "apply", False), + ) + + +if __name__ == "__main__": + print("Run via the main redmine_tickets.py / bin/rtl once the subcommand is registered.") + sys.exit(0) diff --git a/redmine_ticket_view.py b/redmine_ticket_view.py new file mode 100644 index 0000000..6067613 --- /dev/null +++ b/redmine_ticket_view.py @@ -0,0 +1,198 @@ +#!/usr/bin/env python3 +"""Markdown rendering + pager helpers for cached Redmine tickets.""" + +from __future__ import annotations + +import os +import shutil +import subprocess +from pathlib import Path +from typing import Any + + +def _nested_name(issue: dict[str, Any], key: str) -> str: + value = issue.get(key) + if isinstance(value, dict): + name = value.get("name") + return str(name) if name is not None else "" + return "" if value is None else str(value) + + +def _helpdesk(issue: dict[str, Any]) -> dict[str, Any]: + hd = issue.get("helpdesk_ticket") or {} + return hd if isinstance(hd, dict) else {} + + +def _contact(issue: dict[str, Any]) -> dict[str, Any]: + contact = _helpdesk(issue).get("contact") or {} + return contact if isinstance(contact, dict) else {} + + +def render_ticket_markdown(issue: dict[str, Any]) -> str: + issue_id = issue.get("id") + subject = str(issue.get("subject") or "").strip() + status = _nested_name(issue, "status") + project = _nested_name(issue, "project") + category = _nested_name(issue, "category") + assigned = _nested_name(issue, "assigned_to") or "(unassigned)" + author = _nested_name(issue, "author") + priority = _nested_name(issue, "priority") + created = issue.get("created_on") or "" + updated = issue.get("updated_on") or "" + hd = _helpdesk(issue) + contact = _contact(issue) + from_addr = hd.get("from_address") or contact.get("email") or "" + contact_id = contact.get("id") + company = contact.get("company") or "" + contact_name = contact.get("name") or "" + description = str(issue.get("description") or "").strip() + redmine_url = f"https://rm.ldrprep.com/issues/{issue_id}" if issue_id else "" + + lines = [ + f"# #{issue_id} — {subject}", + "", + "## Meta", + "", + f"- **Project:** {project}", + f"- **Status:** {status}", + f"- **Priority:** {priority}", + f"- **Category:** {category or '(none)'}", + f"- **Assignee:** {assigned}", + f"- **Author:** {author}", + f"- **Created:** {created}", + f"- **Updated:** {updated}", + f"- **Redmine:** {redmine_url}", + "", + "## Helpdesk / contact", + "", + f"- **From:** {from_addr}", + f"- **Contact ID:** {contact_id if contact_id is not None else '(none)'}", + f"- **Contact name:** {contact_name or '(none)'}", + f"- **Company:** {company or '(none)'}", + "", + "## Detected entities (heuristic)", + "", + ] + try: + from redmine_ticket_entities import extract_entities_from_issue, format_entities_line + + ents = extract_entities_from_issue(issue) + if ents: + for e in ents: + conf = e.get("confidence") or "high" + lines.append(f"- **{e.get('type')}:** `{e.get('value')}` ({conf})") + else: + lines.append("_(none)_") + except Exception: + lines.append("_(entity extractor unavailable)_") + lines.extend( + [ + "", + "## Description", + "", + description or "_(empty)_", + "", + "## Journals", + "", + ] + ) + + journals = issue.get("journals") or [] + if not journals: + lines.append("_(no journals)_") + else: + for journal in journals: + if not isinstance(journal, dict): + continue + jid = journal.get("id") + jcreated = journal.get("created_on") or "" + user = journal.get("user") or {} + uname = user.get("name") if isinstance(user, dict) else "" + notes = str(journal.get("notes") or "").strip() + details = journal.get("details") or [] + lines.append(f"### Journal #{jid} — {jcreated} — {uname}") + lines.append("") + if details: + for detail in details: + if not isinstance(detail, dict): + continue + lines.append( + f"- change: {detail.get('property')}/{detail.get('name')}: " + f"{detail.get('old_value')!r} → {detail.get('new_value')!r}" + ) + lines.append("") + if notes: + lines.append(notes) + else: + lines.append("_(no notes)_") + lines.append("") + + lines.append("---") + lines.append(f"_Rendered from cache; source updated_on={updated}_") + lines.append("") + return "\n".join(lines) + + +def markdown_cache_path(cache_dir: Path, project: str, issue_id: int | str) -> Path: + return Path(cache_dir) / project / "markdown" / f"{issue_id}.md" + + +def write_ticket_markdown( + issue: dict[str, Any], + cache_dir: Path, + project: str, + *, + force: bool = False, +) -> Path: + """Write markdown cache if missing or ticket updated_on changed (via sidecar stamp).""" + issue_id = issue["id"] + path = markdown_cache_path(cache_dir, project, issue_id) + stamp_path = path.with_suffix(".md.updated_on") + updated_on = str(issue.get("updated_on") or "") + path.parent.mkdir(parents=True, exist_ok=True) + if ( + not force + and path.is_file() + and stamp_path.is_file() + and stamp_path.read_text(encoding="utf-8").strip() == updated_on + ): + return path + path.write_text(render_ticket_markdown(issue), encoding="utf-8") + stamp_path.write_text(updated_on + "\n", encoding="utf-8") + return path + + +def resolve_pager(preferred: str = "auto") -> list[str]: + """Return argv prefix to display markdown (pager program + flags).""" + preferred = (preferred or "auto").lower() + if preferred == "cat": + return ["cat"] + if preferred in {"glow", "bat", "batcat"}: + path = shutil.which(preferred) + if not path: + raise FileNotFoundError(f"Pager {preferred!r} not found on PATH") + if preferred == "glow": + return [path] + return [path, "--language", "markdown", "--style", "plain"] + # auto + for name, args in ( + ("glow", []), + ("bat", ["--language", "markdown", "--style", "plain"]), + ("batcat", ["--language", "markdown", "--style", "plain"]), + ): + path = shutil.which(name) + if path: + return [path, *args] + return ["cat"] + + +def display_markdown_file(path: Path, pager: str = "auto") -> int: + cmd = resolve_pager(pager) + [str(path)] + # Prefer streaming to user terminal + env = os.environ.copy() + try: + completed = subprocess.run(cmd, check=False, env=env) + return int(completed.returncode) + except FileNotFoundError: + print(path.read_text(encoding="utf-8")) + return 0 diff --git a/redmine_tickets.py b/redmine_tickets.py new file mode 100755 index 0000000..1f406c3 --- /dev/null +++ b/redmine_tickets.py @@ -0,0 +1,4928 @@ +#!/usr/bin/env python3 +"""Incremental Redmine issue sync/search foundation. + +This is the ticket-centric companion to redmine_contacts.py. It caches enriched +Redmine issue payloads locally using the standard Redmine issue API with the +local Helpdesk plugin's `include=helpdesk` support. +""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import sys +import time +from dataclasses import asdict, dataclass +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Any, Callable, Iterable + +from redmine_contacts import ( + DEFAULT_BASE_URL, + DEFAULT_PROJECT, + RedmineClient, + RedmineError, + extract_tag_list, + merge_tags, +) + +try: + from redmine_ticket_query import QueryError, filter_by_query +except ImportError: # pragma: no cover + QueryError = ValueError # type: ignore[misc, assignment] + filter_by_query = None # type: ignore[assignment] + +try: + from redmine_ticket_view import display_markdown_file, render_ticket_markdown, write_ticket_markdown +except ImportError: # pragma: no cover + display_markdown_file = None # type: ignore[assignment] + render_ticket_markdown = None # type: ignore[assignment] + write_ticket_markdown = None # type: ignore[assignment] + +try: + from redmine_ticket_filters import ( + DEFAULT_SAVED_FILTERS_DIR, + expand_where_list, + load_queue_definitions, + materialize_queue, + ) +except ImportError: # pragma: no cover + DEFAULT_SAVED_FILTERS_DIR = Path("saved-filters") # type: ignore[misc, assignment] + expand_where_list = None # type: ignore[assignment] + load_queue_definitions = None # type: ignore[assignment] + materialize_queue = None # type: ignore[assignment] + +try: + from redmine_ticket_drafts import ( + KIND_CONTACT_FIELDS, + KIND_CUSTOMER_EMAIL, + KIND_FIELDS, + KIND_INTERNAL_NOTE, + add_draft, + assert_write_target_allowed, + cancel_all_pending, + cancel_draft, + kind_label, + list_drafts, + load_draft, + mark_draft_applied, + mark_draft_error, + normalize_kind, + ) +except ImportError: # pragma: no cover + KIND_CONTACT_FIELDS = "contact_fields" # type: ignore[misc, assignment] + KIND_CUSTOMER_EMAIL = "customer_email" # type: ignore[misc, assignment] + KIND_FIELDS = "fields" # type: ignore[misc, assignment] + KIND_INTERNAL_NOTE = "internal_note" # type: ignore[misc, assignment] + add_draft = None # type: ignore[assignment] + assert_write_target_allowed = None # type: ignore[assignment] + cancel_all_pending = None # type: ignore[assignment] + cancel_draft = None # type: ignore[assignment] + kind_label = None # type: ignore[assignment] + list_drafts = None # type: ignore[assignment] + load_draft = None # type: ignore[assignment] + mark_draft_applied = None # type: ignore[assignment] + mark_draft_error = None # type: ignore[assignment] + normalize_kind = None # type: ignore[assignment] + +try: + from redmine_ticket_control_tags import ( + add_tag as control_tag_add, + annotate_issues_with_control_tags, + filter_issues_by_control_tag, + list_tags as control_tag_list, + remove_tag as control_tag_remove, + tags_for_issue, + tickets_for_tag, + wipe_all as control_tag_wipe, + ) +except ImportError: # pragma: no cover + control_tag_add = None # type: ignore[assignment] + annotate_issues_with_control_tags = None # type: ignore[assignment] + filter_issues_by_control_tag = None # type: ignore[assignment] + control_tag_list = None # type: ignore[assignment] + control_tag_remove = None # type: ignore[assignment] + tags_for_issue = None # type: ignore[assignment] + tickets_for_tag = None # type: ignore[assignment] + control_tag_wipe = None # type: ignore[assignment] + +# Lazy import to avoid circular dependency with redmine_triage +add_triage_parser = None +def _get_add_triage_parser(): + global add_triage_parser + if add_triage_parser is None: + try: + from redmine_triage import add_triage_parser as _atp + add_triage_parser = _atp + except Exception: + add_triage_parser = None + return add_triage_parser + +try: + from redmine_ticket_meta import refresh_meta_cache, resolve_field_names +except ImportError: # pragma: no cover + refresh_meta_cache = None # type: ignore[assignment] + resolve_field_names = None # type: ignore[assignment] + +try: + from redmine_ticket_contacts import ( + ContactCache, + contact_display_name, + contact_id_from_issue, + filter_contacts, + primary_email, + sync_contacts, + ) +except ImportError: # pragma: no cover + ContactCache = None # type: ignore[misc, assignment] + contact_display_name = None # type: ignore[assignment] + contact_id_from_issue = None # type: ignore[assignment] + filter_contacts = None # type: ignore[assignment] + primary_email = None # type: ignore[assignment] + sync_contacts = None # type: ignore[assignment] + +try: + from redmine_ticket_attachments import ( + attachment_dir, + attachments_from_issue, + download_attachment_bytes, + download_url_for_attachment, + pick_attachment, + safe_filename, + upload_file_token, + ) +except ImportError: # pragma: no cover + attachment_dir = None # type: ignore[assignment] + attachments_from_issue = None # type: ignore[assignment] + download_attachment_bytes = None # type: ignore[assignment] + download_url_for_attachment = None # type: ignore[assignment] + pick_attachment = None # type: ignore[assignment] + safe_filename = None # type: ignore[assignment] + upload_file_token = None # type: ignore[assignment] + +try: + from redmine_ticket_entities import ( + extract_entities_from_issue, + format_entities_line, + ) + from redmine_ticket_nmi import brief_for_issue, format_private_note +except ImportError: # pragma: no cover + extract_entities_from_issue = None # type: ignore[assignment] + format_entities_line = None # type: ignore[assignment] + brief_for_issue = None # type: ignore[assignment] + format_private_note = None # type: ignore[assignment] + +try: + from redmine_ticket_control_export import export_control_tag_jsonl +except ImportError: # pragma: no cover + export_control_tag_jsonl = None # type: ignore[assignment] + +try: + from redmine_ticket_browse import announce_draft, prompt_multiline, run_browse_loop +except ImportError: # pragma: no cover + announce_draft = None # type: ignore[assignment] + prompt_multiline = None # type: ignore[assignment] + run_browse_loop = None # type: ignore[assignment] + +try: + from redmine_ticket_tui import add_tui_parser, main_cli as run_tui_cli +except ImportError: # pragma: no cover + add_tui_parser = None # type: ignore[assignment] + run_tui_cli = None # type: ignore[assignment] + +try: + from redmine_ticket_textual_tui import ( + add_textual_tui_parser, + run_textual_tui_cli, + ) +except ImportError: # pragma: no cover + add_textual_tui_parser = None # type: ignore[assignment] + run_textual_tui_cli = None # type: ignore[assignment] + +try: + from redmine_ticket_ocr import ( + attachment_is_ocr_candidate, + build_ocr_note, + extract_text_from_file, + looks_like_generic_scan_subject, + suggest_subject_from_text, + ) +except ImportError: # pragma: no cover + attachment_is_ocr_candidate = None # type: ignore[assignment] + build_ocr_note = None # type: ignore[assignment] + extract_text_from_file = None # type: ignore[assignment] + looks_like_generic_scan_subject = None # type: ignore[assignment] + suggest_subject_from_text = None # type: ignore[assignment] + +# Lazy import to avoid circular dependency with redmine_triage +add_triage_parser = None +def _get_add_triage_parser(): + global add_triage_parser + if add_triage_parser is None: + try: + from redmine_triage import add_triage_parser as _atp + add_triage_parser = _atp + except Exception: + add_triage_parser = None + return add_triage_parser + +try: + from redmine_ticket_meta import fetch_project_categories, load_meta_lists +except ImportError: # pragma: no cover + fetch_project_categories = None # type: ignore[assignment] + load_meta_lists = None # type: ignore[assignment] + +DEFAULT_CACHE_DIR = Path(".cache/redmine_tickets") +ORDER_CHOICES = ( + "as-is", + "oldest-updated-first", + "newest-updated-first", + "id-asc", + "id-desc", +) +SENDER_SPAM_HINTS = re.compile( + r"\b(no-?reply|noreply|deals?|promotions?|marketing|newsletter|offers?|sales?)\b", + re.IGNORECASE, +) +PROMO_LANGUAGE = re.compile( + r"\b(discount|coupon|deal|promo|sale|offer|free|save|% off|limited time|up to \d+% off)\b", + re.IGNORECASE, +) +SALES_PITCH_LANGUAGE = re.compile( + r"\b(commercial cleaning|janitorial|cleaning services|we would like to offer|trying to reach|lead generation|marketing services|advertis(?:e|ing)|staffing services|outsourcing|managed services)\b", + re.IGNORECASE, +) +AUTOMATED_OPERATIONAL = re.compile( + r"\b(fba inventory|unfulfillable fba|inventory removal|listings api|amazon seller|payment status|shipment status|tracking update|invoice|return notification|ereceipt|e-?receipt|money order|undelivered mail|returned to sender|message could not be delivered|delivery status notification|critical security alert|security alert|saved passwords|passwords were found|google account|call transcription|voicemail|new message in mailbox|message transcription|voip\.ms|voipinterface)\b", + re.IGNORECASE, +) +INBOUND_BUSINESS_INQUIRY = re.compile( + r"\b(we(?:'| a)?re interested in purchasing|interested in purchasing|do you have any available|already working with a buyer|still have the capacity to support|workflow we previously discussed|ready to activate|returns workflow we previously discussed|do you process|rate sheet|capabilities and pricing)\b", + re.IGNORECASE, +) +VAULT_ONYX_PREFIX = "https://onyx.int.ldrprep.com/projects/obsidian-ldr/" +ANALYSIS_MODES = ("deterministic", "llm") +# ============================================================================= +# Known Spam Profile System (Fast Path) +# ============================================================================= + +def load_known_spam_profiles(path: Path) -> dict[str, Any]: + """Load a known-spam profile JSON file.""" + with path.open("r", encoding="utf-8") as handle: + data = json.load(handle) + data.setdefault("universal_safety_gates", {}) + data.setdefault("profiles", []) + return data + + +def _compile_profile_regexes(profile: dict[str, Any]) -> dict[str, list[re.Pattern]]: + """Compile sender and subject regexes for a single profile.""" + compiled = {"sender": [], "subject": []} + for pattern in profile.get("sender_regexes", []): + compiled["sender"].append(re.compile(pattern, re.IGNORECASE)) + for pattern in profile.get("subject_regexes", []): + compiled["subject"].append(re.compile(pattern, re.IGNORECASE)) + return compiled + + +def match_known_spam_profile(issue: dict[str, Any], profile: dict[str, Any], compiled: dict[str, list[re.Pattern]]) -> bool: + """Return True if the issue matches the given profile.""" + from_addr = ticket_from_address(issue) + subject = str(issue.get("subject") or "") + + sender_match = any(p.search(from_addr) for p in compiled["sender"]) + subject_match = any(p.search(subject) for p in compiled["subject"]) + + if compiled["sender"] and compiled["subject"]: + return sender_match and subject_match + return sender_match or subject_match + + +def get_known_spam_matches(issue: dict[str, Any], profiles: list[dict[str, Any]]) -> list[dict[str, Any]]: + """Return list of matching profiles for an issue.""" + matches = [] + for profile in profiles: + compiled = _compile_profile_regexes(profile) + if match_known_spam_profile(issue, profile, compiled): + matches.append(profile) + return matches + + +def ticket_was_reopened_after_close(issue: dict[str, Any]) -> bool: + """Check journals for evidence the ticket was closed then reopened by a new message.""" + for journal in issue.get("journals", []) or []: + details = journal.get("details") or [] + for detail in details: + if detail.get("property") == "attr" and detail.get("name") == "status_id": + old = str(detail.get("old_value") or "") + new = str(detail.get("new_value") or "") + if old in ("5", "Closed", "Resolved") and new not in ("5", "Closed", "Resolved"): + return True + return False + + +def should_skip_known_spam_close(issue: dict[str, Any], client: Any, safety_gates: dict[str, Any]) -> tuple[bool, str]: + """Return (should_skip, reason).""" + status_name = (nested_name(issue, "status") or "").lower() + if status_name in ("closed", "resolved"): + return True, "already_closed" + + contact_id = ticket_contact_id(issue) + if contact_id: + contact = client.get_contact(contact_id) + tag_list = [t.lower() for t in extract_tag_list(contact)] + for forbidden in safety_gates.get("skip_contact_tags", ["customer", "maybe-customer"]): + if forbidden.lower() in tag_list: + return True, f"contact_has_{forbidden.lower()}_tag" + + if ticket_was_reopened_after_close(issue): + return True, "reopened_after_close" + + return False, "" + + +def run_known_spam_scan( + cache: TicketCache, + profiles_path: Path, + max_tickets: int | None = None, +) -> list[dict[str, Any]]: + """Scan cached issues and return matches (dry-run).""" + profile_data = load_known_spam_profiles(profiles_path) + profiles = profile_data.get("profiles", []) + + results = [] + issues = list(cache.iter_cached_issues()) + if max_tickets: + issues = issues[:max_tickets] + + for issue in issues: + matches = get_known_spam_matches(issue, profiles) + if matches: + results.append({ + "issue_id": issue.get("id"), + "subject": issue.get("subject"), + "from": ticket_from_address(issue), + "contact_id": ticket_contact_id(issue), + "matched_profiles": [p["name"] for p in matches], + }) + return results + + +def run_known_spam_close( + client: Any, + cache: TicketCache, + profiles_path: Path, + close_status_id: int = 5, + apply: bool = False, + max_tickets: int | None = None, +) -> list[dict[str, Any]]: + """Close issues matching known spam profiles.""" + profile_data = load_known_spam_profiles(profiles_path) + profiles = profile_data.get("profiles", []) + safety_gates = profile_data.get("universal_safety_gates", {}) + impersonate = safety_gates.get("write_impersonate_user", "rebot") + + plans = [] + issues = list(cache.iter_cached_issues()) + if max_tickets: + issues = issues[:max_tickets] + + for issue in issues: + matches = get_known_spam_matches(issue, profiles) + if not matches: + continue + + issue_id = issue.get("id") + if issue_id is None: + continue + + skip, reason = should_skip_known_spam_close(issue, client, safety_gates) + if skip: + if apply: + cache.record_action(issue_id, "known-spam-close", f"skipped:{reason}") + continue + + if cache.has_action(issue_id, "known-spam-close"): + continue + + payload = { + "status_id": close_status_id, + "notes": f"Closed as known promotional spam. Profiles: {', '.join(p['name'] for p in matches)}", + } + + plan = { + "issue_id": issue_id, + "subject": issue.get("subject"), + "from": ticket_from_address(issue), + "matched_profiles": [p["name"] for p in matches], + "payload": payload, + } + plans.append(plan) + + if apply: + client.update_issue(int(issue_id), payload, impersonate_user=impersonate) + cache.record_action(issue_id, "known-spam-close", "closed", {"profiles": [p["name"] for p in matches]}) + + return plans + + +# ============================================================================= +# End Known Spam Profile System +# ============================================================================= + + + +INBOUND_BUSINESS_INQUIRY = re.compile( + r"\b(we(?:'| a)?re interested in purchasing|interested in purchasing|do you have any available|already working with a buyer|still have the capacity to support|workflow we previously discussed|ready to activate|returns workflow we previously discussed|do you process|rate sheet|capabilities and pricing)\b", + re.IGNORECASE, +) +VAULT_ONYX_PREFIX = "https://onyx.int.ldrprep.com/projects/obsidian-ldr/" + + +def slugify_note(value: str) -> str: + text = value.strip().lower() + text = re.sub(r"[’']", "", text) + text = re.sub(r"[^a-z0-9]+", "-", text) + return text.strip("-") or "untitled" + + +def first_sentence(value: str, max_len: int = 240) -> str: + text = re.sub(r"\s+", " ", value or "").strip() + if not text: + return "" + match = re.search(r"(?<=[.!?])\s+", text) + sentence = text[: match.start()] if match else text + if len(sentence) > max_len: + sentence = sentence[: max_len - 1].rstrip() + "…" + return sentence + + +def parse_dt(value: str | None) -> datetime | None: + if not value: + return None + text = value.strip() + if text.endswith("Z"): + text = text[:-1] + "+00:00" + if "T" not in text and len(text) == 10: + text = text + "T00:00:00+00:00" + dt = datetime.fromisoformat(text) + if dt.tzinfo is None: + dt = dt.replace(tzinfo=timezone.utc) + return dt + + +def now_iso() -> str: + return datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z") + + +_RELATIVE_TIME_RE = re.compile( + r"^(?P\d+)\s*(?Pd|day|days|h|hour|hours|m|min|mins|minute|minutes|w|week|weeks)\s*(?Pago)?$", + re.IGNORECASE, +) +_REMOTE_OPERATOR_PREFIXES = (">=", "<=", "><", ">", "<", "=") + + +def format_redmine_time(dt: datetime, *, date_only: bool = False) -> str: + """Format a datetime for Redmine REST date filters.""" + utc = dt.astimezone(timezone.utc).replace(microsecond=0) + if date_only: + return utc.date().isoformat() + return utc.isoformat().replace("+00:00", "Z") + + +def coerce_time_value(value: str, *, now: datetime | None = None) -> tuple[datetime, bool]: + """Parse absolute or relative time into (datetime, is_date_only). + + Accepts: + - YYYY-MM-DD + - ISO datetime (with optional Z) + - relative: 30d, 2h, 15m, 1w, '2 days ago', '30 days ago' + """ + text = (value or "").strip() + if not text: + raise RedmineError("Empty time value.") + + relative = _RELATIVE_TIME_RE.fullmatch(text) + if relative: + amount = int(relative.group("n")) + unit = relative.group("unit").lower() + anchor = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) + if unit.startswith("w"): + delta = amount * 7 * 24 * 3600 + elif unit.startswith("d"): + delta = amount * 24 * 3600 + elif unit.startswith("h"): + delta = amount * 3600 + else: + delta = amount * 60 + return anchor - timedelta(seconds=delta), False + + # Pure calendar date → date-only Redmine filter (start-of-day semantics on server). + if re.fullmatch(r"\d{4}-\d{2}-\d{2}", text): + dt = parse_dt(text) + if dt is None: + raise RedmineError(f"Could not parse date: {value!r}") + return dt, True + + dt = parse_dt(text) + if dt is None: + raise RedmineError( + f"Could not parse time value {value!r}. " + "Use YYYY-MM-DD, ISO datetime, or relative forms like 30d / '2 days ago'." + ) + return dt, False + + +def build_remote_updated_on_value( + raw: str | None = None, + since: str | None = None, + before: str | None = None, + *, + now: datetime | None = None, +) -> str | None: + """Build Redmine `updated_on` API filter value. + + Raw form (official REST syntax) wins if provided: + >=2014-01-02T08:12:32Z + <=2012-03-07 + ><2012-03-01|2012-03-07 + + Convenience flags map to the same operators: + since only → >=... + before only → <=... + both → >= (common cron mistake). + if not raw_text.startswith(_REMOTE_OPERATOR_PREFIXES): + return f">={raw_text}" + return raw_text + + if not since_text and not before_text: + return None + + since_part = None + before_part = None + if since_text: + dt, date_only = coerce_time_value(since_text, now=now) + since_part = format_redmine_time(dt, date_only=date_only) + if before_text: + dt, date_only = coerce_time_value(before_text, now=now) + before_part = format_redmine_time(dt, date_only=date_only) + + if since_part and before_part: + return f"><{since_part}|{before_part}" + if since_part: + return f">={since_part}" + return f"<={before_part}" + + +def nested_name(issue: dict[str, Any], key: str) -> str | None: + value = issue.get(key) + if isinstance(value, dict): + name = value.get("name") + return str(name) if name is not None else None + if value is None: + return None + return str(value) + + +def ticket_contact(issue: dict[str, Any]) -> dict[str, Any]: + helpdesk = issue.get("helpdesk_ticket") or {} + contact = helpdesk.get("contact") or {} + return contact if isinstance(contact, dict) else {} + + +def ticket_from_address(issue: dict[str, Any]) -> str: + helpdesk = issue.get("helpdesk_ticket") or {} + return str(helpdesk.get("from_address") or ticket_contact(issue).get("email") or "") + + +def issue_text_for_decomposition(issue: dict[str, Any]) -> str: + parts = [str(issue.get("subject") or ""), str(issue.get("description") or "")] + for journal in issue.get("journals", []) or []: + notes = str(journal.get("notes") or "").strip() + if notes: + parts.append(notes) + return "\n".join(parts) + + +def clean_message_block(text: str) -> str: + lines: list[str] = [] + for raw_line in (text or "").replace("\r", "\n").split("\n"): + line = raw_line.strip() + if not line: + continue + if re.match(r"^(from|subject|to|date|phone):\s", line, re.IGNORECASE): + continue + if line.lower() == "message body:": + continue + if re.match(r"^-{5,}\s*forwarded message\s*-{5,}$", line, re.IGNORECASE): + continue + if line == "--" or line.startswith("-- "): + break + if line.startswith(">"): + continue + if "this e-mail was sent from a contact form" in line.lower(): + continue + if re.match(r"^on .+ wrote:$", line, re.IGNORECASE): + break + if line.startswith(" str: + """Return a compact source text suitable for decomposition prompts/plans.""" + parts: list[str] = [] + description = clean_message_block(str(issue.get("description") or "")) + if description: + parts.append(description) + for journal in issue.get("journals", []) or []: + notes = clean_message_block(str(journal.get("notes") or "")) + if notes: + user = (journal.get("user") or {}).get("name") if isinstance(journal.get("user"), dict) else None + prefix = f"{user}: " if user else "" + parts.append(prefix + notes) + return "\n".join(dict.fromkeys(parts)) + + +def extract_decomposition_analysis(issue: dict[str, Any], org_name: str, buckets: list[str], tags: list[str]) -> dict[str, Any]: + """Deterministic KarpathyWiki-style structured analysis for the first dry-run slice. + + This intentionally mirrors the plugin's source-analysis shape without calling an LLM yet. + """ + text = clean_ticket_text(issue) + text_l = text.lower() + entities = [ + { + "name": org_name, + "type": "organization", + "summary": f"Organization mentioned in Redmine #{issue.get('id')}.", + "mentions_in_source": [org_name], + "related_entities": [], + "related_concepts": [], + } + ] + concepts: list[dict[str, Any]] = [] + key_points: list[str] = [] + summary = first_sentence(text) or str(issue.get("subject") or "") + + if "gaylord" in text_l: + summary = f"{org_name} asked whether LDR has used Gaylord boxes available to purchase." + key_points.extend([ + "The sender wants to buy used Gaylord boxes from LDR, not sell LDR a generic service.", + "The inquiry may represent a buy-from-us or liquidation-adjacent opportunity.", + ]) + concepts.append({"name": "buy-from-us inquiry", "type": "term", "summary": "An inbound request to buy goods or assets from LDR."}) + if "fulfillment" in text_l or "returns workflow" in text_l or "fba returns" in text_l or "warehousing" in text_l: + summary = f"{org_name} raised a fulfillment/returns opportunity with LDR." + key_points.append("The ticket concerns warehousing, fulfillment, returns management, or FBA returns services.") + concepts.append({"name": "returns management", "type": "field", "summary": "Operational handling of returned goods and related customer workflows."}) + if "rate sheet" in text_l or "calculator" in text_l: + summary = f"{org_name} asked for a returns rate sheet after the pricing/calculator path did not answer their needs." + key_points.extend([ + "The contact reported that the linked calculator did not calculate what they needed.", + "The contact asked for a rate sheet, exposing a possible returns-pricing collateral gap.", + ]) + concepts.append({"name": "rate-sheet gap", "type": "term", "summary": "A sales-collateral gap where pricing must be explained ad hoc instead of through a reusable artifact."}) + + if not key_points and text: + key_points.append(first_sentence(text, 300)) + + # Deduplicate concepts by normalized name. + deduped_concepts: list[dict[str, Any]] = [] + seen: set[str] = set() + for concept in concepts: + key = slugify_note(str(concept.get("name", ""))) + if key and key not in seen: + seen.add(key) + deduped_concepts.append(concept) + + return { + "source_title": str(issue.get("subject") or ""), + "summary": summary, + "entities": entities, + "concepts": deduped_concepts, + "key_points": list(dict.fromkeys(key_points)), + "contradictions": [], + "related_pages": [], + "clean_text": text, + } + + +def issue_event_date(issue: dict[str, Any]) -> str: + journal_dates = [] + for journal in issue.get("journals", []) or []: + if str(journal.get("notes") or "").strip(): + dt = parse_dt(journal.get("created_on")) + if dt: + journal_dates.append(dt) + if journal_dates: + return max(journal_dates).date().isoformat() + dt = parse_dt(issue.get("created_on") or issue.get("updated_on")) or datetime.now(timezone.utc) + return dt.date().isoformat() + + +def infer_organization_name(issue: dict[str, Any]) -> str: + subject = str(issue.get("subject") or "") + sender = ticket_from_address(issue).lower() + text = issue_text_for_decomposition(issue).lower() + if "repackify" in subject.lower() or "repackify" in sender or "repackify" in text: + return "RePackify" + if "u2wear" in subject.lower() or "u2wear" in sender or "u2wear" in text or "elora" in text: + return "u2wear" + if "snugglepuppy" in sender or "snuggle puppy" in text or "snugglepuppy" in text: + return "Snuggle Puppy" + contact = ticket_contact(issue) + for key in ("company", "name", "email"): + value = str(contact.get(key) or "").strip() + if value: + if "@" in value: + value = value.split("@", 1)[1].split(".", 1)[0] + return value + return sender.split("@", 1)[-1].split(".", 1)[0] if sender else "Unknown Organization" + + +def classify_decomposition_buckets(issue: dict[str, Any]) -> tuple[list[str], list[str]]: + text = clean_ticket_text(issue).lower() + buckets: list[str] = [] + tags: list[str] = ["redmine"] + if any(term in text for term in ("interested in purchasing", "gaylord", "already working with a buyer")): + buckets.append("partnership-buy-from-us") + tags.extend(["partnership-opportunity", "buy-from-us", "gaylords"]) + if any(term in text for term in ("fulfillment", "returns workflow", "fba returns", "warehousing")): + buckets.extend(["customer-new-project", "meaningful-event"]) + tags.extend(["customer-inquiry", "returns-management"]) + if "rate sheet" in text or "calculator" in text or "pricing" in text: + buckets.append("meaningful-event") + tags.extend(["pricing", "rate-sheet-gap"]) + if not buckets: + buckets.append("meaningful-event") + return list(dict.fromkeys(buckets)), list(dict.fromkeys(tags)) + + +def decompose_issue_dry_run(issue: dict[str, Any], vault_prefix: str = VAULT_ONYX_PREFIX) -> dict[str, Any]: + """Build a reviewable decomposition plan for one Redmine issue. + + This is the first headless companion slice inspired by KarpathyWiki. It does + not write to the vault, does not update Redmine, and deliberately writes + planned paths outside the plugin-owned `wiki/` tree. + """ + issue_id = int(issue.get("id") or 0) + subject = str(issue.get("subject") or f"Redmine issue {issue_id}").strip().strip('"') + event_date = issue_event_date(issue) + ym = event_date[:7] + org_name = infer_organization_name(issue) + org_slug = slugify_note(org_name) + subject_for_slug = re.sub(re.escape(org_name), "", subject, flags=re.IGNORECASE).strip(" -_:") or subject + event_slug = slugify_note(f"{org_name} {subject_for_slug}") + timeline_rel = f"Timeline/{ym}/{event_date}_{event_slug}.md" + entity_rel = f"entities/organizations/{org_slug}.md" + buckets, tags = classify_decomposition_buckets(issue) + contact = ticket_contact(issue) + contact_id = contact.get("id") + text = issue_text_for_decomposition(issue) + from_address = ticket_from_address(issue) + analysis = extract_decomposition_analysis(issue, org_name, buckets, tags) + summary = analysis["summary"] + confidence = "high" if "customer-new-project" in buckets or "rate-sheet-gap" in tags else "medium" + suggested_contact_tags = [tag for tag in tags if tag not in {"redmine", "customer-inquiry"}] + if "customer-new-project" in buckets and "maybe-customer" not in suggested_contact_tags: + suggested_contact_tags.insert(0, "maybe-customer") + return { + "schema_version": 1, + "mode": "dry-run", + "source": { + "source_type": "redmine-ticket", + "redmine_issue_id": issue_id, + "redmine_url": f"https://rm.ldrprep.com/issues/{issue_id}" if issue_id else None, + "title": subject, + "date": event_date, + "sender": from_address, + "redmine_contact_id": contact_id, + }, + "buckets": buckets, + "tags": tags, + "confidence": confidence, + "analysis": analysis, + "review_required": True, + "timeline_events": [ + { + "path": timeline_rel, + "onyx_url": vault_prefix + timeline_rel, + "title": f"{event_date} — {org_name} {subject}", + "summary": summary, + "entity_links": [f"[[{entity_rel[:-3]}|{org_name}]]"], + "dialectic_sections": ["claim", "supporting_evidence", "counterpoints", "decision_outcome", "follow_up_candidates"], + } + ], + "entities": [ + { + "type": "organization", + "name": org_name, + "path": entity_rel, + "onyx_url": vault_prefix + entity_rel, + "redmine_contact_id": contact_id, + "aliases": [org_name], + "suggested_tags": suggested_contact_tags, + "source_analysis": analysis["entities"][0] if analysis.get("entities") else {}, + } + ], + "concepts": analysis.get("concepts", []), + "redmine_contact_updates": [ + { + "contact_id": contact_id, + "add_tags": suggested_contact_tags, + "obsidian_article_url": vault_prefix + entity_rel, + "apply": False, + } + ] if contact_id else [], + "evidence": { + "from": from_address, + "subject": subject, + "excerpt": first_sentence(analysis.get("clean_text", ""), 500), + }, + "compatibility_notes": [ + "Dry-run only: no vault files or Redmine contacts are modified.", + "Do not write into plugin-owned wiki/ by default; use Timeline/ and entities/organizations/.", + "Use explicit wikilinks if referencing plugin pages such as [[wiki/concepts/returns-management|returns management]].", + ], + } + + +def filter_issues( + issues: Iterable[dict[str, Any]], + updated_since: str | None = None, + updated_before: str | None = None, + status: str | None = None, + category: str | None = None, +) -> list[dict[str, Any]]: + since_dt = parse_dt(updated_since) + before_dt = parse_dt(updated_before) + status_l = status.lower() if status else None + category_l = category.lower() if category else None + filtered: list[dict[str, Any]] = [] + + for issue in issues: + updated_dt = parse_dt(issue.get("updated_on")) + if since_dt and (not updated_dt or updated_dt < since_dt): + continue + if before_dt and updated_dt and updated_dt > before_dt: + continue + if status_l: + issue_status = (nested_name(issue, "status") or "").lower() + if issue_status != status_l: + continue + if category_l: + issue_category = (nested_name(issue, "category") or "").lower() + if issue_category != category_l: + continue + filtered.append(issue) + return filtered + + +def select_issues( + issues: Iterable[dict[str, Any]], + order: str = "as-is", + max_tickets: int | None = None, +) -> list[dict[str, Any]]: + selected = list(issues) + if order == "oldest-updated-first": + selected.sort(key=lambda issue: parse_dt(issue.get("updated_on")) or datetime.min.replace(tzinfo=timezone.utc)) + elif order == "newest-updated-first": + selected.sort(key=lambda issue: parse_dt(issue.get("updated_on")) or datetime.min.replace(tzinfo=timezone.utc), reverse=True) + elif order == "id-asc": + selected.sort(key=lambda issue: int(issue.get("id", 0))) + elif order == "id-desc": + selected.sort(key=lambda issue: int(issue.get("id", 0)), reverse=True) + elif order != "as-is": + raise ValueError(f"Unsupported order: {order}") + + if max_tickets is not None: + selected = selected[:max_tickets] + return selected + + +@dataclass +class SyncResult: + seen: int = 0 + fetched: int = 0 + skipped_cached: int = 0 + written: int = 0 + + +DEFAULT_RUN_LOG_KEEP = 3 +DEFAULT_RUN_LOG_NAME = "run-stats.json" + + +def resolve_run_log_path(raw: str | None, cache_dir: Path) -> Path | None: + """Return the run-log path when enabled, else None. + + CLI uses nargs='?' with const='__default__' so: + --run-log → /run-stats.json + --run-log /path/file.json → /path/file.json + (omitted) → None + """ + if raw is None: + return None + if raw in {"", "__default__"}: + return Path(cache_dir) / DEFAULT_RUN_LOG_NAME + return Path(raw) + + +def load_run_log(path: Path) -> dict[str, Any]: + if not path.is_file(): + return {"version": 1, "keep": DEFAULT_RUN_LOG_KEEP, "runs": []} + try: + with path.open("r", encoding="utf-8") as handle: + data = json.load(handle) + except (OSError, json.JSONDecodeError): + return {"version": 1, "keep": DEFAULT_RUN_LOG_KEEP, "runs": []} + if not isinstance(data, dict): + return {"version": 1, "keep": DEFAULT_RUN_LOG_KEEP, "runs": []} + data.setdefault("version", 1) + data.setdefault("keep", DEFAULT_RUN_LOG_KEEP) + runs = data.get("runs") + if not isinstance(runs, list): + data["runs"] = [] + return data + + +def append_run_stats( + path: Path, + entry: dict[str, Any], + *, + keep: int = DEFAULT_RUN_LOG_KEEP, +) -> Path: + """Append one run entry and retain only the newest `keep` runs. + + Designed for cron: fixed small file, no rotation tooling required. + Newest run is last in the list. + """ + if keep < 1: + keep = DEFAULT_RUN_LOG_KEEP + path = Path(path) + path.parent.mkdir(parents=True, exist_ok=True) + data = load_run_log(path) + runs = [r for r in data.get("runs", []) if isinstance(r, dict)] + runs.append(entry) + data["version"] = 1 + data["keep"] = keep + data["runs"] = runs[-keep:] + data["updated_at"] = now_iso() + tmp_path = path.with_suffix(path.suffix + ".tmp") + with tmp_path.open("w", encoding="utf-8") as handle: + json.dump(data, handle, indent=2, sort_keys=True) + handle.write("\n") + tmp_path.replace(path) + return path + + +def build_sync_run_entry( + *, + started_at: str, + finished_at: str, + duration_seconds: float, + args: argparse.Namespace, + remote_updated_on: str | None, + result: SyncResult | None, + ok: bool, + error: str | None = None, +) -> dict[str, Any]: + return { + "started_at": started_at, + "finished_at": finished_at, + "duration_seconds": round(duration_seconds, 3), + "command": "sync", + "ok": ok, + "error": error, + "base_url": getattr(args, "base_url", None), + "project": getattr(args, "project", None), + "cache_dir": str(getattr(args, "cache_dir", "")), + "dry_run": bool(getattr(args, "dry_run", False)), + "filters": { + "remote_status_id": getattr(args, "status_id", None), + "remote_updated_on": remote_updated_on, + "remote_updated_since": getattr(args, "remote_updated_since", None), + "remote_updated_before": getattr(args, "remote_updated_before", None), + "local_updated_since": getattr(args, "updated_since", None), + "local_updated_before": getattr(args, "updated_before", None), + "local_status": getattr(args, "status", None), + "local_category": getattr(args, "category", None), + "max_tickets": getattr(args, "max_tickets", None), + "order": getattr(args, "order", None), + }, + "stats": asdict(result) if result is not None else None, + } + + +@dataclass +class SpamCheckResult: + issue_id: int + score: int + is_spam: bool + subject: str + from_address: str + reasons: list[str] + category: str = "unknown-review" + + +@dataclass +class CloseIfSpamPlan: + issue_id: int + payload: dict[str, Any] + contact_id: int | None + contact_review: dict[str, Any] + subject: str + + +class ManifestTicketStateStore: + """Manifest-backed ticket state store. + + This is intentionally a thin implementation behind the TicketCache state-store + boundary. A future SQLite store should only need to implement the same small + method set used here. + """ + + def __init__(self, cache_dir: Path, project: str): + self.cache_dir = cache_dir + self.project = project + self.manifest_path = cache_dir / "manifest.json" + self.manifest = self.load_manifest() + + def load_manifest(self) -> dict[str, Any]: + if not self.manifest_path.is_file(): + return {"version": 1, "tickets": {}} + with self.manifest_path.open("r", encoding="utf-8") as handle: + data = json.load(handle) + data.setdefault("version", 1) + data.setdefault("tickets", {}) + return data + + def save(self) -> None: + self.cache_dir.mkdir(parents=True, exist_ok=True) + with self.manifest_path.open("w", encoding="utf-8") as handle: + json.dump(self.manifest, handle, indent=2, sort_keys=True) + handle.write("\n") + + def get_ticket_state(self, issue_id: int | str) -> dict[str, Any] | None: + return self.manifest.get("tickets", {}).get(str(issue_id)) + + def upsert_ticket_state(self, issue_id: int | str, fields: dict[str, Any]) -> None: + ticket_entry = self.manifest.setdefault("tickets", {}).setdefault(str(issue_id), {}) + ticket_entry.setdefault("project", self.project) + ticket_entry.setdefault("actions", {}) + ticket_entry.update(fields) + self.save() + + def has_action(self, issue_id: int | str, action_name: str) -> bool: + entry = self.get_ticket_state(issue_id) or {} + return action_name in entry.get("actions", {}) + + def record_action( + self, + issue_id: int | str, + action_name: str, + result: str, + metadata: dict[str, Any] | None = None, + ) -> None: + ticket_entry = self.manifest.setdefault("tickets", {}).setdefault(str(issue_id), {}) + ticket_entry.setdefault("project", self.project) + actions = ticket_entry.setdefault("actions", {}) + actions[action_name] = { + "last_run": now_iso(), + "result": result, + "metadata": metadata or {}, + } + self.save() + + +class TicketCache: + def __init__(self, cache_dir: Path, project: str, state_store: Any | None = None): + self.cache_dir = cache_dir + self.project = project + self.project_dir = cache_dir / project + self.state_store = state_store or ManifestTicketStateStore(cache_dir, project) + + @property + def manifest(self) -> dict[str, Any]: + return getattr(self.state_store, "manifest", {"version": 1, "tickets": {}}) + + @property + def manifest_path(self) -> Path | None: + return getattr(self.state_store, "manifest_path", None) + + def issue_path(self, issue_id: int | str) -> Path: + return self.project_dir / f"{issue_id}.json" + + def get_entry(self, issue_id: int | str) -> dict[str, Any] | None: + return self.state_store.get_ticket_state(issue_id) + + def should_fetch(self, issue_stub: dict[str, Any]) -> bool: + issue_id = issue_stub.get("id") + if issue_id is None: + return True + entry = self.get_entry(issue_id) + if not entry: + return True + if not self.issue_path(issue_id).is_file(): + return True + cached_updated = entry.get("updated_on") + current_updated = issue_stub.get("updated_on") + return bool(current_updated and cached_updated != current_updated) + + def write_issue(self, issue: dict[str, Any]) -> None: + issue_id = issue["id"] + self.project_dir.mkdir(parents=True, exist_ok=True) + with self.issue_path(issue_id).open("w", encoding="utf-8") as handle: + json.dump(issue, handle, indent=2, sort_keys=True) + handle.write("\n") + self.state_store.upsert_ticket_state( + issue_id, + { + "project": self.project, + "updated_on": issue.get("updated_on"), + "last_synced": now_iso(), + "path": str(self.issue_path(issue_id)), + }, + ) + + def iter_cached_issues(self) -> Iterable[dict[str, Any]]: + if not self.project_dir.is_dir(): + return [] + issues: list[dict[str, Any]] = [] + for path in sorted(self.project_dir.glob("*.json")): + with path.open("r", encoding="utf-8") as handle: + issues.append(json.load(handle)) + return issues + + def has_action(self, issue_id: int | str, action_name: str) -> bool: + return self.state_store.has_action(issue_id, action_name) + + def record_action( + self, + issue_id: int | str, + action_name: str, + result: str, + metadata: dict[str, Any] | None = None, + ) -> None: + self.state_store.record_action(issue_id, action_name, result, metadata) + + +class TicketRedmineClient(RedmineClient): + def list_issues( + self, + limit: int = 100, + status_id: str = "*", + include: str | None = None, + extra_params: dict[str, Any] | None = None, + progress: bool = False, + ) -> list[dict[str, Any]]: + """List issue stubs from Redmine with server-side filters. + + `status_id` and any keys in `extra_params` (e.g. updated_on, sort) are + sent as Redmine REST filters. Do not put heavy includes on the list + path when you plan to fetch full issues one-by-one. + """ + issues: list[dict[str, Any]] = [] + offset = 0 + total = None + page_num = 0 + while total is None or offset < total: + params: dict[str, Any] = { + "limit": limit, + "offset": offset, + "status_id": status_id, + } + if include: + params["include"] = include + if extra_params: + params.update(extra_params) + data = self.request("GET", f"/projects/{self.project}/issues.json", params) + page = data.get("issues", []) + issues.extend(page) + total = int(data.get("total_count", len(issues))) + received = len(page) + page_num += 1 + if progress: + print( + f" list page {page_num}: +{received} " + f"(collected={len(issues)}/{total}, offset={offset})", + flush=True, + ) + if received == 0: + break + offset += received + return issues + + def get_issue_with_includes(self, issue_id: int, includes: Iterable[str]) -> dict[str, Any]: + return self.get_issue(issue_id, includes=includes) + + def update_issue(self, issue_id: int, fields: dict[str, Any], impersonate_user: str | None = None) -> None: + self.request( + "PUT", + f"/issues/{issue_id}.json", + payload={"issue": fields}, + impersonate_user=impersonate_user, + ) + + def add_internal_note( + self, + issue_id: int, + notes: str, + *, + private: bool = False, + impersonate_user: str | None = None, + ) -> None: + """Add a Redmine journal note. Does NOT send Helpdesk/customer email.""" + text = (notes or "").strip() + if not text: + raise RedmineError("Internal note body is required.") + fields: dict[str, Any] = {"notes": text} + if private: + fields["private_notes"] = True + self.update_issue(issue_id, fields, impersonate_user=impersonate_user) + + def send_customer_helpdesk_email( + self, + issue_id: int, + content: str, + *, + status_id: int | None = None, + to_address: str | None = None, + cc_address: str | None = None, + bcc_address: str | None = None, + impersonate_user: str | None = None, + ) -> dict[str, Any]: + """Send a customer-visible Helpdesk email via POST /helpdesk/email_note. + + This is the RedmineUP Helpdesk "send note" path used by redMCP. + It emails the customer. Do not use for internal staff notes. + """ + text = (content or "").strip() + if not text: + raise RedmineError("Customer email body is required.") + message: dict[str, Any] = {"issue_id": int(issue_id), "content": text} + if status_id is not None: + message["status_id"] = int(status_id) + payload: dict[str, Any] = {"message": message} + if to_address: + payload["to_address"] = to_address + if cc_address: + payload["cc_address"] = cc_address + if bcc_address: + payload["bcc_address"] = bcc_address + return self.request( + "POST", + "/helpdesk/email_note.json", + payload=payload, + impersonate_user=impersonate_user, + ) + + def get_issue_attachments(self, issue_id: int) -> list[dict[str, Any]]: + issue = self.get_issue_with_includes(int(issue_id), ("attachments", "journals", "helpdesk")) + atts = issue.get("attachments") + return list(atts) if isinstance(atts, list) else [] + + def attach_upload_to_issue( + self, + issue_id: int, + upload: dict[str, Any], + *, + notes: str | None = None, + impersonate_user: str | None = None, + ) -> None: + """Attach a previously uploaded token to an issue (does not email).""" + token = upload.get("token") + if not token: + raise RedmineError("Upload token missing.") + entry: dict[str, Any] = { + "token": token, + "filename": upload.get("filename") or "attachment.bin", + "content_type": upload.get("content_type") or "application/octet-stream", + } + if upload.get("description"): + entry["description"] = upload["description"] + fields: dict[str, Any] = {"uploads": [entry]} + if notes: + fields["notes"] = notes + self.update_issue(int(issue_id), fields, impersonate_user=impersonate_user) + + def recent_issues_for_contact(self, contact_id: int, limit: int) -> list[dict[str, Any]]: + response = self.helpdesk_issues_by_contact(contact_id, limit) + summaries = response.get("issues", []) if isinstance(response, dict) else [] + issues: list[dict[str, Any]] = [] + for summary in summaries[:limit]: + issue_id = summary.get("issue_id") or summary.get("id") + if issue_id is None: + continue + issues.append(self.get_issue_with_includes(int(issue_id), ("journals", "helpdesk"))) + return issues + + +def sync_tickets( + client: Any, + cache: TicketCache, + updated_since: str | None = None, + updated_before: str | None = None, + status: str | None = None, + category: str | None = None, + max_tickets: int | None = None, + order: str = "as-is", + apply: bool = True, + include_journals: bool = True, + page_size: int = 100, + status_id: str = "*", + remote_updated_on: str | None = None, + progress: bool = True, +) -> SyncResult: + """Incrementally sync tickets. + + Server-side (Redmine API) filters: + - status_id + - updated_on (via remote_updated_on, official operators) + + Local post-filters still available via updated_since/updated_before/status/category. + Per-ticket should_fetch skips full downloads when cache already has same updated_on. + """ + extra_params: dict[str, Any] = {} + if remote_updated_on: + extra_params["updated_on"] = remote_updated_on + # Prefer newest first on the wire so --max-tickets is meaningful under remote filters. + if order == "newest-updated-first": + extra_params["sort"] = "updated_on:desc" + elif order == "oldest-updated-first": + extra_params["sort"] = "updated_on" + elif order == "id-desc": + extra_params["sort"] = "id:desc" + elif order == "id-asc": + extra_params["sort"] = "id" + + if progress: + print(f"Cache dir: {cache.cache_dir}", flush=True) + print(f"Manifest: {cache.manifest_path}", flush=True) + print(f"Project: {cache.project}", flush=True) + print( + f"Remote API filters: status_id={status_id!r} updated_on={remote_updated_on!r}", + flush=True, + ) + if updated_since or updated_before or status or category: + print( + "Local post-filters: " + f"updated_since={updated_since!r} updated_before={updated_before!r} " + f"status={status!r} category={category!r}", + flush=True, + ) + print("Listing issue stubs from Redmine...", flush=True) + + stubs = client.list_issues( + limit=page_size, + status_id=status_id, + include=None, + extra_params=extra_params or None, + progress=progress, + ) + if progress: + print(f"Listed {len(stubs)} stub(s) from API.", flush=True) + + selected = filter_issues( + stubs, + updated_since=updated_since, + updated_before=updated_before, + status=status, + category=category, + ) + selected = select_issues(selected, order=order, max_tickets=max_tickets) + includes = ("journals", "helpdesk", "attachments") if include_journals else ("helpdesk", "attachments") + result = SyncResult(seen=len(selected)) + + if progress: + print(f"Selected {len(selected)} ticket(s) after local filters/order/max.", flush=True) + print("Fetching full tickets (skipping unchanged cache entries)...", flush=True) + + for index, stub in enumerate(selected, start=1): + issue_id = stub.get("id") + if issue_id is None: + continue + if not cache.should_fetch(stub): + result.skipped_cached += 1 + if progress and (index == 1 or index == len(selected) or index % 25 == 0): + print( + f" [{index}/{len(selected)}] #{issue_id} cache-skip " + f"(fetched={result.fetched} skipped={result.skipped_cached} written={result.written})", + flush=True, + ) + continue + full_issue = client.get_issue_with_includes(int(issue_id), includes) + result.fetched += 1 + if apply: + cache.write_issue(full_issue) + result.written += 1 + if progress and (index == 1 or index == len(selected) or index % 10 == 0 or result.fetched <= 3): + print( + f" [{index}/{len(selected)}] #{issue_id} fetched " + f"(fetched={result.fetched} skipped={result.skipped_cached} written={result.written})", + flush=True, + ) + return result + + +def spam_check_text(issue: dict[str, Any]) -> str: + parts = [str(issue.get("subject", "")), str(issue.get("description", ""))] + helpdesk = issue.get("helpdesk_ticket") or {} + if isinstance(helpdesk, dict): + parts.append(str(helpdesk.get("from_address", ""))) + contact = helpdesk.get("contact") or {} + if isinstance(contact, dict): + parts.append(str(contact.get("email", ""))) + parts.append(str(contact.get("name", ""))) + parts.append(str(contact.get("company", ""))) + for journal in issue.get("journals", [])[:5]: + if isinstance(journal, dict): + user = journal.get("user") or {} + user_name = str(user.get("name", "")) if isinstance(user, dict) else "" + if user_name and user_name.lower() != "anonymous": + continue + parts.append(str(journal.get("notes", ""))[:1000]) + return "\n".join(parts) + + +def score_ticket_spam(issue: dict[str, Any], threshold: int = 2) -> SpamCheckResult: + text = spam_check_text(issue) + subject = str(issue.get("subject", "")) + helpdesk = issue.get("helpdesk_ticket") or {} + from_address = "" + if isinstance(helpdesk, dict): + from_address = str(helpdesk.get("from_address", "")) + reasons: list[str] = [] + score = 0 + text_l = text.lower() + + if AUTOMATED_OPERATIONAL.search(text): + return SpamCheckResult( + issue_id=int(issue.get("id", 0)), + score=0, + is_spam=False, + subject=subject, + from_address=from_address, + reasons=["automated operational notification"], + category="automated-operational", + ) + + if INBOUND_BUSINESS_INQUIRY.search(text): + return SpamCheckResult( + issue_id=int(issue.get("id", 0)), + score=0, + is_spam=False, + subject=subject, + from_address=from_address, + reasons=["inbound or established business inquiry"], + category="unknown-review", + ) + + if SENDER_SPAM_HINTS.search(from_address): + score += 1 + reasons.append("sender has promotional/automated hint") + if PROMO_LANGUAGE.search(subject): + score += 2 + reasons.append("subject contains promotional language") + elif PROMO_LANGUAGE.search(text): + score += 1 + reasons.append("body/journal contains promotional language") + if SALES_PITCH_LANGUAGE.search(text): + score += 2 + reasons.append("ticket contains unsolicited sales pitch language") + if SENDER_SPAM_HINTS.search(text): + score += 1 + reasons.append("ticket text contains promotional sender hint") + + is_spam = score >= threshold + category = "spam-promotional" if is_spam else "unknown-review" + return SpamCheckResult( + issue_id=int(issue.get("id", 0)), + score=score, + is_spam=is_spam, + subject=subject, + from_address=from_address, + reasons=reasons, + category=category, + ) + + +def run_spam_check(cache: TicketCache, apply: bool = False, threshold: int = 2) -> list[SpamCheckResult]: + results: list[SpamCheckResult] = [] + for issue in cache.iter_cached_issues(): + issue_id = issue.get("id") + if issue_id is None or cache.has_action(issue_id, "spam-check"): + continue + result = score_ticket_spam(issue, threshold=threshold) + results.append(result) + if apply: + cache.record_action( + issue_id, + "spam-check", + "spam" if result.is_spam else "not-spam", + { + "score": result.score, + "category": result.category, + "subject": result.subject, + "from_address": result.from_address, + "reasons": result.reasons, + }, + ) + return results + + +def print_spam_results(results: Iterable[SpamCheckResult]) -> None: + for result in results: + verdict = result.category if result.is_spam else result.category + reasons = "; ".join(result.reasons) if result.reasons else "no spam signals" + print( + f"#{result.issue_id}\t{verdict}\tscore={result.score}\t" + f"from={result.from_address}\t{result.subject}\t{reasons}" + ) + + +def ticket_contact_id(issue: dict[str, Any]) -> int | None: + helpdesk = issue.get("helpdesk_ticket") or {} + if not isinstance(helpdesk, dict): + return None + contact = helpdesk.get("contact") or {} + if isinstance(contact, dict) and contact.get("id") is not None: + return int(contact["id"]) + if helpdesk.get("contact_id") is not None: + return int(helpdesk["contact_id"]) + return None + + +def action_result(cache: TicketCache, issue_id: int | str, action_name: str) -> str | None: + entry = cache.get_entry(issue_id) or {} + action = entry.get("actions", {}).get(action_name) or {} + result = action.get("result") + return str(result) if result is not None else None + + +def action_category(cache: TicketCache, issue_id: int | str, action_name: str) -> str | None: + entry = cache.get_entry(issue_id) or {} + action = entry.get("actions", {}).get(action_name) or {} + metadata = action.get("metadata", {}) + if not isinstance(metadata, dict): + metadata = {} + category = metadata.get("category") or action.get("result") + return str(category) if category is not None else None + + +def action_metadata(cache: TicketCache, issue_id: int | str, action_name: str) -> dict[str, Any]: + entry = cache.get_entry(issue_id) or {} + action = entry.get("actions", {}).get(action_name) or {} + metadata = action.get("metadata", {}) + return metadata if isinstance(metadata, dict) else {} + + +def contact_has_tag(contact: dict[str, Any], tag: str) -> bool: + wanted = tag.strip().lower() + return wanted in {existing.strip().lower() for existing in extract_tag_list(contact)} + + +def contact_is_customer(client: Any, contact_id: int | None) -> bool: + if contact_id is None: + return False + contact = client.get_contact(contact_id) + return contact_has_tag(contact, "customer") + + +def analyze_contact_spam(issues: Iterable[dict[str, Any]], threshold: int = 2) -> dict[str, Any]: + scored = [score_ticket_spam(issue, threshold=threshold) for issue in issues] + spam_ids = [result.issue_id for result in scored if result.is_spam] + total = len(scored) + likely_spammer = bool(total and (len(spam_ids) >= 2 or len(spam_ids) / total >= 0.5)) + return { + "checked": total, + "spam_count": len(spam_ids), + "spam_issue_ids": spam_ids, + "likely_spammer": likely_spammer, + } + + +def build_close_payload(close_status_id: int, notes: str | None = None) -> dict[str, Any]: + return { + "status_id": close_status_id, + "notes": notes or "Closed as likely spam by redmine_tickets.py close-if-spam.", + } + + +def spam_check_reasons(cache: TicketCache, issue_id: int | str) -> list[str]: + metadata = action_metadata(cache, issue_id, "spam-check") + reasons = metadata.get("reasons", []) + return [str(reason) for reason in reasons] if isinstance(reasons, list) else [] + + +def run_tag_maybe_customer_contacts( + client: Any, + cache: TicketCache, + apply: bool = False, + tag_name: str = "maybe-customer", + impersonate_user: str = "rebot", +) -> list[dict[str, Any]]: + results: list[dict[str, Any]] = [] + seen_contacts: set[int] = set() + for issue in cache.iter_cached_issues(): + issue_id = issue.get("id") + if issue_id is None: + continue + reasons = spam_check_reasons(cache, issue_id) + if "inbound or established business inquiry" not in reasons: + continue + contact_id = ticket_contact_id(issue) + if contact_id is None or contact_id in seen_contacts: + continue + seen_contacts.add(contact_id) + contact = client.get_contact(contact_id) + if contact_has_tag(contact, "customer") or contact_has_tag(contact, tag_name): + continue + new_tags = merge_tags(extract_tag_list(contact), [tag_name], []) + result = {"contact_id": contact_id, "tag_list": new_tags, "source_issue_id": int(issue_id)} + results.append(result) + if apply: + client.update_contact(contact_id, {"tag_list": new_tags}, impersonate_user=impersonate_user) + cache.record_action( + issue_id, + "tag-maybe-customer", + "tagged", + {"contact_id": contact_id, "tag": tag_name}, + ) + return results + + +def run_close_if_spam( + client: Any, + cache: TicketCache, + close_status_id: int, + contact_ticket_limit: int = 10, + apply: bool = False, + threshold: int = 2, + impersonate_user: str = "rebot", + close_categories: tuple[str, ...] = ("spam-promotional", "spam"), +) -> list[CloseIfSpamPlan]: + plans: list[CloseIfSpamPlan] = [] + for issue in cache.iter_cached_issues(): + issue_id = issue.get("id") + if issue_id is None: + continue + if action_category(cache, issue_id, "spam-check") not in close_categories: + continue + if cache.has_action(issue_id, "close-if-spam"): + continue + + contact_id = ticket_contact_id(issue) + if contact_is_customer(client, contact_id): + if apply: + cache.record_action( + issue_id, + "close-if-spam", + "skipped-customer-contact", + {"contact_id": contact_id}, + ) + continue + recent_issues = client.recent_issues_for_contact(contact_id, contact_ticket_limit) if contact_id else [] + contact_review = analyze_contact_spam(recent_issues, threshold=threshold) + payload = build_close_payload(close_status_id) + plan = CloseIfSpamPlan( + issue_id=int(issue_id), + payload=payload, + contact_id=contact_id, + contact_review=contact_review, + subject=str(issue.get("subject", "")), + ) + plans.append(plan) + + if apply: + client.update_issue(int(issue_id), payload, impersonate_user=impersonate_user) + cache.record_action( + issue_id, + "close-if-spam", + "closed", + {"payload": payload, "contact_id": contact_id}, + ) + cache.record_action( + issue_id, + "contact-spam-review", + "likely-spammer" if contact_review.get("likely_spammer") else "single-ticket-spam", + contact_review, + ) + return plans + + +def print_close_if_spam_plans(plans: Iterable[CloseIfSpamPlan]) -> None: + for plan in plans: + review = plan.contact_review + print( + f"#{plan.issue_id}\tclose_payload={json.dumps(plan.payload, sort_keys=True)}\t" + f"contact_id={plan.contact_id}\tcontact_spam={review.get('spam_count')}/{review.get('checked')}\t" + f"likely_spammer={review.get('likely_spammer')}\t{plan.subject}" + ) + + +def print_issue_rows(issues: Iterable[dict[str, Any]]) -> None: + for issue in issues: + status = nested_name(issue, "status") or "" + category = nested_name(issue, "category") or "" + assignee = nested_name(issue, "assigned_to") or "" + contact = "" + helpdesk_raw = issue.get("helpdesk_ticket") + helpdesk = helpdesk_raw if isinstance(helpdesk_raw, dict) else {} + c = helpdesk.get("contact") if helpdesk else None + if isinstance(c, dict): + contact = str(c.get("email") or c.get("name") or c.get("company") or "") + elif helpdesk.get("from_address"): + contact = str(helpdesk.get("from_address")) + tags = issue.get("_control_tags") + tag_s = ",".join(tags) if isinstance(tags, list) and tags else "" + subject = str(issue.get("subject", "")).replace("\n", " ") + print( + f"#{issue.get('id')}\t{issue.get('updated_on', '')}\t{status}\t" + f"{assignee}\t{contact}\t{category}\t{tag_s}\t{subject}" + ) + + +def load_issue_for_decompose(args: argparse.Namespace, cache: TicketCache) -> dict[str, Any]: + if args.input_json: + with Path(args.input_json).open("r", encoding="utf-8") as handle: + return json.load(handle) + if args.issue_id is None: + raise RedmineError("decompose requires --issue-id or --input-json") + cached_path = cache.issue_path(args.issue_id) + if cached_path.is_file(): + with cached_path.open("r", encoding="utf-8") as handle: + return json.load(handle) + client = client_from_args(args) + return client.get_issue_with_includes(args.issue_id, ("journals", "helpdesk")) + + +def render_decomposition_markdown(plan: dict[str, Any]) -> str: + source = plan["source"] + lines = [ + f"# Decomposition dry-run — Redmine #{source.get('redmine_issue_id')}", + "", + "## Source", + "", + f"- Redmine: {source.get('redmine_url')}", + f"- Title: {source.get('title')}", + f"- Date: {source.get('date')}", + f"- Sender: {source.get('sender')}", + f"- Contact ID: {source.get('redmine_contact_id')}", + "", + "## Buckets", + "", + ] + lines.extend(f"- `{bucket}`" for bucket in plan.get("buckets", [])) + analysis = plan.get("analysis", {}) + lines.extend(["", "## Structured analysis", "", f"**Summary:** {analysis.get('summary', '')}", "", "**Key points:**", ""]) + lines.extend(f"- {point}" for point in analysis.get("key_points", [])) + concepts = plan.get("concepts", []) + if concepts: + lines.extend(["", "**Concepts:**", ""]) + for concept in concepts: + lines.append(f"- `{concept.get('name')}` ({concept.get('type')}) — {concept.get('summary')}") + lines.extend(["", "## Planned timeline events", ""]) + for event in plan.get("timeline_events", []): + lines.extend([ + f"- `{event['path']}`", + f" - Onyx: {event['onyx_url']}", + f" - Summary: {event['summary']}", + ]) + lines.extend(["", "## Planned entities", ""]) + for entity in plan.get("entities", []): + lines.extend([ + f"- `{entity['path']}` — {entity['name']}", + f" - Onyx: {entity['onyx_url']}", + f" - Suggested tags: {', '.join(entity.get('suggested_tags', []))}", + ]) + lines.extend(["", "## Redmine contact update proposals", ""]) + for update in plan.get("redmine_contact_updates", []): + lines.extend([ + f"- Contact `{update['contact_id']}`", + f" - Add tags: {', '.join(update.get('add_tags', []))}", + f" - Obsidian article: {update.get('obsidian_article_url')}", + " - Apply: false", + ]) + lines.extend(["", "## Evidence excerpt", "", plan.get("evidence", {}).get("excerpt", ""), "", "## Compatibility notes", ""]) + lines.extend(f"- {note}" for note in plan.get("compatibility_notes", [])) + return "\n".join(lines) + + +def allow_prod_writes_enabled(args: argparse.Namespace) -> bool: + """True if parent or subcommand requested prod writes.""" + return bool(getattr(args, "allow_prod_writes", False)) + + +def print_draft_queued( + path: Path, + *, + kind: str, + issue_id: int | None = None, + contact_id: int | None = None, + base_url: str | None, + emails_customer: bool = False, +) -> str: + """Print machine- and human-friendly draft queue info. Returns draft_id.""" + draft_id = path.stem + print(f"draft_id={draft_id}") + print(f"draft_path={path}") + if issue_id is not None: + print(f"issue_id={issue_id}") + if contact_id is not None: + print(f"contact_id={contact_id}") + print(f"kind={kind}") + print(f"base_url={base_url or ''}") + print(f"emails_customer={'YES' if emails_customer else 'NO'}") + if emails_customer: + print( + f"apply_hint=draft apply {draft_id} --i-understand-this-emails-the-customer" + + (" --allow-prod-writes" if base_url and "rm.ldrprep.com" in str(base_url) else "") + ) + else: + hint = f"apply_hint=draft apply {draft_id}" + if base_url and "rm.ldrprep.com" in str(base_url): + hint += " --allow-prod-writes" + print(hint) + return draft_id + + +def load_runtime_config(path: Path) -> dict[str, Any]: + """Load a minimal YAML or JSON config for global CLI defaults. + + Supported shapes: + {base-url, project, cache-dir, allow-prod-writes, api-key-file} + {redmine_tickets: { ... same keys ... }} + YAML requires PyYAML if available; otherwise JSON only. + """ + text = path.read_text(encoding="utf-8") + data: Any + if path.suffix.lower() in {".json"}: + data = json.loads(text) + else: + try: + import yaml # type: ignore + except ImportError as exc: + # try JSON anyway for .yaml misnamed + try: + data = json.loads(text) + except json.JSONDecodeError: + raise RedmineError( + f"Cannot load config {path}: need PyYAML for YAML, or use .json. ({exc})" + ) from exc + else: + data = yaml.safe_load(text) + if not isinstance(data, dict): + raise RedmineError(f"Config root must be a mapping: {path}") + if isinstance(data.get("redmine_tickets"), dict): + data = data["redmine_tickets"] + # normalize keys: base_url / base-url + out: dict[str, Any] = {} + for key, value in data.items(): + norm = str(key).replace("_", "-").lower() + if norm in {"base-url", "project", "cache-dir", "allow-prod-writes", "api-key-file"}: + out[norm] = value + # ignore nested command sections for this minimal implementation + return out + + +def apply_runtime_config_to_argv(argv: list[str]) -> list[str]: + """If --config is present, inject missing parent defaults from the file. + + Does not override flags already on the command line. + """ + if "--config" not in argv: + return argv + try: + idx = argv.index("--config") + config_path = Path(argv[idx + 1]) + except (ValueError, IndexError): + return argv + if not config_path.is_file(): + # also try relative to script + alt = Path(__file__).resolve().parent / config_path + if alt.is_file(): + config_path = alt + else: + raise RedmineError(f"Config file not found: {argv[idx + 1]}") + cfg = load_runtime_config(config_path) + # strip --config from argv for argparse (we still re-add as optional known arg) + # Keep --config for argparse to record path; inject defaults before it. + injected: list[str] = [] + existing = set() + i = 0 + while i < len(argv): + a = argv[i] + if a.startswith("--") and "=" not in a: + existing.add(a) + elif a.startswith("--") and "=" in a: + existing.add(a.split("=", 1)[0]) + i += 1 + mapping = { + "base-url": "--base-url", + "project": "--project", + "cache-dir": "--cache-dir", + "api-key-file": "--api-key-file", + } + for key, flag in mapping.items(): + if key in cfg and flag not in existing: + injected.extend([flag, str(cfg[key])]) + if cfg.get("allow-prod-writes") in (True, "true", "yes", 1, "1") and "--allow-prod-writes" not in existing: + injected.append("--allow-prod-writes") + # place injected globals at the front (before subcommand) + return injected + argv + + +def resolve_apply_base_url(args: argparse.Namespace, draft: dict[str, Any]) -> str: + """Prefer draft-stored base_url so apply hits the host the draft was written for.""" + draft_url = (draft.get("base_url") or "").strip() + cli_url = (getattr(args, "base_url", None) or "").strip() + default_url = (os.getenv("REDMINE_BASE_URL") or DEFAULT_BASE_URL).rstrip("/") + if draft_url: + # If user passed an explicit different --base-url, require override + cli_is_default = cli_url.rstrip("/") in {default_url.rstrip("/"), DEFAULT_BASE_URL.rstrip("/")} + # Also treat as explicit if it differs from draft + if cli_url and cli_url.rstrip("/") != draft_url.rstrip("/") and not cli_is_default: + if not getattr(args, "allow_base_url_override", False): + raise RedmineError( + f"Draft was created for base_url={draft_url!r} but CLI has --base-url={cli_url!r}. " + "Refusing mismatched apply. Re-run without overriding --base-url, " + "or pass --allow-base-url-override if intentional." + ) + print(f"WARNING: applying draft to CLI base_url={cli_url} (draft was {draft_url})", file=sys.stderr) + return cli_url + return draft_url + print( + "WARNING: draft has no stored base_url (legacy); using CLI --base-url=" + f"{cli_url or default_url}", + file=sys.stderr, + ) + return cli_url or default_url + + +def resolve_body_text(args: argparse.Namespace) -> str: + body = getattr(args, "body", None) + body_file = getattr(args, "body_file", None) + if body_file: + body = Path(body_file).read_text(encoding="utf-8") + if not body or not str(body).strip(): + raise RedmineError("Requires --body or --body-file with non-empty content.") + text = str(body) + if getattr(args, "to_textile", False): + text = maybe_convert_to_textile(text) + return text + + +def maybe_convert_to_textile(text: str) -> str: + """If pandoc is available, convert Markdown (or plain) → Textile for Redmine journals. + + Falls back to original text with a stderr warning when pandoc is missing/fails. + """ + import shutil + import subprocess + + pandoc = shutil.which("pandoc") + if not pandoc: + print( + "warning: --to-textile requested but pandoc not found; sending body unchanged", + file=sys.stderr, + ) + return text + try: + proc = subprocess.run( + [pandoc, "-f", "markdown", "-t", "textile"], + input=text, + text=True, + capture_output=True, + check=False, + timeout=30, + ) + except Exception as exc: # noqa: BLE001 + print(f"warning: pandoc failed ({exc}); sending body unchanged", file=sys.stderr) + return text + if proc.returncode != 0 or not (proc.stdout or "").strip(): + err = (proc.stderr or "").strip()[:200] + print(f"warning: pandoc conversion empty/failed ({err}); sending body unchanged", file=sys.stderr) + return text + return proc.stdout + + +def is_closed_status(issue: dict[str, Any]) -> bool: + status = issue.get("status") + if not isinstance(status, dict): + return False + if status.get("is_closed") is True: + return True + name = str(status.get("name") or "").strip().lower() + return name in {"closed", "rejected", "resolved", "cancelled", "canceled", "done"} + + +def build_fields_payload_from_args( + args: argparse.Namespace, + *, + resolved_ids: dict[str, int] | None = None, +) -> dict[str, Any]: + """Build Redmine issue field payload from set-fields CLI args. + + Prefer numeric *id flags; name flags should be pre-resolved into resolved_ids. + """ + payload: dict[str, Any] = {} + resolved_ids = resolved_ids or {} + if getattr(args, "field_status_id", None) is not None: + payload["status_id"] = int(args.field_status_id) + elif "status_id" in resolved_ids: + payload["status_id"] = int(resolved_ids["status_id"]) + if getattr(args, "assigned_to_id", None) is not None: + aid = int(args.assigned_to_id) + payload["assigned_to_id"] = None if aid == 0 else aid + elif "assigned_to_id" in resolved_ids: + aid = int(resolved_ids["assigned_to_id"]) + payload["assigned_to_id"] = None if aid == 0 else aid + if getattr(args, "priority_id", None) is not None: + payload["priority_id"] = int(args.priority_id) + elif "priority_id" in resolved_ids: + payload["priority_id"] = int(resolved_ids["priority_id"]) + if getattr(args, "category_id", None) is not None: + cid = int(args.category_id) + payload["category_id"] = None if cid == 0 else cid + elif "category_id" in resolved_ids: + cid = int(resolved_ids["category_id"]) + payload["category_id"] = None if cid == 0 else cid + if getattr(args, "due_date", None): + payload["due_date"] = str(args.due_date).strip() + if getattr(args, "subject", None): + payload["subject"] = str(args.subject) + if getattr(args, "notes", None): + payload["notes"] = str(args.notes) + if getattr(args, "private", False): + payload["private_notes"] = True + if not payload: + raise RedmineError( + "set-fields requires at least one of: --status/--status-id, " + "--assigned-to/--assigned-to-id, --priority/--priority-id, " + "--category/--category-id, --due-date, --subject, --notes" + ) + return payload + + +def resolve_set_fields_names(args: argparse.Namespace) -> dict[str, int]: + """Resolve --status/--assigned-to/--priority/--category name flags to IDs.""" + wants = any( + getattr(args, name, None) + for name in ("status_name", "assignee_name", "priority_name", "category_name") + ) + if not wants: + return {} + if resolve_field_names is None: + raise RedmineError("Name resolution helpers unavailable (redmine_ticket_meta.py).") + client = None + # Refresh meta if missing — needs API + try: + client = client_from_args(args) + except Exception: + client = None + try: + return resolve_field_names( + Path(args.cache_dir), + args.project, + status_name=getattr(args, "status_name", None), + assignee_name=getattr(args, "assignee_name", None), + priority_name=getattr(args, "priority_name", None), + category_name=getattr(args, "category_name", None), + client=client, + refresh=bool(getattr(args, "refresh_meta", False)), + ) + except LookupError as exc: + raise RedmineError(str(exc)) from exc + + +def apply_draft_to_redmine( + client: TicketRedmineClient, + draft: dict[str, Any], + *, + impersonate_user: str, + allow_customer_email: bool, +) -> dict[str, Any]: + kind = normalize_kind(str(draft.get("kind") or "")) if normalize_kind else str(draft.get("kind")) + body = str(draft.get("body") or "") + if kind == KIND_CONTACT_FIELDS: + contact_id = draft.get("contact_id") + if contact_id is None: + raise RedmineError("Contact draft missing contact_id.") + fields = draft.get("fields") if isinstance(draft.get("fields"), dict) else {} + if not fields: + raise RedmineError("Contact draft has empty fields payload.") + payload = dict(fields) + # Support TUI shorthand for contact tags (add_tag / remove_tag) by merging properly + if "add_tag" in payload: + tag = str(payload.pop("add_tag")).strip() + if tag: + try: + contact = client.get_contact(int(contact_id)) + current = extract_tag_list(contact) if extract_tag_list else [] + new_tags = merge_tags(current, [tag], []) if merge_tags else list(dict.fromkeys(current + [tag])) + payload = {"tag_list": new_tags} + except Exception: + payload = {"tag_list": [tag]} + if "remove_tag" in payload: + tag = str(payload.pop("remove_tag")).strip() + if tag: + try: + contact = client.get_contact(int(contact_id)) + current = extract_tag_list(contact) if extract_tag_list else [] + new_tags = merge_tags(current, [], [tag]) if merge_tags else [t for t in current if str(t).lower() != tag.lower()] + payload = {"tag_list": new_tags} + except Exception: + pass + client.update_contact(int(contact_id), payload, impersonate_user=impersonate_user) + return { + "kind": kind, + "contact_id": int(contact_id), + "emailed_customer": False, + "fields": fields, + } + if draft.get("issue_id") is None: + raise RedmineError(f"Draft kind {kind!r} requires issue_id.") + issue_id = int(draft["issue_id"]) + if kind == KIND_INTERNAL_NOTE: + client.add_internal_note( + issue_id, + body, + private=bool(draft.get("private")), + impersonate_user=impersonate_user, + ) + return {"kind": kind, "issue_id": issue_id, "emailed_customer": False} + if kind == KIND_CUSTOMER_EMAIL: + if not allow_customer_email: + raise RedmineError( + "Refusing to apply customer_email draft without " + "--i-understand-this-emails-the-customer. " + "This path SENDS email to the customer via Helpdesk." + ) + result = client.send_customer_helpdesk_email( + issue_id, + body, + impersonate_user=impersonate_user, + ) + return {"kind": kind, "issue_id": issue_id, "emailed_customer": True, "api_result": result} + if kind == KIND_FIELDS: + fields = draft.get("fields") if isinstance(draft.get("fields"), dict) else {} + if not fields: + raise RedmineError("Field draft has empty fields payload.") + client.update_issue(issue_id, dict(fields), impersonate_user=impersonate_user) + return {"kind": kind, "issue_id": issue_id, "emailed_customer": False, "fields": fields} + raise RedmineError(f"Cannot apply draft kind {kind!r}") + + +def load_cached_issue(cache: TicketCache, issue_id: int) -> dict[str, Any]: + path = cache.issue_path(issue_id) + if not path.is_file(): + raise RedmineError(f"No cached ticket #{issue_id} at {path}") + + with path.open("r", encoding="utf-8") as handle: + return json.load(handle) + + +def refresh_cached_issue( + cache: "TicketCache", + client: Any, + issue_id: int, + *, + includes: list[str] | None = None, +) -> dict[str, Any] | None: + """Fetch the current issue from Redmine and overwrite the local cache file. + + Use after any field update / note / etc. so that the next + `rtl browse`, `rtl tui`, etc. see the fresh data (updated_on, journals, etc.) + instead of stale local copy. + """ + if client is None: + return None + try: + inc = includes or ["journals", "helpdesk", "attachments"] + if hasattr(client, "get_issue_with_includes"): + fresh = client.get_issue_with_includes(int(issue_id), inc) + elif hasattr(client, "get_issue"): + fresh = client.get_issue(int(issue_id), includes=inc) + else: + return None + if isinstance(fresh, dict) and fresh.get("id"): + cache.write_issue(fresh) + return fresh + except Exception: + # non-fatal; the user can always run a full sync + pass + return None + + +def select_cached_issues_for_cli(cache: TicketCache, args: argparse.Namespace) -> list[dict[str, Any]]: + issues = list(cache.iter_cached_issues()) + matches = filter_issues( + issues, + updated_since=getattr(args, "updated_since", None), + updated_before=getattr(args, "updated_before", None), + status=getattr(args, "status", None), + category=getattr(args, "category", None), + ) + where_raw = list(getattr(args, "where", None) or []) + if where_raw: + if filter_by_query is None: + raise RedmineError("Property query module redmine_ticket_query.py is unavailable.") + saved_dir = Path(getattr(args, "saved_filters_dir", None) or DEFAULT_SAVED_FILTERS_DIR) + # Prefer repo-relative saved-filters next to this script if cwd differs + script_saved = Path(__file__).resolve().parent / "saved-filters" + search_dirs = [saved_dir] + if script_saved not in search_dirs and script_saved.is_dir(): + search_dirs.append(script_saved) + try: + if expand_where_list is not None: + where_exprs = expand_where_list(where_raw, search_dirs) + else: + where_exprs = where_raw + matches = filter_by_query(matches, expressions=where_exprs) + except QueryError as exc: + raise RedmineError(f"Invalid --where expression: {exc}") from exc + + control_tag = getattr(args, "control_tag", None) + if control_tag: + if filter_issues_by_control_tag is None: + raise RedmineError("Control-tag helpers unavailable (redmine_ticket_control_tags.py).") + matches = filter_issues_by_control_tag(matches, Path(args.cache_dir), str(control_tag)) + + if annotate_issues_with_control_tags is not None: + matches = annotate_issues_with_control_tags(matches, Path(args.cache_dir)) + + if getattr(args, "has_attachments", False): + matches = [ + i + for i in matches + if isinstance(i.get("attachments"), list) and len(i.get("attachments") or []) > 0 + ] + if getattr(args, "open_only", False): + matches = [i for i in matches if not is_closed_status(i)] + order = getattr(args, "order", "newest-updated-first") + max_tickets = getattr(args, "max_tickets", None) + return select_issues(matches, order=order, max_tickets=max_tickets) + + +def run_walk_session(issues: list[dict[str, Any]], cache: TicketCache, pager: str = "auto") -> int: + if not issues: + print("No tickets to walk.") + return 0 + if render_ticket_markdown is None or write_ticket_markdown is None or display_markdown_file is None: + raise RedmineError("Ticket view helpers unavailable (redmine_ticket_view.py).") + + index = 0 + total = len(issues) + print(f"Walking {total} ticket(s). Commands: n/enter=next, p=prev, q=quit, r=redraw") + while True: + issue = issues[index] + issue_id = issue.get("id") + print(f"\n=== [{index + 1}/{total}] #{issue_id} ===") + md_path = write_ticket_markdown(issue, cache.cache_dir, cache.project) + display_markdown_file(md_path, pager=pager) + try: + raw = input("walk> ").strip().lower() + except (EOFError, KeyboardInterrupt): + print() + return 0 + if raw in {"", "n", "next", "j"}: + if index + 1 >= total: + print("End of list.") + else: + index += 1 + continue + if raw in {"p", "prev", "k", "b", "back"}: + if index <= 0: + print("Start of list.") + else: + index -= 1 + continue + if raw in {"q", "quit", "exit"}: + return 0 + if raw in {"r", "redraw"}: + continue + print("Unknown command. Use n / p / q / r") + + +def client_from_args(args: argparse.Namespace) -> TicketRedmineClient: + api_key = os.getenv("REDMINE_API_KEY") + if not api_key and getattr(args, "api_key_file", None): + key_path = Path(args.api_key_file) + if key_path.is_file(): + api_key = key_path.read_text(encoding="utf-8").strip() + if not api_key: + raise RedmineError("Set REDMINE_API_KEY in the environment or supply --api-key-file.") + return TicketRedmineClient(base_url=args.base_url, api_key=api_key, project=args.project) + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + description="Incrementally sync and search enriched Redmine issue payloads.", + formatter_class=argparse.ArgumentDefaultsHelpFormatter, + ) + parser.add_argument("--base-url", default=os.getenv("REDMINE_BASE_URL", DEFAULT_BASE_URL)) + parser.add_argument("--project", default=os.getenv("REDMINE_PROJECT", DEFAULT_PROJECT)) + parser.add_argument("--cache-dir", type=Path, default=DEFAULT_CACHE_DIR) + parser.add_argument("--api-key-file", type=Path, default=None, help="Path to a file containing only the Redmine API key (600 perms recommended).") + parser.add_argument( + "--allow-prod-writes", + action="store_true", + help=( + "Allow write/apply operations against non-LAN hosts (e.g. https://rm.ldrprep.com). " + "Parent-level flag so it works with: ... --allow-prod-writes draft apply " + ), + ) + parser.add_argument( + "--config", + type=Path, + default=None, + help="Optional YAML/JSON config for base defaults (base-url, project, cache-dir, allow-prod-writes, api-key-file).", + ) + + subparsers = parser.add_subparsers(dest="command") + sync_parser = subparsers.add_parser("sync", help="Incrementally download enriched issues into cache.") + add_common_filters(sync_parser) + add_remote_sync_filters(sync_parser) + sync_parser.add_argument("--page-size", type=int, default=100) + sync_parser.add_argument( + "--dry-run", + action="store_true", + help="Fetch but do not write cache files/manifest.", + ) + sync_parser.add_argument( + "--no-journals", + action="store_true", + help="Request include=helpdesk only instead of journals,helpdesk on full issue fetch.", + ) + sync_parser.add_argument( + "--quiet", + action="store_true", + help="Suppress per-stage progress output (final summary still prints).", + ) + sync_parser.add_argument( + "--run-log", + nargs="?", + const="__default__", + default=None, + metavar="PATH", + help=( + "Record this sync's stats into a fixed small ring-buffer file " + f"(keeps last {DEFAULT_RUN_LOG_KEEP} runs). " + f"With no PATH, uses /{DEFAULT_RUN_LOG_NAME}. " + "Intended for cron so you get recent stats without managing log rotation." + ), + ) + sync_parser.add_argument( + "--run-log-keep", + type=int, + default=DEFAULT_RUN_LOG_KEEP, + help=f"How many recent runs to keep in --run-log (default: {DEFAULT_RUN_LOG_KEEP}).", + ) + + search_parser = subparsers.add_parser("search", help="Search/filter already cached issues.") + atp = _get_add_triage_parser() + if atp: atp(subparsers) + + add_common_filters(search_parser) + search_parser.add_argument( + "--where", + action="append", + default=[], + metavar="EXPR", + help=( + "LOCAL property query against cached ticket JSON. Repeatable (AND). " + 'Examples: subject.includes("rate sheet"); ' + "@unassigned or alias.helpdeskEmailMatchCompany (saved-filters/); " + 'status.name == "New" AND assigned_to IS NULL.' + ), + ) + search_parser.add_argument( + "--saved-filters-dir", + type=Path, + default=None, + help="Directory of saved filter .expr files (default: ./saved-filters or repo saved-filters/).", + ) + search_parser.add_argument("--json", action="store_true", help="Print matching issues as JSON.") + search_parser.add_argument("--jsonl", action="store_true", help="Print matching issues as JSON Lines.") + search_parser.add_argument( + "--ids-only", + action="store_true", + help="Print only matching issue ids (one per line).", + ) + search_parser.add_argument( + "--has-attachments", + action="store_true", + help="Only tickets whose cached JSON includes a non-empty attachments list.", + ) + search_parser.add_argument( + "--open-only", + action="store_true", + help="Exclude closed/resolved-ish statuses (is_closed or common closed names).", + ) + search_parser.add_argument( + "--control-tag", + help="LOCAL control-tag filter: only tickets labeled with this local tag (not a Redmine tag).", + ) + + view_parser = subparsers.add_parser( + "view", + help="Render a cached ticket as Markdown and show it (glow/bat/cat).", + ) + view_parser.add_argument("--issue-id", type=int, help="Redmine issue id (cache-first).") + view_parser.add_argument( + "issue_id_pos", + nargs="?", + type=int, + help="Issue id positional alternative to --issue-id.", + ) + view_parser.add_argument( + "--pager", + default="auto", + help="Markdown display: auto|glow|bat|batcat|cat (default auto).", + ) + view_parser.add_argument( + "--print", + dest="print_only", + action="store_true", + help="Print Markdown to stdout only (no pager).", + ) + view_parser.add_argument( + "--write-md", + action="store_true", + help="Write/update Markdown under //markdown/.md (regen when updated_on changes).", + ) + view_parser.add_argument( + "--force-md", + action="store_true", + help="Force regenerate Markdown cache even if updated_on stamp matches.", + ) + + walk_parser = subparsers.add_parser( + "walk", + help="Walk tickets interactively (n/p/q). MVP: --ids or same filters as search.", + ) + add_common_filters(walk_parser) + walk_parser.add_argument( + "--where", + action="append", + default=[], + metavar="EXPR", + help="Same property query as search --where (repeatable AND; supports @saved aliases).", + ) + walk_parser.add_argument( + "--saved-filters-dir", + type=Path, + default=None, + help="Directory of saved filter .expr files.", + ) + walk_parser.add_argument( + "--ids", + help="Comma-separated issue ids to walk (overrides search selection when set).", + ) + walk_parser.add_argument( + "--pager", + default="auto", + help="Markdown display: auto|glow|bat|batcat|cat (default auto).", + ) + walk_parser.add_argument( + "--control-tag", + help="LOCAL control-tag filter (same as search --control-tag).", + ) + walk_parser.add_argument( + "--open-only", + action="store_true", + help="Exclude closed/resolved-ish statuses.", + ) + walk_parser.add_argument( + "--has-attachments", + action="store_true", + help="Only tickets with attachments metadata in cache.", + ) + + mq_parser = subparsers.add_parser( + "materialize-queues", + help="Refresh named queue folders of symlinks from saved filters / queue config JSON.", + ) + mq_parser.add_argument( + "--config", + type=Path, + default=Path("queues.example.json"), + help="JSON queue definitions (see queues.example.json).", + ) + mq_parser.add_argument( + "--queues-dir", + type=Path, + default=None, + help="Root directory for queue folders (default: /queues).", + ) + mq_parser.add_argument( + "--saved-filters-dir", + type=Path, + default=None, + help="Directory of saved filter .expr files.", + ) + mq_parser.add_argument( + "--dry-run", + action="store_true", + help="Print what would be linked without writing symlinks.", + ) + + draft_parser = subparsers.add_parser( + "draft", + help="Local draft queue: list/cancel/apply staged notes or customer emails.", + ) + draft_sub = draft_parser.add_subparsers(dest="draft_command") + # Legacy "draft add --kind" removed from help focus; use note / customer-email commands. + draft_add = draft_sub.add_parser( + "add", + help="(Prefer `note` / `customer-email` commands.) Queue a draft by explicit kind.", + ) + draft_add.add_argument("--issue-id", type=int, required=True) + draft_add.add_argument( + "--kind", + choices=(KIND_INTERNAL_NOTE, KIND_CUSTOMER_EMAIL, "note", "helpdesk_reply"), + required=True, + help="REQUIRED explicit kind: internal_note (no email) or customer_email (emails customer).", + ) + draft_add.add_argument("--body", help="Draft body text.") + draft_add.add_argument("--body-file", type=Path, help="Read body from file.") + draft_add.add_argument( + "--private", + action="store_true", + help="Only valid for internal_note. Private notes never email customers.", + ) + draft_list = draft_sub.add_parser("list", help="List local drafts.") + draft_list.add_argument("--all", action="store_true", help="Include cancelled/applied, not only pending.") + draft_cancel = draft_sub.add_parser("cancel", help="Cancel a pending draft by id.") + draft_cancel.add_argument("draft_id", nargs="?", help="Draft id (or unique prefix).") + draft_cancel.add_argument("--all-pending", action="store_true", help="Cancel every pending draft.") + draft_apply = draft_sub.add_parser( + "apply", + help="Apply a pending draft to Redmine (LAN/test by default; prod needs --allow-prod-writes).", + ) + draft_apply.add_argument("draft_id", help="Draft id (or unique prefix).") + draft_apply.add_argument( + "--impersonate-user", + default="rebot", + help="X-Redmine-Switch-User for writes (default: rebot).", + ) + draft_apply.add_argument( + "--i-understand-this-emails-the-customer", + action="store_true", + help="REQUIRED when applying a customer_email draft. Refused otherwise.", + ) + draft_apply.add_argument( + "--allow-base-url-override", + action="store_true", + help="Allow applying a draft to a different --base-url than the one stored on the draft.", + ) + + note_parser = subparsers.add_parser( + "note", + help="INTERNAL journal note only — does NOT email the customer. Default queues a draft.", + ) + note_parser.add_argument("--issue-id", type=int, required=True) + note_parser.add_argument("--body", help="Note body text.") + note_parser.add_argument("--body-file", type=Path, help="Read body from file.") + note_parser.add_argument( + "--to-textile", + action="store_true", + help="If pandoc is installed, convert body (Markdown) to Textile before queue/apply.", + ) + note_parser.add_argument( + "--private", + action="store_true", + help="Mark as private journal note (staff-only; still does not email).", + ) + note_parser.add_argument( + "--apply", + action="store_true", + help="Write to Redmine now instead of only queueing a local draft.", + ) + note_parser.add_argument( + "--impersonate-user", + default="rebot", + help="Used only with --apply (default: rebot).", + ) + + email_parser = subparsers.add_parser( + "customer-email", + help="CUSTOMER EMAIL via Helpdesk — WILL email the customer. Separate from `note` on purpose.", + ) + email_parser.add_argument("--issue-id", type=int, required=True) + email_parser.add_argument("--body", help="Email body text.") + email_parser.add_argument("--body-file", type=Path, help="Read body from file.") + email_parser.add_argument( + "--to-textile", + action="store_true", + help="If pandoc is installed, convert body (Markdown) to Textile before queue/send.", + ) + email_parser.add_argument( + "--i-understand-this-emails-the-customer", + action="store_true", + required=True, + help="MANDATORY confirmation flag. Without it the command will not even queue.", + ) + email_parser.add_argument( + "--apply", + action="store_true", + help="Send via Helpdesk API now instead of only queueing a local draft.", + ) + email_parser.add_argument( + "--impersonate-user", + default="rebot", + help="Used only with --apply (default: rebot).", + ) + email_parser.add_argument("--to-address", help="Optional override To: for Helpdesk send.") + email_parser.add_argument("--cc-address", help="Optional Cc: for Helpdesk send.") + email_parser.add_argument("--status-id", type=int, help="Optional status_id when sending customer email.") + + fields_parser = subparsers.add_parser( + "set-fields", + help="Update issue fields (status/assignee/priority/category/due_date). Does NOT email. Default queues a draft.", + ) + fields_parser.add_argument("--issue-id", type=int, required=True) + fields_parser.add_argument( + "--status-id", + dest="field_status_id", + type=int, + help="Redmine status_id (numeric).", + ) + fields_parser.add_argument( + "--status", + dest="status_name", + help="Status by name (e.g. Closed). Resolved via cached meta or live API.", + ) + fields_parser.add_argument( + "--assigned-to-id", + type=int, + help="Assignee user id; use 0 to unassign.", + ) + fields_parser.add_argument( + "--assigned-to", + dest="assignee_name", + help="Assignee by name/login (substring OK). Use 'unassigned' to clear.", + ) + fields_parser.add_argument("--priority-id", type=int, help="Redmine priority_id.") + fields_parser.add_argument( + "--priority", + dest="priority_name", + help="Priority by name (e.g. Normal).", + ) + fields_parser.add_argument( + "--category-id", + type=int, + help="Category id; use 0 to clear.", + ) + fields_parser.add_argument( + "--category", + dest="category_name", + help="Category by name; use 'none' to clear.", + ) + fields_parser.add_argument("--due-date", help="Due date YYYY-MM-DD.") + fields_parser.add_argument("--subject", help="Replace issue subject.") + fields_parser.add_argument( + "--notes", + help="Optional internal journal note attached to the field change (still no email).", + ) + fields_parser.add_argument( + "--private", + action="store_true", + help="If --notes is set, mark that journal note private.", + ) + fields_parser.add_argument( + "--apply", + action="store_true", + help="Write to Redmine now instead of only queueing a local draft.", + ) + fields_parser.add_argument( + "--impersonate-user", + default="rebot", + help="Used only with --apply (default: rebot).", + ) + fields_parser.add_argument( + "--refresh-meta", + action="store_true", + help="Force refresh of status/user/priority/category name→id cache before resolve.", + ) + + meta_parser = subparsers.add_parser( + "meta-refresh", + help="Refresh local name→id metadata cache (statuses, priorities, categories, project users).", + ) + meta_parser.add_argument( + "--force", + action="store_true", + help="Refresh even if cache files are recent.", + ) + + contacts_parser = subparsers.add_parser( + "contacts-sync", + help="Sync project contacts into /contacts/ (durable local mirror).", + ) + contacts_parser.add_argument("--page-size", type=int, default=100) + contacts_parser.add_argument("--max-contacts", type=int, help="Cap how many contacts to process.") + contacts_parser.add_argument("--dry-run", action="store_true") + contacts_parser.add_argument("--quiet", action="store_true") + + csearch_parser = subparsers.add_parser( + "contacts-search", + help="Search locally cached contacts (after contacts-sync).", + ) + csearch_parser.add_argument("--query", help="Free-text match on name/email/company/tags.") + csearch_parser.add_argument("--email", help="Substring match on email.") + csearch_parser.add_argument("--company", help="Substring match on company.") + csearch_parser.add_argument("--tag", help="Match Redmine contact tag name.") + csearch_parser.add_argument("--max-contacts", type=int, default=50) + csearch_parser.add_argument("--ids-only", action="store_true") + csearch_parser.add_argument("--json", action="store_true") + + cshow_parser = subparsers.add_parser( + "contact-show", + help="Show one cached contact (cache-first; optional live fetch).", + ) + cshow_parser.add_argument("--contact-id", type=int, required=True) + cshow_parser.add_argument( + "--fetch", + action="store_true", + help="If missing from cache, fetch from Redmine and write cache.", + ) + cshow_parser.add_argument("--json", action="store_true", help="Print full JSON.") + + cset_parser = subparsers.add_parser( + "contact-set", + help="Update a CRM contact (tags/company/etc). Does NOT email. Default queues a draft.", + ) + cset_parser.add_argument("--contact-id", type=int, required=True) + cset_parser.add_argument( + "--add-tag", + action="append", + default=[], + help="Redmine CRM tag to add (repeatable). Not a local control-tag.", + ) + cset_parser.add_argument( + "--remove-tag", + action="append", + default=[], + help="Redmine CRM tag to remove (repeatable).", + ) + cset_parser.add_argument("--company", help="Set company string.") + cset_parser.add_argument("--first-name", dest="first_name", help="Set first name.") + cset_parser.add_argument("--last-name", dest="last_name", help="Set last name.") + cset_parser.add_argument( + "--set", + action="append", + default=[], + dest="contact_sets", + metavar="KEY=VALUE", + help="Raw contact field assignment (e.g. job_title=Buyer). Repeatable.", + ) + cset_parser.add_argument( + "--apply", + action="store_true", + help="Write to Redmine now instead of only queueing a local draft.", + ) + cset_parser.add_argument( + "--impersonate-user", + default="rebot", + help="Used only with --apply (default: rebot).", + ) + cset_parser.add_argument( + "--refresh-cache", + action="store_true", + help="After apply, re-fetch contact into local contacts cache.", + ) + + tcontact_parser = subparsers.add_parser( + "ticket-contact", + help="Bridge: resolve helpdesk contact id from a cached ticket (handoff to contact-*).", + ) + tcontact_parser.add_argument("--issue-id", type=int, required=True) + tcontact_parser.add_argument( + "--show", + action="store_true", + help="Also print contact name/email/company after resolving id (local contacts cache).", + ) + tcontact_parser.add_argument( + "--fetch", + action="store_true", + help="Fetch/refresh contact from Redmine into local cache and show it (implies --show).", + ) + tcontact_parser.add_argument("--json", action="store_true", help="Emit JSON with issue/contact ids.") + + att_parser = subparsers.add_parser( + "attachments", + help="List/download/upload ticket file attachments (does NOT email customer).", + ) + att_sub = att_parser.add_subparsers(dest="attachments_command") + att_list = att_sub.add_parser("list", help="List attachments on a ticket (cache-first; --refresh for live).") + att_list.add_argument("--issue-id", type=int, required=True) + att_list.add_argument( + "--refresh", + action="store_true", + help="Fetch issue with include=attachments from Redmine and update local cache entry.", + ) + att_list.add_argument("--json", action="store_true") + att_dl = att_sub.add_parser("download", help="Download one or all attachments for a ticket.") + att_dl.add_argument("--issue-id", type=int, required=True) + att_dl.add_argument("--attachment-id", type=int, help="Specific attachment id (required if multiple unless --all).") + att_dl.add_argument("--all", action="store_true", help="Download every attachment on the issue.") + att_dl.add_argument( + "--out-dir", + type=Path, + default=None, + help="Destination dir (default: //attachments//).", + ) + att_dl.add_argument( + "--refresh", + action="store_true", + help="Refresh attachment metadata from Redmine first.", + ) + att_up = att_sub.add_parser( + "upload", + help="Upload a local file and attach it to a ticket (does NOT email). Requires --apply to write.", + ) + att_up.add_argument("--issue-id", type=int, required=True) + att_up.add_argument("--file", type=Path, required=True, help="Local file path to upload.") + att_up.add_argument("--filename", help="Override filename sent to Redmine.") + att_up.add_argument("--description", default="", help="Attachment description.") + att_up.add_argument("--notes", help="Optional internal journal note with the attach (no email).") + att_up.add_argument( + "--apply", + action="store_true", + help="Actually upload+attach. Without this, only validates the local file path.", + ) + att_up.add_argument( + "--impersonate-user", + default="rebot", + help="Used only with --apply (default: rebot).", + ) + + ent_parser = subparsers.add_parser( + "entities", + help="Extract tracking/order/SKU-like entities from tickets (pattern-based; optional NMI enrich).", + ) + ent_sub = ent_parser.add_subparsers(dest="entities_command") + ent_one = ent_sub.add_parser("show", help="Extract entities from one ticket (cache).") + ent_one.add_argument("--issue-id", type=int, required=True) + ent_one.add_argument("--json", action="store_true") + ent_scan = ent_sub.add_parser("scan", help="Scan search results / cache for entity hits.") + ent_scan.add_argument("--where", action="append", default=[], help="Local property filter (repeatable AND).") + ent_scan.add_argument("--type", dest="entity_type", help="Only this entity type (e.g. tracking_ups).") + ent_scan.add_argument("--max-tickets", type=int, default=50) + ent_scan.add_argument("--ids-only", action="store_true", help="Print only issue ids that had hits.") + ent_scan.add_argument("--jsonl", action="store_true", help="One JSON object per hit line.") + ent_nmi = ent_sub.add_parser( + "nmi", + help="Resolve entities against NMI (bin/nmi) and print cs-brief / structured plan.", + ) + ent_nmi.add_argument("--issue-id", type=int, required=True) + ent_nmi.add_argument( + "--format", + choices=["brief", "json", "note"], + default="brief", + help="brief=cs-brief text; json=full payload; note=private-note body for Redmine.", + ) + ent_nmi.add_argument( + "--nmi-mode", + default=None, + help="NMI client mode (default: env NMI_MODE or dev).", + ) + ent_nmi.add_argument( + "--nmi-env-file", + default=None, + help="Path to NMI .env (default: mrscClient examples/modern/.env).", + ) + ent_nmi.add_argument( + "--nmi-bin", + default=None, + help="Path to bin/nmi (default: ~/projects/mrscClient-php/bin/nmi).", + ) + ent_nmi.add_argument( + "--nmi-endpoint", + default=None, + help="Optional NMI endpoint override.", + ) + ent_nmi.add_argument( + "--with-history", + action="store_true", + default=True, + help="Include item history from new getItemHistory endpoint (billings, moves, conditions) when item_no resolved. Default on.", + ) + ent_nmi.add_argument( + "--no-history", + dest="with_history", + action="store_false", + help="Disable item history fetch for this run.", + ) + ent_nmi.add_argument( + "--queue-private-note", + action="store_true", + help="Queue a private note draft with the brief (never emails). Use draft apply separately.", + ) + ent_nmi.add_argument( + "--apply-private-note", + action="store_true", + help="Also apply the private note (requires normal write gates / --allow-prod-writes on prod).", + ) + + browse_parser = subparsers.add_parser( + "browse", + help="Interactive loop: fzf/list → pager (glow/bat) → action menu → back to list.", + ) + browse_parser.add_argument("--where", action="append", default=[], help="Local property filter.") + browse_parser.add_argument("--control-tag", help="Local control-tag filter.") + browse_parser.add_argument("--status", help="Status name filter.") + browse_parser.add_argument("--max-tickets", type=int, default=80) + browse_parser.add_argument( + "--order", + choices=ORDER_CHOICES, + default="newest-updated-first", + ) + browse_parser.add_argument( + "--pager", + default="auto", + help="Markdown pager: auto|glow|bat|batcat|cat (same as view).", + ) + browse_parser.add_argument( + "--once", + action="store_true", + help="Pick one ticket, open pager, exit (no action menu / no loop).", + ) + browse_parser.add_argument( + "--has-attachments", + action="store_true", + help="Only tickets with attachments metadata in cache.", + ) + browse_parser.add_argument( + "--open-only", + action="store_true", + help="Exclude closed/resolved-ish statuses.", + ) + browse_parser.add_argument( + "--no-preview", + action="store_true", + help="Disable fzf right-hand preview pane.", + ) + + if add_tui_parser is not None: + add_tui_parser(subparsers) + + if add_textual_tui_parser is not None: + add_textual_tui_parser(subparsers) + + ocr_parser = subparsers.add_parser( + "ocr-rename", + help="OCR/extract text from ticket attachments; propose subject rename + private note draft.", + ) + ocr_parser.add_argument("--issue-id", type=int, help="Single issue id.") + ocr_parser.add_argument( + "--where", + action="append", + default=[], + help="Scan cached tickets matching --where (e.g. subject.includes(\"Scanned\")).", + ) + ocr_parser.add_argument( + "--max-tickets", + type=int, + default=20, + help="Max tickets when scanning with --where (default 20).", + ) + ocr_parser.add_argument( + "--only-generic-subjects", + action="store_true", + help="When scanning, skip tickets whose subject no longer looks like a generic scan title.", + ) + ocr_parser.add_argument( + "--queue-subject", + action="store_true", + default=True, + help="Queue set-fields draft with proposed subject (default on).", + ) + ocr_parser.add_argument( + "--no-queue-subject", + action="store_false", + dest="queue_subject", + help="Do not queue subject change draft.", + ) + ocr_parser.add_argument( + "--queue-note", + action="store_true", + default=True, + help="Queue private internal note with OCR text snippet (default on).", + ) + ocr_parser.add_argument( + "--no-queue-note", + action="store_false", + dest="queue_note", + help="Do not queue OCR note draft.", + ) + ocr_parser.add_argument( + "--apply-subject", + action="store_true", + help="Apply subject change now (LAN/prod safety gates apply). Still queues note unless --no-queue-note.", + ) + ocr_parser.add_argument( + "--impersonate-user", + default="rebot", + help="Used only with --apply-subject (default: rebot).", + ) + ocr_parser.add_argument( + "--json", + action="store_true", + help="Print machine-readable results.", + ) + + cat_parser = subparsers.add_parser( + "categories", + help="List (and later manage) project issue categories for triage.", + ) + cat_sub = cat_parser.add_subparsers(dest="categories_command") + cat_list = cat_sub.add_parser("list", help="List issue categories for the project.") + cat_list.add_argument( + "--refresh", + action="store_true", + help="Refresh from Redmine API (also updates meta cache).", + ) + cat_list.add_argument("--json", action="store_true") + + ctag_parser = subparsers.add_parser( + "control-tag", + help="LOCAL-only control tags (not Redmine tags). Label tickets for filters/automation.", + ) + ctag_sub = ctag_parser.add_subparsers(dest="control_tag_command") + ctag_add = ctag_sub.add_parser("add", help="Tag a ticket locally.") + ctag_add.add_argument("--tag", required=True, help="Local tag name (no spaces; e.g. bavaria).") + ctag_add.add_argument("--issue-id", type=int, required=True) + ctag_add.add_argument( + "--property", + action="append", + default=[], + dest="properties", + help="JSON path that justified the tag (repeatable).", + ) + ctag_add.add_argument("--annotation", default="", help="Optional free-text reason.") + ctag_rm = ctag_sub.add_parser("remove", help="Remove a local tag from a ticket.") + ctag_rm.add_argument("--tag", required=True) + ctag_rm.add_argument("--issue-id", type=int, required=True) + ctag_list = ctag_sub.add_parser("list", help="List local control tags and ticket counts.") + ctag_show = ctag_sub.add_parser("show", help="Show tickets for a local tag.") + ctag_show.add_argument("--tag", required=True) + ctag_for = ctag_sub.add_parser("for-issue", help="List local control tags on one ticket.") + ctag_for.add_argument("--issue-id", type=int, required=True) + ctag_suggest = ctag_sub.add_parser( + "suggest", + help="Rule-based suggestions for a control tag over cached tickets (does not write unless --apply).", + ) + ctag_suggest.add_argument("--tag", required=True) + ctag_suggest.add_argument( + "--where", + action="append", + default=[], + help="Property query candidates (repeatable AND). Example: subject.includes(\"bavaria\")", + ) + ctag_suggest.add_argument( + "--max-tickets", + type=int, + default=50, + help="Max candidates to propose.", + ) + ctag_suggest.add_argument( + "--apply", + action="store_true", + help="Write suggested tags locally (still not Redmine).", + ) + ctag_suggest.add_argument( + "--property", + action="append", + default=["subject"], + dest="properties", + help="Properties to record as rationale when applying (default: subject).", + ) + ctag_export = ctag_sub.add_parser( + "export", + help="Export tagged tickets as JSONL for training / LLM refine (local only).", + ) + ctag_export.add_argument("--tag", help="Only this control tag (default: all tags).") + ctag_export.add_argument( + "--out", + type=Path, + required=True, + help="Output .jsonl path.", + ) + ctag_wipe = ctag_sub.add_parser( + "wipe", + help="Delete the entire local control-tag store (does not touch Redmine or ticket cache).", + ) + ctag_wipe.add_argument( + "--i-understand-this-wipes-local-control-tags", + action="store_true", + required=True, + help="Mandatory confirmation.", + ) + + spam_parser = subparsers.add_parser("spam-check", help="Score cached issues for obvious spam/promo signals.") + spam_parser.add_argument("--apply", action="store_true", help="Record spam-check result in manifest for idempotency.") + spam_parser.add_argument("--threshold", type=int, default=2, help="Minimum score to report SPAM.") + + close_parser = subparsers.add_parser("close-if-spam", help="Close cached tickets previously marked spam and review sender history.") + close_parser.add_argument("--apply", action="store_true", help="Actually update Redmine issues and record close/contact-review markers.") + close_parser.add_argument("--close-status-id", type=int, default=5, help="Redmine status_id to set when closing spam tickets.") + close_parser.add_argument("--contact-ticket-limit", type=int, default=10, help="Recent sender/contact tickets to review before contact-level spam judgment.") + close_parser.add_argument("--threshold", type=int, default=2, help="Minimum ticket spam score used during contact lookback.") + close_parser.add_argument("--impersonate-user", default="rebot", help="Redmine username for write calls; sent as X-Redmine-Switch-User only when --apply updates issues.") + + maybe_parser = subparsers.add_parser("tag-maybe-customer", help="Tag contacts needing manual customer review based on inbound/business inquiry evidence.") + maybe_parser.add_argument("--apply", action="store_true", help="Actually add the tag to matching Redmine contacts.") + maybe_parser.add_argument("--tag", default="maybe-customer", help="Contact tag to add for manual review.") + maybe_parser.add_argument("--impersonate-user", default="rebot", help="Redmine username for contact tag writes; sent only with --apply.") + + decompose_parser = subparsers.add_parser("decompose", help="Build a dry-run ticket decomposition plan without writing vault or Redmine state.") + decompose_parser.add_argument("--issue-id", type=int, help="Redmine issue id to decompose. Uses cache first, then live API if needed.") + decompose_parser.add_argument("--input-json", type=Path, help="Read a cached/enriched issue JSON file instead of fetching by id.") + decompose_parser.add_argument("--format", choices=("json", "markdown"), default="json", help="Output format for the dry-run plan.") + kscan_parser = subparsers.add_parser("known-spam-scan", help="Scan for tickets matching known spam profiles (dry-run).") + kscan_parser.add_argument("--profiles", type=Path, default=Path("profiles/known-spam/customer-service.json"), help="Path to known-spam profiles JSON.") + kscan_parser.add_argument("--max-tickets", type=int, help="Limit number of tickets to scan.") + + kclose_parser = subparsers.add_parser("known-spam-close", help="Close tickets matching known spam profiles.") + kclose_parser.add_argument("--profiles", type=Path, default=Path("profiles/known-spam/customer-service.json"), help="Path to known-spam profiles JSON.") + kclose_parser.add_argument("--apply", action="store_true", help="Actually close matching Redmine tickets.") + kclose_parser.add_argument("--close-status-id", type=int, default=5, help="Redmine status_id to use when closing.") + kclose_parser.add_argument("--max-tickets", type=int, help="Limit number of tickets to process.") + kclose_parser.add_argument("--impersonate-user", default="rebot", help="Redmine username for write calls.") + + return parser + + +def add_common_filters(parser: argparse.ArgumentParser) -> None: + parser.add_argument( + "--updated-since", + help="LOCAL post-filter only: keep issues updated on/after date or datetime (applied after API list).", + ) + parser.add_argument( + "--updated-before", + help="LOCAL post-filter only: keep issues updated on/before date or datetime (applied after API list).", + ) + parser.add_argument( + "--status", + help="LOCAL post-filter only: filter by status name after fetch/cache load (e.g. New, Closed).", + ) + parser.add_argument( + "--category", + help="LOCAL post-filter only: filter by category name after fetch/cache load.", + ) + parser.add_argument("--max-tickets", type=int, help="Maximum ticket count after filters/order.") + parser.add_argument("--order", choices=ORDER_CHOICES, default="newest-updated-first") + + +def add_remote_sync_filters(parser: argparse.ArgumentParser) -> None: + """Server-side Redmine REST filters used by the sync list call.""" + parser.add_argument( + "--status-id", + "--remote-status-id", + dest="status_id", + default="*", + help=( + "SERVER-SIDE Redmine API filter `status_id`. " + "Values: * (open+closed), open, closed, or numeric status id. Default: *." + ), + ) + parser.add_argument( + "--remote-updated-on", + help=( + "SERVER-SIDE raw Redmine API filter for `updated_on` (official operators). " + "Examples: '>=2026-06-14', '>=2014-01-02T08:12:32Z', " + "'><2026-01-01|2026-06-01'. Mutually exclusive with the convenience flags." + ), + ) + parser.add_argument( + "--remote-updated-since", + help=( + "SERVER-SIDE convenience for updated_on >= .... Accepts YYYY-MM-DD, ISO datetime, " + "or relative forms: 30d, 2h, 15m, 1w, '2 days ago'. Builds official Redmine operator syntax." + ), + ) + parser.add_argument( + "--remote-updated-before", + help=( + "SERVER-SIDE convenience for updated_on <= .... Same value forms as --remote-updated-since. " + "Combined with --remote-updated-since as Redmine range operator > list[str]: + """Insert `sync` when the user omitted a subcommand. + + Argparse subparsers only accept their own flags *after* the command name. + Without this, a natural invocation like: + + ./redmine_tickets.py --project customer-service --updated-since 2026-07-12 + + fails with: invalid choice: '2026-07-12' (argparse treats the date as the command). + """ + if any(arg in KNOWN_COMMANDS for arg in argv): + return argv + + i = 0 + while i < len(argv): + arg = argv[i] + if arg in KNOWN_COMMANDS: + return argv + if arg in _PARENT_VALUE_OPTIONS: + i += 2 + continue + if arg.startswith("--base-url=") or arg.startswith("--project=") or arg.startswith("--cache-dir="): + i += 1 + continue + if arg in {"-h", "--help"}: + # Leave bare --help alone so top-level help still works. + return argv + # First unknown option/positional belongs to the default sync command. + return argv[:i] + ["sync"] + argv[i:] + # No args after parent globals (or empty): default to sync. + return argv + ["sync"] + + +def main(argv: list[str] | None = None) -> int: + raw_argv = list(argv) if argv is not None else sys.argv[1:] + try: + raw_argv = apply_runtime_config_to_argv(raw_argv) + except RedmineError as exc: + print(f"error: {exc}", file=sys.stderr) + return 2 + normalized_argv = inject_default_sync_command(raw_argv) + parser = build_parser() + args = parser.parse_args(normalized_argv) + + # Post-process common globals so subparsers (tui/browse/...) that re-declare them + # (to support "cmd --allow-prod-writes ...") don't leave them as None. + if getattr(args, "cache_dir", None) in (None, ""): + from pathlib import Path as _P + args.cache_dir = _P(".cache/redmine_tickets") + if getattr(args, "project", None) in (None, ""): + args.project = "customer-service" + if getattr(args, "base_url", None) in (None, ""): + args.base_url = os.getenv("REDMINE_BASE_URL", "https://rm.ldrprep.com") + + # Also normalize the write flag so tui/browse can rely on it + if not getattr(args, "allow_prod_writes", False): + env_flag = os.getenv("REDMINE_ALLOW_PROD_WRITES", "").lower() in ("1", "true", "yes") + setattr(args, "allow_prod_writes", env_flag) + + cache = TicketCache(args.cache_dir, args.project) + + # Keep the interactive TUI dispatch independent of the legacy command chain + # below. That chain historically became nested under the sync branch, which + # made `rtl tui` parse normally and then silently fall through to return 1. + if args.command == "tui": + if run_tui_cli is None: + raise RedmineError("TUI unavailable (redmine_ticket_tui.py failed to load).") + return run_tui_cli(args, cache) + + if args.command == "sync": + run_log_path = resolve_run_log_path(getattr(args, "run_log", None), args.cache_dir) + started_at = now_iso() + started_mono = time.monotonic() + remote_updated_on = None + sync_result: SyncResult | None = None + try: + client = client_from_args(args) + remote_updated_on = build_remote_updated_on_value( + raw=getattr(args, "remote_updated_on", None), + since=getattr(args, "remote_updated_since", None), + before=getattr(args, "remote_updated_before", None), + ) + sync_result = sync_tickets( + client, + cache, + updated_since=args.updated_since, + updated_before=args.updated_before, + status=args.status, + category=args.category, + max_tickets=args.max_tickets, + order=args.order, + apply=not args.dry_run, + include_journals=not args.no_journals, + page_size=args.page_size, + status_id=args.status_id, + remote_updated_on=remote_updated_on, + progress=not args.quiet, + ) + print( + f"Sync complete: seen={sync_result.seen} fetched={sync_result.fetched} " + f"skipped_cached={sync_result.skipped_cached} written={sync_result.written}" + ) + if args.dry_run: + print("Dry run only. Cache was not written.") + if run_log_path is not None: + entry = build_sync_run_entry( + started_at=started_at, + finished_at=now_iso(), + duration_seconds=time.monotonic() - started_mono, + args=args, + remote_updated_on=remote_updated_on, + result=sync_result, + ok=True, + ) + written_path = append_run_stats( + run_log_path, + entry, + keep=max(1, int(getattr(args, "run_log_keep", DEFAULT_RUN_LOG_KEEP) or DEFAULT_RUN_LOG_KEEP)), + ) + if not args.quiet: + print(f"Run log updated: {written_path} (kept last {args.run_log_keep})") + return 0 + except RedmineError as exc: + if run_log_path is not None: + entry = build_sync_run_entry( + started_at=started_at, + finished_at=now_iso(), + duration_seconds=time.monotonic() - started_mono, + args=args, + remote_updated_on=remote_updated_on, + result=sync_result, + ok=False, + error=str(exc), + ) + try: + append_run_stats( + run_log_path, + entry, + keep=max(1, int(getattr(args, "run_log_keep", DEFAULT_RUN_LOG_KEEP) or DEFAULT_RUN_LOG_KEEP)), + ) + except OSError: + pass + raise + + + if args.command == "search": + matches = select_cached_issues_for_cli(cache, args) + if not matches: + print("No matching tickets.", file=sys.stderr) + return 0 + ids_only = getattr(args, "ids_only", False) + do_json = getattr(args, "json", False) + do_jsonl = getattr(args, "jsonl", False) + if ids_only: + for issue in matches: + print(issue.get("id")) + elif do_json: + print(json.dumps(matches, indent=2, sort_keys=True, default=str)) + elif do_jsonl: + for issue in matches: + print(json.dumps(issue, default=str)) + else: + for issue in matches: + iid = issue.get("id") + subj = str(issue.get("subject") or "").replace("\n", " ")[:70] + st = (issue.get("status") or {}).get("name", "?") if isinstance(issue.get("status"), dict) else "?" + who = (issue.get("assigned_to") or {}).get("name", "unassigned") if isinstance(issue.get("assigned_to"), dict) else "unassigned" + print(f"#{iid}\t{st}\t{who}\t{subj}") + return 0 + + # Command dispatch below MUST stay at this indent (sibling of search/sync/tui). + # Nesting these under search after `return 0` made ocr-rename/view/attachments + # silently fall through to exit 1 with no output (2026-09-09). + + if args.command == "view": + if render_ticket_markdown is None or write_ticket_markdown is None or display_markdown_file is None: + raise RedmineError("Ticket view helpers unavailable (redmine_ticket_view.py).") + issue_id = getattr(args, "issue_id", None) or getattr(args, "issue_id_pos", None) + if issue_id is None: + raise RedmineError("view requires --issue-id or a positional issue id.") + try: + issue = load_cached_issue(cache, int(issue_id)) + except RedmineError: + client = client_from_args(args) + issue = client.get_issue_with_includes( + int(issue_id), ("attachments", "journals", "helpdesk") + ) + try: + cache.write_issue(issue) + except Exception: + pass + md = render_ticket_markdown(issue) + if getattr(args, "write_md", False) or getattr(args, "force_md", False) or not getattr(args, "print_only", False): + md_path = write_ticket_markdown( + issue, + Path(args.cache_dir), + args.project, + force=bool(getattr(args, "force_md", False)), + ) + else: + md_path = None + if getattr(args, "print_only", False): + print(md, end="" if md.endswith("\n") else "\n") + return 0 + return display_markdown_file(md_path, pager=getattr(args, "pager", "auto") or "auto") + + if args.command == "walk": + if args.ids: + ids: list[int] = [] + for part in str(args.ids).split(","): + part = part.strip() + if not part: + continue + ids.append(int(part)) + issues = [] + for issue_id in ids: + try: + issues.append(load_cached_issue(cache, issue_id)) + except RedmineError as exc: + print(f"warning: {exc}", file=sys.stderr) + else: + issues = select_cached_issues_for_cli(cache, args) + return run_walk_session(issues, cache, pager=args.pager) + + if args.command == "materialize-queues": + if load_queue_definitions is None or materialize_queue is None or filter_by_query is None: + raise RedmineError("Queue helpers unavailable (redmine_ticket_filters.py).") + config_path = args.config + if not config_path.is_file(): + # try next to script + alt = Path(__file__).resolve().parent / config_path + if alt.is_file(): + config_path = alt + if not config_path.is_file(): + raise RedmineError(f"Queue config not found: {args.config}") + queues = load_queue_definitions(config_path) + queues_root = args.queues_dir or (Path(args.cache_dir) / "queues") + saved_dir = Path(args.saved_filters_dir or DEFAULT_SAVED_FILTERS_DIR) + script_saved = Path(__file__).resolve().parent / "saved-filters" + search_dirs = [saved_dir] + if script_saved.is_dir() and script_saved not in search_dirs: + search_dirs.append(script_saved) + + all_issues = list(cache.iter_cached_issues()) + for qdef in queues: + name = str(qdef["name"]) + where_raw = list(qdef.get("where") or []) + if isinstance(qdef.get("filter"), str): + where_raw.append(str(qdef["filter"])) + try: + if expand_where_list is not None: + where_exprs = expand_where_list(where_raw, search_dirs) + else: + where_exprs = where_raw + matched = filter_by_query(all_issues, expressions=where_exprs) if where_exprs else list(all_issues) + except QueryError as exc: + raise RedmineError(f"Queue {name!r} filter error: {exc}") from exc + matched = select_issues( + matched, + order=str(qdef.get("order") or "newest-updated-first"), + max_tickets=qdef.get("max_tickets"), + ) + if args.dry_run: + print(f"queue={name} would_link={len(matched)} ids={[i.get('id') for i in matched[:20]]}") + continue + manifest = materialize_queue( + name=name, + issues=matched, + cache_dir=Path(args.cache_dir), + project=args.project, + queues_root=queues_root, + ) + print( + f"queue={name} linked={manifest['linked']} " + f"missing_source={manifest['missing_source']} dir={queues_root / name}" + ) + return 0 + + if args.command == "note": + body = resolve_body_text(args) + print("ACTION: INTERNAL NOTE") + print(" emails_customer: NO") + print(f" issue_id: {args.issue_id}") + print(f" private: {bool(args.private)}") + if not args.apply: + if add_draft is None: + raise RedmineError("Draft helpers unavailable.") + path = add_draft( + Path(args.cache_dir), + issue_id=int(args.issue_id), + kind=KIND_INTERNAL_NOTE, + body=body, + private=bool(args.private), + project=args.project, + base_url=args.base_url, + ) + print_draft_queued( + path, + kind=KIND_INTERNAL_NOTE, + issue_id=int(args.issue_id), + base_url=args.base_url, + emails_customer=False, + ) + return 0 + if assert_write_target_allowed is None: + raise RedmineError("Write safety helpers unavailable.") + try: + assert_write_target_allowed(args.base_url, allow_prod_writes=allow_prod_writes_enabled(args)) + except RuntimeError as exc: + raise RedmineError(str(exc)) from exc + client = client_from_args(args) + client.add_internal_note( + int(args.issue_id), + body, + private=bool(args.private), + impersonate_user=args.impersonate_user, + ) + + # Sync local cache so the new journal/note is visible in TUI/browse without full re-sync + try: + if refresh_cached_issue: + c = TicketCache(Path(args.cache_dir), args.project) + refresh_cached_issue(c, client, int(args.issue_id)) + except Exception: + pass + + print(f"applied internal note on #{args.issue_id} (no customer email)") + return 0 + + if args.command == "set-fields": + resolved = resolve_set_fields_names(args) + fields = build_fields_payload_from_args(args, resolved_ids=resolved) + print("ACTION: FIELD UPDATE") + print(" emails_customer: NO") + print(f" issue_id: {args.issue_id}") + if resolved: + print(f" resolved_names: {json.dumps(resolved, sort_keys=True)}") + print(f" fields: {json.dumps(fields, sort_keys=True)}") + if not args.apply: + if add_draft is None: + raise RedmineError("Draft helpers unavailable.") + path = add_draft( + Path(args.cache_dir), + issue_id=int(args.issue_id), + kind=KIND_FIELDS, + body="", + fields=fields, + project=args.project, + base_url=args.base_url, + ) + print_draft_queued( + path, + kind=KIND_FIELDS, + issue_id=int(args.issue_id), + base_url=args.base_url, + emails_customer=False, + ) + return 0 + if assert_write_target_allowed is None: + raise RedmineError("Write safety helpers unavailable.") + try: + assert_write_target_allowed(args.base_url, allow_prod_writes=allow_prod_writes_enabled(args)) + except RuntimeError as exc: + raise RedmineError(str(exc)) from exc + client = client_from_args(args) + client.update_issue(int(args.issue_id), fields, impersonate_user=args.impersonate_user) + + # Sync local cache with live so TUI/browse/etc. see the updated status/assignee/etc. immediately + try: + if refresh_cached_issue: + c = TicketCache(Path(args.cache_dir), args.project) + refresh_cached_issue(c, client, int(args.issue_id)) + except Exception: + pass + + print(f"applied field update on #{args.issue_id} (no customer email)") + return 0 + + if args.command == "meta-refresh": + if refresh_meta_cache is None: + raise RedmineError("Meta helpers unavailable (redmine_ticket_meta.py).") + client = client_from_args(args) + paths = refresh_meta_cache( + client, + Path(args.cache_dir), + args.project, + force=bool(args.force), + ) + for name, path in paths.items(): + print(f"{name}\t{path}") + return 0 + + if args.command == "contacts-sync": + if ContactCache is None or sync_contacts is None: + raise RedmineError("Contact helpers unavailable (redmine_ticket_contacts.py).") + client = client_from_args(args) + ccache = ContactCache(Path(args.cache_dir)) + result = sync_contacts( + client, + ccache, + page_size=int(args.page_size), + max_contacts=args.max_contacts, + progress=not bool(args.quiet), + dry_run=bool(args.dry_run), + ) + print(json.dumps(result, sort_keys=True)) + return 0 + + if args.command == "contacts-search": + if ContactCache is None or filter_contacts is None: + raise RedmineError("Contact helpers unavailable (redmine_ticket_contacts.py).") + ccache = ContactCache(Path(args.cache_dir)) + contacts = list(ccache.iter_contacts()) + matches = filter_contacts( + contacts, + query=args.query, + email=args.email, + company=args.company, + tag=args.tag, + ) + if args.max_contacts is not None: + matches = matches[: int(args.max_contacts)] + if args.json: + print(json.dumps(matches, indent=2, sort_keys=True)) + return 0 + if args.ids_only: + for c in matches: + if c.get("id") is not None: + print(c["id"]) + return 0 + print(f"matches={len(matches)}", file=sys.stderr) + for c in matches: + name = contact_display_name(c) if contact_display_name else c.get("name") + email = primary_email(c) if primary_email else c.get("email") + print(f"#{c.get('id')}\t{name}\t{email}\t{c.get('company') or ''}") + return 0 + + if args.command == "contact-show": + if ContactCache is None: + raise RedmineError("Contact helpers unavailable (redmine_ticket_contacts.py).") + ccache = ContactCache(Path(args.cache_dir)) + cid = int(args.contact_id) + try: + contact = ccache.load_contact(cid) + except FileNotFoundError: + if not args.fetch: + raise RedmineError( + f"No cached contact #{cid}. Run contacts-sync or contact-show --fetch." + ) + client = client_from_args(args) + contact = client.get_contact(cid) + ccache.write_contact(contact) + if args.json: + print(json.dumps(contact, indent=2, sort_keys=True)) + else: + name = contact_display_name(contact) if contact_display_name else contact.get("name") + email = primary_email(contact) if primary_email else contact.get("email") + print(f"id={contact.get('id')}") + print(f"name={name}") + print(f"email={email}") + print(f"company={contact.get('company') or ''}") + print(f"updated_on={contact.get('updated_on') or ''}") + return 0 + + if args.command == "ticket-contact": + if contact_id_from_issue is None: + raise RedmineError("Contact bridge helper unavailable.") + issue = load_cached_issue(cache, int(args.issue_id)) + cid = contact_id_from_issue(issue) + if cid is None: + raise RedmineError( + f"Ticket #{args.issue_id} has no helpdesk contact id in cache. " + "Sync with journals,helpdesk or open a non-helpdesk issue?" + ) + if args.json: + print( + json.dumps( + { + "issue_id": int(args.issue_id), + "contact_id": cid, + "from_address": ( + (issue.get("helpdesk_ticket") or {}).get("from_address") + if isinstance(issue.get("helpdesk_ticket"), dict) + else None + ), + }, + indent=2, + sort_keys=True, + ) + ) + else: + print(f"issue_id={args.issue_id}") + print(f"contact_id={cid}") + print(f"handoff_hint=contact-show --contact-id {cid}") + print(f"handoff_hint=contact-set --contact-id {cid} --add-tag maybe-customer") + print( + "handoff_hint_full_crm=" + f"redmine_contacts.py --base-url {args.base_url} --project {args.project} " + f"show --contact-id {cid}" + ) + # --fetch implies show; alone it used to silently do nothing (bug). + want_show = bool(getattr(args, "show", False) or getattr(args, "fetch", False)) + want_fetch = bool(getattr(args, "fetch", False)) + if want_show: + if ContactCache is None: + raise RedmineError("Contact helpers unavailable.") + ccache = ContactCache(Path(args.cache_dir)) + contact: dict[str, Any] + if want_fetch: + client = client_from_args(args) + contact = client.get_contact(cid) + ccache.write_contact(contact) + print(f"fetched_contact=yes contact_id={cid}") + else: + try: + contact = ccache.load_contact(cid) + except FileNotFoundError as exc: + raise RedmineError( + f"Contact #{cid} not in local contacts cache. " + "Re-run with --fetch (or contacts-sync / contact-show --fetch)." + ) from exc + name = contact_display_name(contact) if contact_display_name else contact.get("name") + email = primary_email(contact) if primary_email else contact.get("email") + print(f"name={name}") + print(f"email={email}") + print(f"company={contact.get('company') or ''}") + return 0 + + if args.command == "contact-set": + if ContactCache is None: + raise RedmineError("Contact helpers unavailable.") + cid = int(args.contact_id) + # Prefer cache, fall back to API for tag merge base + contact: dict[str, Any] | None = None + ccache = ContactCache(Path(args.cache_dir)) + try: + contact = ccache.load_contact(cid) + except FileNotFoundError: + contact = None + if contact is None and (args.add_tag or args.remove_tag or args.apply): + client = client_from_args(args) + contact = client.get_contact(cid) + ccache.write_contact(contact) + + fields: dict[str, Any] = {} + for assignment in list(args.contact_sets or []): + if "=" not in assignment: + raise RedmineError(f"Invalid --set value: {assignment!r} (want KEY=VALUE)") + key, value = assignment.split("=", 1) + key = key.strip() + if not key: + raise RedmineError(f"Invalid --set value: {assignment!r}") + fields[key] = value + if args.company is not None: + fields["company"] = args.company + if args.first_name is not None: + fields["first_name"] = args.first_name + if args.last_name is not None: + fields["last_name"] = args.last_name + if args.add_tag or args.remove_tag: + if contact is None: + raise RedmineError( + "Tag changes need a contact base payload. Fetch failed or contact missing." + ) + fields["tag_list"] = merge_tags( + extract_tag_list(contact), + list(args.add_tag or []), + list(args.remove_tag or []), + ) + if not fields: + raise RedmineError( + "contact-set requires at least one of: --add-tag, --remove-tag, " + "--company, --first-name, --last-name, --set KEY=VALUE" + ) + + print("ACTION: CONTACT UPDATE") + print(" emails_customer: NO") + print(f" contact_id: {cid}") + print(f" fields: {json.dumps(fields, sort_keys=True)}") + if not args.apply: + if add_draft is None: + raise RedmineError("Draft helpers unavailable.") + path = add_draft( + Path(args.cache_dir), + contact_id=cid, + kind=KIND_CONTACT_FIELDS, + fields=fields, + project=args.project, + base_url=args.base_url, + ) + print_draft_queued( + path, + kind=KIND_CONTACT_FIELDS, + contact_id=cid, + base_url=args.base_url, + emails_customer=False, + ) + return 0 + if assert_write_target_allowed is None: + raise RedmineError("Write safety helpers unavailable.") + try: + assert_write_target_allowed(args.base_url, allow_prod_writes=allow_prod_writes_enabled(args)) + except RuntimeError as exc: + raise RedmineError(str(exc)) from exc + client = client_from_args(args) + client.update_contact(cid, fields, impersonate_user=args.impersonate_user) + print(f"applied contact update on contact #{cid} (no customer email)") + if args.refresh_cache: + fresh = client.get_contact(cid) + ccache.write_contact(fresh) + print(f"refreshed local contact cache for #{cid}") + return 0 + + if args.command == "attachments": + if attachments_from_issue is None or pick_attachment is None: + raise RedmineError("Attachment helpers unavailable (redmine_ticket_attachments.py).") + cmd = args.attachments_command + if cmd is None: + raise RedmineError("attachments requires subcommand: list | download | upload") + + def load_issue_atts(issue_id: int, refresh: bool) -> tuple[dict[str, Any], list[dict[str, Any]]]: + issue: dict[str, Any] | None = None + if not refresh: + try: + issue = load_cached_issue(cache, issue_id) + except RedmineError: + issue = None + if refresh or issue is None or not attachments_from_issue(issue or {}): + client = client_from_args(args) + issue = client.get_issue_with_includes( + issue_id, ("attachments", "journals", "helpdesk") + ) + # best-effort write-through to ticket cache + try: + cache.write_issue(issue) + except Exception: + pass + assert issue is not None + return issue, attachments_from_issue(issue) + + if cmd == "list": + issue, atts = load_issue_atts(int(args.issue_id), bool(args.refresh)) + if args.json: + print(json.dumps(atts, indent=2, sort_keys=True)) + return 0 + print(f"issue_id={args.issue_id} attachments={len(atts)}", file=sys.stderr) + for att in atts: + print( + f"#{att.get('id')}\t{att.get('filesize') or att.get('file_size') or ''}\t" + f"{att.get('content_type') or ''}\t{att.get('filename') or att.get('filename')}" + ) + if not atts: + print( + "No attachments in payload. Try: attachments list --issue-id N --refresh", + file=sys.stderr, + ) + return 0 + + if cmd == "download": + if download_attachment_bytes is None or download_url_for_attachment is None: + raise RedmineError("Download helpers unavailable.") + if safe_filename is None or attachment_dir is None: + raise RedmineError("Path helpers unavailable.") + issue_id = int(args.issue_id) + _issue, atts = load_issue_atts(issue_id, bool(args.refresh)) + if args.all: + targets = atts + else: + try: + targets = [pick_attachment(atts, args.attachment_id)] + except LookupError as exc: + raise RedmineError(str(exc)) from exc + out_dir = args.out_dir or attachment_dir(Path(args.cache_dir), args.project, issue_id) + out_dir = Path(out_dir) + out_dir.mkdir(parents=True, exist_ok=True) + api_key = os.getenv("REDMINE_API_KEY") + if not api_key: + raise RedmineError("REDMINE_API_KEY required to download attachments.") + for att in targets: + url = download_url_for_attachment(att, args.base_url) + name = safe_filename(str(att.get("filename") or f"attachment-{att.get('id')}")) + dest = out_dir / name + # avoid clobber if same name different id + if dest.exists() and att.get("id") is not None: + dest = out_dir / f"{att.get('id')}_{name}" + data = download_attachment_bytes(url=url, api_key=api_key) + dest.write_bytes(data) + print(f"downloaded attachment_id={att.get('id')} path={dest} bytes={len(data)}") + return 0 + + if cmd == "upload": + if upload_file_token is None: + raise RedmineError("Upload helpers unavailable.") + path = Path(args.file) + if not path.is_file(): + raise RedmineError(f"File not found: {path}") + print("ACTION: ATTACHMENT UPLOAD") + print(" emails_customer: NO") + print(f" issue_id: {args.issue_id}") + print(f" file: {path}") + print(f" size_bytes: {path.stat().st_size}") + if not args.apply: + print("dry_run=yes (pass --apply to upload+attach)") + print( + f"apply_hint=attachments upload --issue-id {args.issue_id} " + f"--file {path} --apply" + ) + return 0 + if assert_write_target_allowed is None: + raise RedmineError("Write safety helpers unavailable.") + try: + assert_write_target_allowed( + args.base_url, allow_prod_writes=allow_prod_writes_enabled(args) + ) + except RuntimeError as exc: + raise RedmineError(str(exc)) from exc + api_key = os.getenv("REDMINE_API_KEY") + if not api_key: + raise RedmineError("REDMINE_API_KEY required for upload.") + upload = upload_file_token( + base_url=args.base_url, + api_key=api_key, + path=path, + filename=args.filename, + impersonate_user=args.impersonate_user, + ) + if args.description: + upload["description"] = args.description + client = client_from_args(args) + client.attach_upload_to_issue( + int(args.issue_id), + upload, + notes=args.notes, + impersonate_user=args.impersonate_user, + ) + print( + f"attached token={upload.get('token')} filename={upload.get('filename')} " + f"to issue #{args.issue_id}" + ) + return 0 + + raise RedmineError(f"Unknown attachments subcommand: {cmd}") + + if args.command == "entities": + if extract_entities_from_issue is None: + raise RedmineError("Entity helpers unavailable (redmine_ticket_entities.py).") + ecmd = args.entities_command + if ecmd is None: + raise RedmineError("entities requires subcommand: show | scan | nmi") + if ecmd == "show": + issue = load_cached_issue(cache, int(args.issue_id)) + ents = extract_entities_from_issue(issue) + if args.json: + print(json.dumps(ents, indent=2, sort_keys=True)) + else: + print(f"issue_id={args.issue_id} entities={len(ents)}") + for e in ents: + print(f"{e.get('type')}\t{e.get('value')}\t{e.get('confidence')}") + if not ents: + print("(no pattern hits)", file=sys.stderr) + return 0 + if ecmd == "scan": + issues = select_cached_issues_for_cli(cache, args) + # select uses where/max from args - ensure max_tickets/order present + hits_out = [] + for issue in issues: + ents = extract_entities_from_issue(issue) + if getattr(args, "entity_type", None): + ents = [e for e in ents if e.get("type") == args.entity_type] + if not ents: + continue + iid = issue.get("id") + if args.ids_only: + print(iid) + elif args.jsonl: + for e in ents: + print( + json.dumps( + { + "issue_id": iid, + "subject": issue.get("subject"), + **e, + }, + sort_keys=True, + ) + ) + else: + line = format_entities_line(ents) if format_entities_line else str(ents) + subj = str(issue.get("subject") or "").replace("\n", " ")[:70] + print(f"#{iid}\t{line}\t{subj}") + hits_out.append(iid) + print(f"tickets_with_hits={len(hits_out)}", file=sys.stderr) + return 0 + if ecmd == "nmi": + if brief_for_issue is None or format_private_note is None: + raise RedmineError("NMI bridge unavailable (redmine_ticket_nmi.py).") + issue = load_cached_issue(cache, int(args.issue_id)) + ents = extract_entities_from_issue(issue) + kwargs = {} + if getattr(args, "nmi_mode", None): + kwargs["mode"] = args.nmi_mode + if getattr(args, "nmi_env_file", None): + kwargs["env_file"] = args.nmi_env_file + if getattr(args, "nmi_bin", None): + kwargs["nmi_bin"] = args.nmi_bin + if getattr(args, "nmi_endpoint", None): + kwargs["endpoint"] = args.nmi_endpoint + # History is fetched inside when item_no present; respect explicit --no-history + if not getattr(args, "with_history", True): + # caller can still pass to skip in future + pass + payload = brief_for_issue(issue, ents, **kwargs) + fmt = getattr(args, "format", "brief") or "brief" + note_body = format_private_note(payload) + + if getattr(args, "queue_private_note", False) or getattr(args, "apply_private_note", False): + if add_draft is None: + raise RedmineError("Draft helpers unavailable.") + draft_path = add_draft( + Path(args.cache_dir), + issue_id=int(args.issue_id), + kind=KIND_INTERNAL_NOTE, + body=note_body, + private=True, + base_url=getattr(args, "base_url", None), + project=getattr(args, "project", None), + ) + draft_id = draft_path.stem if hasattr(draft_path, "stem") else Path(str(draft_path)).stem + if print_draft_queued is not None: + print_draft_queued( + draft_path, + kind=KIND_INTERNAL_NOTE, + issue_id=int(args.issue_id), + base_url=getattr(args, "base_url", None), + emails_customer=False, + ) + else: + print(f"draft_id={draft_id}") + print(f"draft_path={draft_path}") + print(f"issue_id={args.issue_id}") + print(f"kind={KIND_INTERNAL_NOTE}") + print("emails_customer=NO") + print(f"apply_hint=draft apply {draft_id}") + + if getattr(args, "apply_private_note", False): + if assert_write_target_allowed is None: + raise RedmineError("Write safety helpers unavailable.") + try: + assert_write_target_allowed( + args.base_url, + allow_prod_writes=allow_prod_writes_enabled(args), + ) + except RuntimeError as exc: + raise RedmineError(str(exc)) from exc + client = client_from_args(args) + _, draft_data = load_draft(Path(args.cache_dir), draft_id) + apply_draft_to_redmine( + client, + draft_data, + impersonate_user=getattr(args, "impersonate_user", None) or "rebot", + allow_customer_email=False, + ) + if mark_draft_applied is not None: + mark_draft_applied(Path(args.cache_dir), draft_id) + try: + c = TicketCache(Path(args.cache_dir), getattr(args, "project", "customer-service")) + refresh_cached_issue(c, client, int(args.issue_id)) + except Exception: + pass + print(f"applied private NMI brief note on #{args.issue_id} (no customer email)", file=sys.stderr) + + if fmt == "json": + print(json.dumps(payload, indent=2, sort_keys=True, default=str)) + elif fmt == "note": + print(note_body, end="" if note_body.endswith("\n") else "\n") + else: + brief = (payload.get("brief") or {}) + if brief.get("ok") and brief.get("text"): + print(brief["text"], end="" if str(brief["text"]).endswith("\n") else "\n") + else: + print(note_body, end="" if note_body.endswith("\n") else "\n") + return 1 if not payload.get("ok") else 0 + return 0 if payload.get("ok") else 1 + raise RedmineError(f"Unknown entities subcommand: {ecmd}") + + + if args.command == "tui": + if run_tui_cli is None: + raise RedmineError("TUI unavailable (redmine_ticket_tui.py). Run with python -c or check import.") + return run_tui_cli(args, cache) + + if args.command == "textual-tui": + if run_textual_tui_cli is None: + raise RedmineError("Textual TUI unavailable (redmine_ticket_textual_tui.py).") + return run_textual_tui_cli(args, cache) + + if args.command == "browse": + if run_browse_loop is None: + raise RedmineError("Browse loop helper unavailable (redmine_ticket_browse.py).") + matches = select_cached_issues_for_cli(cache, args) + if not matches: + print("No matching tickets.", file=sys.stderr) + return 0 + + def make_handlers(issue_id: int, issue: dict[str, Any]) -> dict[str, Callable[[], None]]: + def do_note(private: bool = False) -> None: + if add_draft is None or prompt_multiline is None: + raise RedmineError("Draft/note helpers unavailable.") + body = prompt_multiline( + "Internal note body" + (" (PRIVATE)" if private else "") + ) + if not body: + print("cancelled", file=sys.stderr) + return + path = add_draft( + Path(args.cache_dir), + issue_id=int(issue_id), + kind=KIND_INTERNAL_NOTE, + body=body, + private=private, + project=args.project, + base_url=args.base_url, + ) + print_draft_queued( + path, + kind=KIND_INTERNAL_NOTE, + issue_id=int(issue_id), + base_url=args.base_url, + emails_customer=False, + ) + if announce_draft: + announce_draft(path) + else: + print("Queued only — apply later with: draft apply ", file=sys.stderr) + + def do_fields() -> None: + if add_draft is None: + raise RedmineError("Draft helpers unavailable.") + print("Field update (name-based). Blank line cancels.") + status = input("status name (e.g. Closed) or empty: ").strip() + assignee = input("assignee name/login or 'unassigned' or empty: ").strip() + notes = input("optional internal notes: ").strip() + # Build a fake namespace for resolver + ns = argparse.Namespace( + field_status_id=None, + status_name=status or None, + assigned_to_id=None, + assignee_name=assignee or None, + priority_id=None, + priority_name=None, + category_id=None, + category_name=None, + due_date=None, + subject=None, + notes=notes or None, + private=False, + refresh_meta=False, + base_url=args.base_url, + project=args.project, + cache_dir=args.cache_dir, + api_key=os.getenv("REDMINE_API_KEY"), + ) + if not any([status, assignee, notes]): + print("cancelled", file=sys.stderr) + return + resolved = resolve_set_fields_names(ns) + fields = build_fields_payload_from_args(ns, resolved_ids=resolved) + if not fields: + print("no fields to set", file=sys.stderr) + return + path = add_draft( + Path(args.cache_dir), + issue_id=int(issue_id), + kind=KIND_FIELDS, + fields=fields, + project=args.project, + base_url=args.base_url, + ) + print_draft_queued( + path, + kind=KIND_FIELDS, + issue_id=int(issue_id), + base_url=args.base_url, + emails_customer=False, + ) + if announce_draft: + announce_draft(path) + + def do_tag() -> None: + if control_tag_add is None: + raise RedmineError("Control-tag helpers unavailable.") + tag = input("local control-tag name: ").strip() + if not tag: + print("cancelled", file=sys.stderr) + return + ann = input("annotation (optional): ").strip() + detail = control_tag_add( + Path(args.cache_dir), + tag=tag, + issue_id=int(issue_id), + properties=["browse"], + annotation=ann, + ) + print(json.dumps(detail, indent=2, sort_keys=True)) + + def do_contact() -> None: + if contact_id_from_issue is None: + raise RedmineError("Contact bridge unavailable.") + cid = contact_id_from_issue(issue) + print(f"issue_id={issue_id}") + print(f"contact_id={cid}") + if cid is not None: + print(f"handoff=contact-show --contact-id {cid}") + print(f"handoff=contact-set --contact-id {cid} --add-tag maybe-customer") + + def do_entities() -> None: + if extract_entities_from_issue is None: + raise RedmineError("Entity helpers unavailable.") + ents = extract_entities_from_issue(issue) + if not ents: + print("(no pattern hits)") + return + for e in ents: + print(f"{e.get('type')}\t{e.get('value')}\t{e.get('confidence')}") + + def do_nmi() -> None: + if extract_entities_from_issue is None or brief_for_issue is None or format_private_note is None: + raise RedmineError("NMI helpers unavailable.") + if add_draft is None: + raise RedmineError("Draft helpers unavailable.") + print("Running NMI brief (may take a few seconds)…", file=sys.stderr) + ents = extract_entities_from_issue(issue) + payload = brief_for_issue(issue, ents) + note_body = format_private_note(payload) + print(note_body[:2000]) + if not payload.get("ok"): + print( + f"NMI lookup incomplete ({payload.get('error')}); " + "still queueing private note with attempt log.", + file=sys.stderr, + ) + ans = input("Queue as PRIVATE internal note draft? [Y/n] ").strip().lower() + if ans in {"n", "no"}: + print("cancelled", file=sys.stderr) + return + path = add_draft( + Path(args.cache_dir), + issue_id=int(issue_id), + kind=KIND_INTERNAL_NOTE, + body=note_body, + private=True, + project=args.project, + base_url=args.base_url, + ) + print_draft_queued( + path, + kind=KIND_INTERNAL_NOTE, + issue_id=int(issue_id), + base_url=args.base_url, + emails_customer=False, + ) + if announce_draft: + announce_draft(path) + + def do_id() -> None: + print(issue_id) + + return { + "n": lambda: do_note(False), + "p": lambda: do_note(True), + "f": do_fields, + "t": do_tag, + "c": do_contact, + "e": do_entities, + "m": do_nmi, + "i": do_id, + } + + return int( + run_browse_loop( + matches, + cache_dir=Path(args.cache_dir), + project=args.project, + pager=str(getattr(args, "pager", "auto") or "auto"), + once=bool(getattr(args, "once", False)), + preview=not bool(getattr(args, "no_preview", False)), + action_factory=make_handlers, + ) + ) + + if args.command == "categories": + cmd = args.categories_command + if cmd is None: + raise RedmineError("categories requires subcommand: list") + if cmd == "list": + cats: list[dict[str, Any]] = [] + if args.refresh or load_meta_lists is None: + if fetch_project_categories is None: + raise RedmineError("Category helpers unavailable.") + client = client_from_args(args) + cats = fetch_project_categories(client, args.project) + # best-effort meta write-through + if refresh_meta_cache is not None: + try: + refresh_meta_cache( + client, Path(args.cache_dir), args.project, force=True + ) + except Exception: + pass + else: + lists = load_meta_lists(Path(args.cache_dir), args.project) + cats = list(lists.get("categories") or []) + if not cats and fetch_project_categories is not None: + client = client_from_args(args) + cats = fetch_project_categories(client, args.project) + if args.json: + print(json.dumps(cats, indent=2, sort_keys=True)) + return 0 + print(f"project={args.project} categories={len(cats)}", file=sys.stderr) + for c in cats: + print(f"#{c.get('id')}\t{c.get('name')}") + return 0 + raise RedmineError(f"Unknown categories subcommand: {cmd}") + + if args.command == "ocr-rename": + if ( + extract_text_from_file is None + or suggest_subject_from_text is None + or attachment_is_ocr_candidate is None + ): + raise RedmineError("OCR helpers unavailable (redmine_ticket_ocr.py).") + if download_attachment_bytes is None or download_url_for_attachment is None: + raise RedmineError("Attachment download helpers unavailable.") + if attachment_dir is None or safe_filename is None: + raise RedmineError("Attachment path helpers unavailable.") + + targets: list[dict[str, Any]] = [] + if args.issue_id is not None: + try: + targets = [load_cached_issue(cache, int(args.issue_id))] + except RedmineError: + client = client_from_args(args) + targets = [ + client.get_issue_with_includes( + int(args.issue_id), ("attachments", "journals", "helpdesk") + ) + ] + elif args.where: + targets = select_cached_issues_for_cli(cache, args) + else: + raise RedmineError("ocr-rename requires --issue-id or --where.") + + api_key = os.getenv("REDMINE_API_KEY") + if not api_key: + raise RedmineError("REDMINE_API_KEY required to download attachments for OCR.") + + results: list[dict[str, Any]] = [] + for issue in targets: + iid = int(issue.get("id") or 0) + if not iid: + continue + subj = str(issue.get("subject") or "") + if args.only_generic_subjects and looks_like_generic_scan_subject and not looks_like_generic_scan_subject(subj): + results.append({"issue_id": iid, "skipped": "subject_not_generic", "subject": subj}) + continue + + # ensure attachments present + atts = issue.get("attachments") if isinstance(issue.get("attachments"), list) else [] + if not atts: + client = client_from_args(args) + issue = client.get_issue_with_includes(iid, ("attachments", "journals", "helpdesk")) + try: + cache.write_issue(issue) + except Exception: + pass + atts = issue.get("attachments") if isinstance(issue.get("attachments"), list) else [] + + cand_atts = [a for a in atts if isinstance(a, dict) and attachment_is_ocr_candidate(a)] + if not cand_atts: + results.append({"issue_id": iid, "skipped": "no_ocr_attachments", "subject": subj}) + continue + + out_dir = attachment_dir(Path(args.cache_dir), args.project, iid) + file_results: list[dict[str, str]] = [] + combined_text_parts: list[str] = [] + for att in cand_atts[:5]: + name = safe_filename(str(att.get("filename") or f"att-{att.get('id')}")) + dest = out_dir / name + try: + if not dest.is_file(): + from redmine_ticket_attachments import download_url_candidates + urls = download_url_candidates(att, args.base_url) + data = download_attachment_bytes( + url=urls[0], + api_key=api_key, + fallback_urls=urls[1:], + ) + dest.write_bytes(data) + text, method = extract_text_from_file(dest) + except Exception as exc: + file_results.append( + { + "filename": name, + "method": "error", + "text": "", + "path": str(dest), + "error": str(exc)[:200], + } + ) + continue + file_results.append( + { + "filename": name, + "method": method, + "text": text, + "path": str(dest), + } + ) + if text: + combined_text_parts.append(text) + + combined = "\n\n".join(combined_text_parts) + proposed = suggest_subject_from_text(combined, original_subject=subj) + row: dict[str, Any] = { + "issue_id": iid, + "old_subject": subj, + "proposed_subject": proposed, + "files": [ + { + "filename": f["filename"], + "method": f["method"], + "chars": len(f.get("text") or ""), + "path": f["path"], + } + for f in file_results + ], + "drafts": {}, + } + + if args.queue_subject and proposed and proposed != subj: + if args.apply_subject: + if assert_write_target_allowed is None: + raise RedmineError("Write safety helpers unavailable.") + try: + assert_write_target_allowed( + args.base_url, + allow_prod_writes=allow_prod_writes_enabled(args), + ) + except RuntimeError as exc: + raise RedmineError(str(exc)) from exc + client = client_from_args(args) + client.update_issue( + iid, + {"subject": proposed}, + impersonate_user=args.impersonate_user, + ) + row["drafts"]["subject_applied"] = True + + # Keep local cache in sync with the live ticket we just mutated + try: + if refresh_cached_issue: + c = TicketCache(Path(args.cache_dir), args.project) + refresh_cached_issue(c, client, int(iid)) + except Exception: + pass + elif add_draft is not None: + path = add_draft( + Path(args.cache_dir), + issue_id=iid, + kind=KIND_FIELDS, + fields={"subject": proposed}, + project=args.project, + base_url=args.base_url, + ) + row["drafts"]["subject_draft_id"] = path.stem + if not args.json: + print_draft_queued( + path, + kind=KIND_FIELDS, + issue_id=iid, + base_url=args.base_url, + emails_customer=False, + ) + + if args.queue_note and build_ocr_note is not None and add_draft is not None: + note_body = build_ocr_note( + issue_id=iid, + files=file_results, + proposed_subject=proposed, + ) + path = add_draft( + Path(args.cache_dir), + issue_id=iid, + kind=KIND_INTERNAL_NOTE, + body=note_body, + private=True, + project=args.project, + base_url=args.base_url, + ) + row["drafts"]["note_draft_id"] = path.stem + if not args.json: + print_draft_queued( + path, + kind=KIND_INTERNAL_NOTE, + issue_id=iid, + base_url=args.base_url, + emails_customer=False, + ) + + results.append(row) + if not args.json: + print( + f"#{iid}\told={subj!r}\n proposed={proposed!r}\n" + f" files={len(file_results)} drafts={row['drafts']}" + ) + + if args.json: + print(json.dumps(results, indent=2, sort_keys=True)) + else: + print(f"ocr_rename_done tickets={len(results)}", file=sys.stderr) + return 0 + + if args.command == "control-tag": + if control_tag_add is None or control_tag_list is None: + raise RedmineError("Control-tag helpers unavailable (redmine_ticket_control_tags.py).") + cmd = args.control_tag_command + if cmd is None: + raise RedmineError( + "control-tag requires subcommand: add|remove|list|show|for-issue|suggest|wipe" + ) + cache_path = Path(args.cache_dir) + if cmd == "add": + detail = control_tag_add( + cache_path, + tag=args.tag, + issue_id=int(args.issue_id), + properties=list(args.properties or []), + annotation=args.annotation or "", + ) + print(f"tagged #{args.issue_id} with local control-tag {args.tag!r}") + print(json.dumps(detail, indent=2, sort_keys=True)) + return 0 + if cmd == "remove": + if control_tag_remove is None: + raise RedmineError("Control-tag remove helper unavailable.") + ok = control_tag_remove(cache_path, tag=args.tag, issue_id=int(args.issue_id)) + if not ok: + raise RedmineError(f"No local tag {args.tag!r} on #{args.issue_id}") + print(f"removed local control-tag {args.tag!r} from #{args.issue_id}") + return 0 + if cmd == "list": + for row in control_tag_list(cache_path): + print(f"{row['tag']}\tcount={row['ticket_count']}\tids={row['issue_ids'][:20]}") + return 0 + if cmd == "show": + ids = tickets_for_tag(cache_path, args.tag) if tickets_for_tag else [] + for iid in ids: + print(iid) + return 0 + if cmd == "for-issue": + tags = tags_for_issue(cache_path, int(args.issue_id)) if tags_for_issue else [] + print(",".join(tags) if tags else "") + return 0 + if cmd == "suggest": + if filter_by_query is None or control_tag_add is None: + raise RedmineError("Suggest requires query + control-tag helpers.") + where = list(args.where or []) + if not where: + # default: subject includes tag name + where = [f'subject.includes("{args.tag}")'] + issues = list(cache.iter_cached_issues()) + try: + candidates = filter_by_query(issues, expressions=where) + except QueryError as exc: + raise RedmineError(f"Invalid --where for suggest: {exc}") from exc + candidates = select_issues( + candidates, + order="newest-updated-first", + max_tickets=args.max_tickets, + ) + already = set(tickets_for_tag(cache_path, args.tag) if tickets_for_tag else []) + proposed = [i for i in candidates if i.get("id") is not None and int(i["id"]) not in already] + print( + f"suggest tag={args.tag!r} candidates={len(candidates)} " + f"already_tagged={len(already)} new={len(proposed)} apply={bool(args.apply)}" + ) + for issue in proposed: + iid = int(issue["id"]) + subj = str(issue.get("subject") or "").replace("\n", " ")[:80] + print(f"#{iid}\t{subj}") + if args.apply: + control_tag_add( + cache_path, + tag=args.tag, + issue_id=iid, + properties=list(args.properties or ["subject"]), + annotation=f"auto-suggest where={where}", + ) + if args.apply: + print(f"applied_local_tags={len(proposed)}") + return 0 + if cmd == "export": + if export_control_tag_jsonl is None: + raise RedmineError("Export helper unavailable (redmine_ticket_control_export.py).") + from redmine_ticket_control_tags import load_store as _load_store + store = _load_store(cache_path) + issues_by_id = {} + for issue in cache.iter_cached_issues(): + if issue.get("id") is not None: + issues_by_id[int(issue["id"])] = issue + n = export_control_tag_jsonl( + cache_dir=cache_path, + tags_payload=store, + issues_by_id=issues_by_id, + tag=args.tag, + out_path=Path(args.out), + ) + print(f"exported_records={n} path={args.out}") + return 0 + if cmd == "wipe": + if not getattr(args, "i_understand_this_wipes_local_control_tags", False): + raise RedmineError("wipe requires confirmation flag") + path = control_tag_wipe(cache_path) if control_tag_wipe else None + print(f"wiped local control-tag store: {path}") + return 0 + raise RedmineError(f"Unknown control-tag subcommand: {cmd}") + + if args.command == "customer-email": + # Flag is required=True on parser; keep explicit checks for safety. + if not getattr(args, "i_understand_this_emails_the_customer", False): + raise RedmineError( + "customer-email requires --i-understand-this-emails-the-customer. " + "This command is for emailing the customer, not internal notes. " + "Use `note` for staff-only journal notes." + ) + body = resolve_body_text(args) + print("ACTION: CUSTOMER EMAIL (Helpdesk)") + print(" emails_customer: YES — this will email the customer when applied") + print(f" issue_id: {args.issue_id}") + print(" private: NOT ALLOWED on this path") + if not args.apply: + if add_draft is None: + raise RedmineError("Draft helpers unavailable.") + path = add_draft( + Path(args.cache_dir), + issue_id=int(args.issue_id), + kind=KIND_CUSTOMER_EMAIL, + body=body, + private=False, + project=args.project, + base_url=args.base_url, + ) + print_draft_queued( + path, + kind=KIND_CUSTOMER_EMAIL, + issue_id=int(args.issue_id), + base_url=args.base_url, + emails_customer=True, + ) + return 0 + if assert_write_target_allowed is None: + raise RedmineError("Write safety helpers unavailable.") + try: + assert_write_target_allowed(args.base_url, allow_prod_writes=allow_prod_writes_enabled(args)) + except RuntimeError as exc: + raise RedmineError(str(exc)) from exc + client = client_from_args(args) + result = client.send_customer_helpdesk_email( + int(args.issue_id), + body, + status_id=getattr(args, "status_id", None), + to_address=getattr(args, "to_address", None), + cc_address=getattr(args, "cc_address", None), + impersonate_user=args.impersonate_user, + ) + print(f"sent customer Helpdesk email for #{args.issue_id}") + print(json.dumps(result, indent=2, sort_keys=True) if result else "(empty response)") + return 0 + + if args.command == "draft": + if add_draft is None or list_drafts is None or cancel_draft is None: + raise RedmineError("Draft helpers unavailable (redmine_ticket_drafts.py).") + if args.draft_command is None: + raise RedmineError("draft requires a subcommand: add | list | cancel | apply") + if args.draft_command == "add": + body = resolve_body_text(args) + try: + kind = normalize_kind(args.kind) if normalize_kind else args.kind + except ValueError as exc: + raise RedmineError(str(exc)) from exc + if kind == KIND_CUSTOMER_EMAIL: + print("WARNING: queuing CUSTOMER EMAIL draft (will email when applied).") + else: + print("Queuing INTERNAL NOTE draft (will NOT email when applied).") + try: + path = add_draft( + Path(args.cache_dir), + issue_id=int(args.issue_id), + kind=str(kind), + body=body, + private=bool(args.private), + project=args.project, + base_url=args.base_url, + ) + except ValueError as exc: + raise RedmineError(str(exc)) from exc + print_draft_queued( + path, + kind=str(kind), + issue_id=int(args.issue_id), + base_url=args.base_url, + emails_customer=(str(kind) == KIND_CUSTOMER_EMAIL), + ) + return 0 + if args.draft_command == "list": + status = None if args.all else "pending" + for draft in list_drafts(Path(args.cache_dir), status=status): + preview = (draft.get("body") or "").replace("\n", " ")[:80] + emails = "EMAILS_CUSTOMER" if draft.get("kind") == KIND_CUSTOMER_EMAIL or draft.get("emails_customer") else "no-email" + target = ( + f"contact#{draft.get('contact_id')}" + if draft.get("contact_id") is not None + else f"#{draft.get('issue_id')}" + ) + print( + f"{draft.get('id')}\t{draft.get('status')}\t{target}\t" + f"{draft.get('kind')}\t{emails}\tbase_url={draft.get('base_url') or ''}\t" + f"private={draft.get('private')}\t{preview}" + ) + return 0 + if args.draft_command == "cancel": + if args.all_pending: + n = cancel_all_pending(Path(args.cache_dir)) if cancel_all_pending else 0 + print(f"cancelled_pending={n}") + return 0 + if not args.draft_id: + raise RedmineError("draft cancel requires draft_id or --all-pending") + ok = cancel_draft(Path(args.cache_dir), args.draft_id) + if not ok: + raise RedmineError(f"Draft not found: {args.draft_id}") + print(f"cancelled {args.draft_id}") + return 0 + if args.draft_command == "apply": + if load_draft is None or mark_draft_applied is None or assert_write_target_allowed is None: + raise RedmineError("Draft apply helpers unavailable.") + try: + _path, draft = load_draft(Path(args.cache_dir), args.draft_id) + except FileNotFoundError as exc: + raise RedmineError(str(exc)) from exc + if draft.get("status") != "pending": + raise RedmineError(f"Draft {draft.get('id')} is not pending (status={draft.get('status')})") + kind = normalize_kind(str(draft.get("kind"))) if normalize_kind else str(draft.get("kind")) + apply_base = resolve_apply_base_url(args, draft) + print(f"Applying draft {draft.get('id')}") + print(f" kind: {kind_label(kind) if kind_label else kind}") + if draft.get("issue_id") is not None: + print(f" issue_id: {draft.get('issue_id')}") + if draft.get("contact_id") is not None: + print(f" contact_id: {draft.get('contact_id')}") + print(f" base_url: {apply_base}") + print(f" emails_customer: {kind == KIND_CUSTOMER_EMAIL}") + try: + assert_write_target_allowed( + apply_base, + allow_prod_writes=allow_prod_writes_enabled(args), + ) + except RuntimeError as exc: + raise RedmineError(str(exc)) from exc + # Point client at draft host + args.base_url = apply_base + client = client_from_args(args) + try: + result = apply_draft_to_redmine( + client, + draft, + impersonate_user=args.impersonate_user, + allow_customer_email=bool(args.i_understand_this_emails_the_customer), + ) + mark_draft_applied(Path(args.cache_dir), str(draft["id"]), result=result) + + # Refresh local cache immediately so browse / tui / listings see the live state + # (status, journals, updated_on, etc.). This matches "update existing cache files on material change". + try: + issue_id = draft.get("issue_id") + if issue_id and refresh_cached_issue: + c = TicketCache(Path(args.cache_dir), args.project) + refresh_cached_issue(c, client, int(issue_id)) + except Exception: + pass + + except Exception as exc: + if mark_draft_error is not None: + mark_draft_error(Path(args.cache_dir), str(draft["id"]), str(exc)) + raise + print(f"applied draft {draft.get('id')}: {json.dumps(result, sort_keys=True)}") + return 0 + raise RedmineError(f"Unknown draft subcommand: {args.draft_command}") + + if args.command == "spam-check": + results = run_spam_check(cache, apply=args.apply, threshold=args.threshold) + print_spam_results(results) + print(f"Spam check complete: checked={len(results)} apply={args.apply}") + if not args.apply: + print("Dry run only. Re-run with --apply to record spam-check idempotency markers.") + return 0 + + if args.command == "close-if-spam": + client = client_from_args(args) + plans = run_close_if_spam( + client, + cache, + close_status_id=args.close_status_id, + contact_ticket_limit=args.contact_ticket_limit, + apply=args.apply, + threshold=args.threshold, + impersonate_user=args.impersonate_user, + ) + print_close_if_spam_plans(plans) + print(f"Close-if-spam complete: planned={len(plans)} apply={args.apply}") + if not args.apply: + print("Dry run only. Re-run with --apply to close Redmine issues and record idempotency markers.") + return 0 + + if args.command == "tag-maybe-customer": + client = client_from_args(args) + results = run_tag_maybe_customer_contacts( + client, + cache, + apply=args.apply, + tag_name=args.tag, + impersonate_user=args.impersonate_user, + ) + for result in results: + print( + f"contact_id={result['contact_id']}\t" + f"source_issue_id={result['source_issue_id']}\t" + f"tag_list={json.dumps(result['tag_list'])}" + ) + print(f"Tag-maybe-customer complete: planned={len(results)} apply={args.apply}") + if not args.apply: + print("Dry run only. Re-run with --apply to tag Redmine contacts.") + return 0 + + if args.command == "decompose": + issue = load_issue_for_decompose(args, cache) + plan = decompose_issue_dry_run(issue) + if args.format == "markdown": + print(render_decomposition_markdown(plan)) + else: + print(json.dumps(plan, indent=2, sort_keys=True)) + return 0 + + print(f"error: unhandled command {getattr(args, 'command', None)!r}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/redmine_triage.py b/redmine_triage.py new file mode 100644 index 0000000..baddd0b --- /dev/null +++ b/redmine_triage.py @@ -0,0 +1,316 @@ +#!/usr/bin/env python3 +""" +Lightweight Redmine triage for "tickets that truly need attention". + +Designed for fast iteration. Run manually or from cron/magitek. + +Usage examples: + ./bin/rtl triage + ./bin/rtl triage --open-only --max 30 --jsonl + ./bin/rtl triage --queue-triage-note # adds internal notes with context for the top items + +Output is human-first + machine-friendly so it can feed other agents. +""" + +from __future__ import annotations +import argparse +import json +from dataclasses import dataclass +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Optional + +# Reuse the existing power-user infrastructure +try: + from redmine_tickets import ( + TicketCache, + select_cached_issues_for_cli, + ) +except Exception as e: + raise RuntimeError(f"Could not import redmine power client: {e}") from e + +try: + from redmine_ticket_entities import extract_entities_from_issue, format_entities_line +except Exception: + extract_entities_from_issue = None + format_entities_line = None + +try: + from redmine_ticket_drafts import add_draft, KIND_INTERNAL_NOTE +except Exception: + add_draft = None + KIND_INTERNAL_NOTE = "internal_note" + + +@dataclass +class TriageScore: + issue_id: int + score: float + reasons: list[str] + age_hours: float + last_update: str + status: str + assigned: str + subject: str + context_block: str # ready-to-paste Textile / Markdown context + + +def _parse_dt(s: Optional[str]) -> Optional[datetime]: + if not s: + return None + for fmt in ("%Y-%m-%dT%H:%M:%SZ", "%Y-%m-%dT%H:%M:%S%z", "%Y-%m-%d %H:%M:%S"): + try: + dt = datetime.strptime(s, fmt) + if dt.tzinfo is None: + dt = dt.replace(tzinfo=timezone.utc) + return dt + except Exception: + continue + return None + + +def _hours_since(dt: Optional[datetime]) -> float: + if not dt: + return 999.0 + now = datetime.now(timezone.utc) + return max(0.0, (now - dt).total_seconds() / 3600.0) + + +def _get_status(issue: dict) -> str: + st = issue.get("status") or {} + if isinstance(st, dict): + return st.get("name", "") or "" + return str(st) + + +def _get_assigned(issue: dict) -> str: + a = issue.get("assigned_to") or {} + if isinstance(a, dict): + return a.get("name") or a.get("login") or "" + return str(a or "") + + +def _last_journal_age_and_type(issue: dict) -> tuple[float, str]: + journals = issue.get("journals") or [] + if not journals: + return 999.0, "no-journals" + try: + last = journals[-1] + created = _parse_dt(last.get("created_on")) + age = _hours_since(created) + notes = last.get("notes") or "" + user = (last.get("user") or {}).get("name", "") + is_our_note = "internal" in str(last).lower() or "rebot" in user.lower() + typ = "our-internal" if is_our_note else "customer-or-other" + return age, typ + except Exception: + return 999.0, "parse-err" + + +def score_issue(issue: dict, now: datetime | None = None) -> TriageScore: + iid = int(issue.get("id", 0)) + subject = (issue.get("subject") or "")[:80] + status = _get_status(issue) + assigned = _get_assigned(issue) + + updated = _parse_dt(issue.get("updated_on")) + age_h = _hours_since(updated) + + last_j_age, last_j_type = _last_journal_age_and_type(issue) + + score = 0.0 + reasons: list[str] = [] + + # Core staleness + if age_h > 48: + score += 40 + reasons.append(f"stale>{int(age_h)}h") + elif age_h > 24: + score += 25 + reasons.append(f"stale>{int(age_h)}h") + elif age_h > 8: + score += 12 + reasons.append(f"stale>{int(age_h)}h") + + # Status pressure + st_lower = status.lower() + if "new" in st_lower: + score += 30 + reasons.append("status:new") + elif "feedback" in st_lower: + score += 28 + reasons.append("status:feedback") + elif "progress" in st_lower and age_h > 12: + score += 15 + reasons.append("progress+stale") + + # Journal gap (customer spoke, we didn't reply recently) + if last_j_type != "our-internal" and last_j_age < 72: + if last_j_age < 4: + score += 20 + reasons.append("recent-customer-msg") + elif last_j_age < 24: + score += 10 + reasons.append("customer-msg-no-our-reply") + + # Unassigned pressure + if not assigned or assigned.lower() in ("", "none", "nobody"): + if "new" in st_lower or "feedback" in st_lower: + score += 15 + reasons.append("unassigned+open") + + # Keyword boost (tunable) + text = (subject + " " + str(issue.get("description", ""))).lower() + for kw, pts in [ + ("urgent", 18), ("complaint", 15), ("delay", 10), ("stuck", 12), + ("not received", 12), ("missing", 8), ("wrong", 8), + ]: + if kw in text: + score += pts + reasons.append(f"kw:{kw}") + + # Entity / NMI context is a positive signal (we have something to act on) + if extract_entities_from_issue: + try: + ents = extract_entities_from_issue(issue) + if ents: + score += 8 + reasons.append("has-entities") + except Exception: + pass + + # Build compact context block (Textile-friendly for internal notes) + ctx_lines = [ + f"h4. Triage context for #{iid}", + f"*Status*: {status} | *Assigned*: {assigned or '—'} | *Updated*: {issue.get('updated_on', '')[:16]} ({int(age_h)}h ago)", + ] + if extract_entities_from_issue: + try: + ents = extract_entities_from_issue(issue) + line = format_entities_line(ents) if format_entities_line and ents else "" + if line: + ctx_lines.append(f"*Entities*: {line}") + except Exception: + pass + + # Last journal teaser + journals = issue.get("journals") or [] + if journals: + last = journals[-1] + notes = (last.get("notes") or "")[:200].replace("\n", " ") + user = (last.get("user") or {}).get("name", "?") + ctx_lines.append(f"*Last journal* ({user}): {notes}...") + + ctx_lines.append("") + ctx_lines.append("Suggested: review, add context note, or assign + paint status.") + + context_block = "\n".join(ctx_lines) + + return TriageScore( + issue_id=iid, + score=round(score, 1), + reasons=reasons, + age_hours=round(age_h, 1), + last_update=str(issue.get("updated_on", ""))[:16], + status=status, + assigned=assigned, + subject=subject, + context_block=context_block, + ) + + +def run_triage(args: argparse.Namespace) -> int: + cache = TicketCache(args.cache_dir, args.project) + + # Build a light filter set that favors things that need attention + filt = argparse.Namespace() + filt.project = args.project + filt.cache_dir = args.cache_dir + filt.base_url = args.base_url + filt.allow_prod_writes = getattr(args, "allow_prod_writes", False) + filt.open_only = getattr(args, "open_only", True) + filt.max_tickets = getattr(args, "max_tickets", 200) + filt.updated_since = None + filt.updated_before = None + filt.ids = None + filt.where = [] + filt.control_tag = None + filt.has_attachments = False + + issues = select_cached_issues_for_cli(cache, filt) + + scored: list[TriageScore] = [] + for iss in issues: + try: + ts = score_issue(iss) + if ts.score > 0.5: # only things with real signal + scored.append(ts) + except Exception: + continue + + scored.sort(key=lambda x: (-x.score, x.age_hours)) + + if args.max: + scored = scored[: args.max] + + if args.jsonl: + for s in scored: + print(json.dumps({ + "id": s.issue_id, + "score": s.score, + "reasons": s.reasons, + "subject": s.subject, + "status": s.status, + "assigned": s.assigned, + })) + return 0 + + # Human output + print(f"Triage (project={args.project}, open-ish, scored for attention)\n") + for s in scored: + print(f"#{s.issue_id} score={s.score:5.1f} {s.status[:12]:<12} {s.assigned[:10]:<10} {s.subject}") + print(f" age={s.age_hours:5.1f}h reasons={','.join(s.reasons[:4])}") + if args.show_context: + print(s.context_block) + print("") + + if args.queue_triage_note and add_draft: + for s in scored[: args.queue_limit]: + try: + add_draft( + args.cache_dir, + issue_id=s.issue_id, + kind=KIND_INTERNAL_NOTE, + body=s.context_block, + private=True, + project=args.project, + ) + print(f"Queued triage note for #{s.issue_id}") + except Exception as e: + print(f"Failed to queue for #{s.issue_id}: {e}") + + print(f"\nTotal high-signal: {len(scored)}") + return 0 + + +def add_triage_parser(subparsers: argparse._SubParsersAction) -> None: + p = subparsers.add_parser("triage", help="Score and list tickets that need real attention (fast triage force-multiplier)") + p.add_argument("--project", default="customer-service") + p.add_argument("--cache-dir", default=Path(".cache/redmine_tickets"), type=Path) + p.add_argument("--base-url", default="https://rm.ldrprep.com") + p.add_argument("--open-only", action="store_true", default=True, help="Only open-ish tickets (default true)") + p.add_argument("--max", type=int, default=25, help="Max tickets to show") + p.add_argument("--max-tickets", type=int, default=300, help="How many to pull from cache before scoring") + p.add_argument("--show-context", action="store_true", help="Print the ready-to-paste context block for each") + p.add_argument("--jsonl", action="store_true", help="Machine readable output") + p.add_argument("--queue-triage-note", action="store_true", help="Queue internal note with the triage context (uses draft system)") + p.add_argument("--queue-limit", type=int, default=10, help="How many top tickets to queue notes for") + p.add_argument("--allow-prod-writes", action="store_true") + p.set_defaults(func=run_triage) + + +if __name__ == "__main__": + ap = argparse.ArgumentParser() + add_triage_parser(ap) + args = ap.parse_args() + run_triage(args) diff --git a/roadmap/2026-07-03.md b/roadmap/2026-07-03.md new file mode 100644 index 0000000..4461b04 --- /dev/null +++ b/roadmap/2026-07-03.md @@ -0,0 +1,128 @@ +Having a look at the projects `~/projects/obsidian_vault_maintainers` and '~/projects/redmine/`, I would like to have a set of scripts, cronjobs, or agents/prompts that does something like the following: + + +## First, import the `PROJECT.md` from `configs` and apply it to `~/projects/redmine` + +This is to ensure that further work on this project is documented and worked on in the manner we're beginning to standardize for most of our projects. + + +## Second, critical update to the redmine project script: `redmine_contacts.py` + + +- [ ] Fetch and search should have an option to only show contacts based on optional filters (if possible): + - [ ] Filter what contacts are fetched based on: + - [ ] created_on and/or updated_on + - [ ] presence or lack of a specific tag, "no tag at all", etc +- [ ] Add command to fetch an entire ticket by issue_id. This should be similar to how the `redMCP` handles being asked to retrieve a complete ticket (it should include the journals/notes, helpdesk info, etc. as well as the basic ticket info). +- [ ] Add an options for `show`, which outputs the full data/json object for a specific contact_id + - [ ] Should be able to pull from the local cache if it exists, so we don't keep calling the API without good reason + - [ ] Should also be able to pull directly from the API +- [ ] **Cleaner support for particular aspects of the `update` command**: + - [ ] **tag_list**: + - [ ] It should be easy to add or remove a specific tag. As it stands, using update and `--set tag_list=` requires listing all of the desired tags (delimited by commas), and overwrites any pre-existing value. + - [ ] Instead, we should just have a simple '--addTag ' and '--removeTag ' options for the `update` command + - [ ] **custom_fields**: + - [ ] We should be able to set a specific custom field to a new value, without deleting/overwriting other customer fields. + - [ ] E.g. "Account Number" is customer field id 70. We should be able to easily set a particular contact's account number to something new without overwriting, deleting, or needing to re-send the values of other customer fields (like field 42, their Facebook profile/link) + - [ ] **Specify by kebab-case custom-field name, rather than numeric id**: + - [ ] Examples: + - Custom field id 70 is "Account Number". Should be able to do --setCustomField account-number="whatever" + - Custom field id 88 is "X/Twitter". Should be able to do --setCustomField x-twitter="foobar" +- [ ] **Conduct some related research**: research the following and add it to documentation within the project: + - [ ] **Possible duplicates**: The web interface for the helpdesk_contacts plugin has a feature which shows possible duplicate contacts, and supports a workflow for "merging them". We need to know if it's feasible to implement this feature via the API or through a bash script, so we can merge contacts if needed. + - [ ] The existing web workflow ends up at a url like this: `/contacts//duplicates?project_id=`, and allows submitting a form which passes `duplicate_id`. + - E.g. If we wanted to merge contact 1500 with contact 11, we'd end up at `/contacts/1500/duplicates?project_id=customer-service` and then submit the form with `duplicate_id=11` as a POSTed data field. Doing so would: + - Delete contact 1500 + - Any tickets, deals, etc. previously associated with 1500 would then be associated with 11 + - [ ] **Subgoal 1**: Check if the API seems to support a call for replicating this workflow. + - [ ] If it does, spec out how easily it could be added as an option/tool for this script + - [ ] If it does not, check if utilizing something like `curl` to achieve it would be safe and feasible. Do so by investigating how the normal browser path would work. + +It's not super important to implement this exactly the way I'm saying. Please pay _more_ attention to how I want to _use_ the data. + +### Some context: + +#### Local testing copy/instance of Redmine + +`192.168.50.170` hosts a local instance of Redmine that is synced from the production system daily, so it acts and behaves like the real website. This can be used for testing/verifying features/changes. To do so, in any of these commands which reference a `--base-url` option, use `http://192.168.50.170` instead of `https://rm.ldrprep.com`. + + + +#### How to fetch a JSON blob of all visible contacts: + +``` +~/projects/redmine/redmine_contacts.py --project customer-service --base-url https://rm.ldrprep.com fetch +``` + +The above will save a cached JSON blob of all visible contacts in `~/.cache/redmine_contacts/rm.ldrprep.com_customer-service.json` + + +#### Fetch recent tickets for a specific contact + +``` +# Outdated +# ~/projects/redmine/redmine_contacts.py --base-url https://rm.ldrprep.com --project customer-service helpdesk-timeline --limit 5 + + +# Use this method +~/projects/redmine/redmine_contacts.py --base-url https://rm.ldrprep.com --project customer-service helpdesk-issues --limit 5 +``` + +This will show metadata about the five most recent tickets involving that contact (if there are any). The results will include the subject of the ticket, `issue_id` for pulling up more details, `assigned_to_name` (person or group assigned to, if any), `updated_on` (date and time the ticket last changed by anybody), `created_on` (date and time the email chain started), `from_address` (actual email address the contact used to email us), and other fields. + + +## Overall project objectives + +- [ ] Support programmatic recall/fetch of Redmine contacts in these kinds of ways: + - [ ] All contacts who have been created since _date1_ but have not been updated more recently than _date2_. + - This is for the purpose of doing things such as cycling through all of our contacts to apply basic changes to them like fixing their name or possibly adding basic tags to them. The idea being, we could ask for "20 newest contacts not updated within the last 7 days" or something, update them, and then go again. Eventually, we would get through all of the contacts. + - [ ] Contacts with a specific combination of tags (specific tag or couple tags, TagA but not TagB, no tags at all, etc) + - E.g. "All contacts with the tag 'customer', but without the tag 'prep'" + - [ ] No background entry recorded +- [ ] **Core outcomes we want to enable**: + - [ ] **Tagging contacts as a guide for other automated tools**: We want to be able to use tags to control how other automated tools may do things, such as triaging new tickets received, automatically adding additional context, etc. + - [ ] **Identify and link 'company' contacts with people who work at that company**: + - [ ] From context within tickets, we can tell that a contact works at a company, and we can infer the name of that company. + - [ ] If it does not already exist, add the company as a contact in Redmine, then: + - [ ] Identify contacts who are people that work at that company, and update their profile so the `company` field=the company they work at. + - [ ] **Support robust profiles across obsidian, Redmine, NMI, and other systems**: + - [ ] Outside the scope of this project right now, but we want to be able to do enrichment with our contacts, working to complete missing information like their LinkedIn profile or job title. + - [ ] Obsidian should have a matching entry for any significant contact, such as a customer, employee, or vendor. + - [ ] Each of these profiles should cross-reference each other, where possible. For instance: + - [ ] Redmine profile contacts `user_account.user_account_id` value from NMI which corresponds to that contact's account in the NMI portal (if they are a customer or employee) + - [ ] User account notes in NMI link to the highest-level contact entity in Redmine (typically the company the person works at/for) + - [ ] Accurate phone number appears the same in all relevant places + - [ ] **Compile/calculated Gist**: + - [ ] Given a particular contact and a timespan, we should be able to fetch all `customer-service` tickets from Redmine that were updated within that timespan and which involve that contact. + - [ ] If we specified a contact that is a company, this should fetch all tickets involving anybody at that company + - [ ] We should then be able to generate an AI summary of the main topics or situations that contact was interested in or talking about during that timespan. + - [ ] This kind of summary/gist can be displayed or referenced in various places, and would be saved within obsidian in a manner that ties it together with the contact and timespan. +- [ ] Identify and mark contacts as customers as follows: + - [ ] Fetch contacts that do not have the "customer" tag and have not been updated recently + - [ ] Check for the 5 most recent tickets involving that contact, and do as folows: + - [ ] **Determine if they are likely a customer, and then document them**: + - [ ] The following things indicate the contact is a customer. The more of these which apply, the stronger the chance is they are a customer: + - [ ] The ticket(s) are assigned to a specific person (a name); + - [ ] Someone on our side replied to the ticket, appearing to answer their question or provide information; + - [ ] The content of the ticket is talking about something like a package, shipment, order, prep, or pictures; + - [ ] The contact does not appear to be selling us anything or trying to get us to register to a service of some kind; + - [ ] If they are a customer, update the contact as follows: + - [ ] Add the tag "customer" + - [ ] Add the tag "needs-enriched" + - [ ] Add the tag "needs-nmi-sync" + - [ ] Check obsidian-ldr to see if there is a file about the customer. + - [ ] If there isn't, create one. Place it under `people/DRAFT/` to indicate a human should confirm it seems to be accurate. + - [ ] The file should cross-reference/link to the Redmine contact, including the following information: + - [ ] First and last name, name of company they work for, email address, phone number, and contact_id + - [ ] **They are not a customer, but are clearly something else**: + - [ ] **Automated Mailer**: The contact is most likely an automated mailer. + - [ ] This is determined by checking for the following criteria. If most of these are true, they're probably an automated mailer: + - [ ] There are five or more tickets from them, and those tickets all seem to be about the same general topic; + - [ ] There are no replies to their tickets from anyone on our team; + - [ ] The tickets are not assigned to named person _or_ group; + - [ ] The email body says something like "this inbox is not monitored"; + - [ ] Their email address includes "no-reply@" or "noreply@" + - [ ] Apply the tag "Automailer" to the contact +- [ ] This system should have a "dry run" feature for testing purposes, and should work so it can be limited to updating a specific number of contacts each time it is run. + +The main goal to start with is tuning up and improving the redmine_contacts.py CLI script. Feel free to note and sideline goals which seem too ambicious to handle at the moment. \ No newline at end of file diff --git a/saved-filters/helpdeskEmailMatchCompany.expr b/saved-filters/helpdeskEmailMatchCompany.expr new file mode 100644 index 0000000..814e32d --- /dev/null +++ b/saved-filters/helpdeskEmailMatchCompany.expr @@ -0,0 +1,2 @@ +# Contact email equals company string (common form-spam / incomplete CRM shape) +helpdesk_ticket.contact.email == helpdesk_ticket.contact.company diff --git a/saved-filters/scanned-document.expr b/saved-filters/scanned-document.expr new file mode 100644 index 0000000..1ecd3b0 --- /dev/null +++ b/saved-filters/scanned-document.expr @@ -0,0 +1,2 @@ +# Generic "Scanned document..." tickets that usually need OCR rename +subject.includes("Scanned") OR subject.includes("scanned document") OR subject.includes("HP Scan") diff --git a/saved-filters/triage-needed-5d.expr b/saved-filters/triage-needed-5d.expr new file mode 100644 index 0000000..a02856f --- /dev/null +++ b/saved-filters/triage-needed-5d.expr @@ -0,0 +1,2 @@ +# Recent unassigned triage candidates (relative window) +assigned_to IS NULL AND updated_on.since("5 days") diff --git a/saved-filters/unassigned.expr b/saved-filters/unassigned.expr new file mode 100644 index 0000000..7f2d871 --- /dev/null +++ b/saved-filters/unassigned.expr @@ -0,0 +1,2 @@ +# Unassigned customer-service tickets (use with project cache already scoped) +assigned_to IS NULL diff --git a/tests/redmine_crm_tests/__init__.py b/tests/redmine_crm_tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/redmine_crm_tests/test_redmine_crm.py b/tests/redmine_crm_tests/test_redmine_crm.py new file mode 100644 index 0000000..636195e --- /dev/null +++ b/tests/redmine_crm_tests/test_redmine_crm.py @@ -0,0 +1,272 @@ +"""Unit tests for Redmine client, mapper, and CRM upsert.""" + +from __future__ import annotations + +import re +from http import HTTPStatus +from unittest import mock + +import pytest +import responses + +from redmine_crm.crm_client import CrmClient, CrmError +from redmine_crm.mapper import enrich, map_contact +from redmine_crm.redmine_client import ( + RedmineAuthError, + RedmineClient, + RedmineNotFoundError, + RedmineTransientError, + RetryConfig, +) +from redmine_crm.sync import SyncState, run_sync + + +BASE_URL = "https://redmine.example.com" + + +@responses.activate +def test_fetch_contacts_success(): + contacts = [{"id": 1, "first_name": "Ada", "last_name": "Lovelace"}] + responses.add( + responses.GET, + f"{BASE_URL}/contacts.json", + json={"contacts": contacts}, + status=200, + ) + + client = RedmineClient(BASE_URL, "secret") + result = client.fetch_contacts() + + assert result == contacts + request = responses.calls[0].request + assert request.headers["X-Redmine-API-Key"] == "secret" + + +@responses.activate +def test_fetch_contacts_transient_retry_then_success(): + responses.add(responses.GET, f"{BASE_URL}/contacts.json", status=503) + responses.add( + responses.GET, + f"{BASE_URL}/contacts.json", + json={"contacts": [{"id": 1, "first_name": "Grace", "last_name": "Hopper"}]}, + status=200, + ) + + client = RedmineClient( + BASE_URL, + "secret", + retry=RetryConfig(max_attempts=2, base_delay_seconds=0.001), + ) + result = client.fetch_contacts() + + assert len(result) == 1 + assert len(responses.calls) == 2 + + +@responses.activate +def test_fetch_contacts_permanent_failure_after_retries(): + responses.add(responses.GET, f"{BASE_URL}/contacts.json", status=503) + + client = RedmineClient( + BASE_URL, + "secret", + retry=RetryConfig(max_attempts=1, base_delay_seconds=0.001), + ) + with pytest.raises(RedmineTransientError): + client.fetch_contacts() + + +@responses.activate +def test_fetch_contact_not_found_raises(): + responses.add(responses.GET, f"{BASE_URL}/contacts/99.json", status=404) + + client = RedmineClient(BASE_URL, "secret") + with pytest.raises(RedmineNotFoundError): + client.fetch_contact(99) + + +@responses.activate +def test_auth_error_is_permanent(): + responses.add(responses.GET, f"{BASE_URL}/contacts.json", status=401) + + client = RedmineClient(BASE_URL, "secret") + with pytest.raises(RedmineAuthError): + client.fetch_contacts() + + +def test_map_contact_basic(): + raw = { + "id": 7, + "first_name": "Alan", + "last_name": "Turing", + "company": "Bletchley Park", + "job_title": "Mathematician", + "email": "alan@example.com", + "phone": "+44 345 678", + "website": "https://turing.example.com", + "birthday": "1912-06-23", + "background": "Father of computer science.", + "tag_list": ["Hero", "pioneer"], + "updated_at": "2026-07-01T12:00:00Z", + "address_attributes": { + "street": "1 Enigma Way", + "city": "Bletchley", + "state": "England", + "country": "UK", + "zip": "MK3 9HB", + }, + "custom_fields": [ + {"name": "LinkedIn", "value": "https://linkedin.com/in/alan"}, + {"name": "Source", "value": "conference"}, + ], + } + + mapped = map_contact(raw, base_url=BASE_URL) + + assert mapped["redmine_contact_id"] == 7 + assert mapped["full_name"] == "Alan Turing" + assert mapped["company"] == "Bletchley Park" + assert mapped["title"] == "Mathematician" + assert mapped["emails"] == [{"value": "alan@example.com", "type": "work"}] + assert mapped["urls"] == [ + {"value": "https://turing.example.com", "type": "work"}, + {"value": "https://linkedin.com/in/alan", "type": "linkedin"}, + ] + assert mapped["address"]["city"] == "Bletchley" + assert mapped["address"]["postal_code"] == "MK3 9HB" + assert "Father of computer science" in mapped["notes"] + assert "LinkedIn" not in mapped["notes"] # promoted to URL + assert "Source" in mapped["notes"] + assert "from-redmine" in mapped["tags"] + assert mapped["tags"] == sorted({"hero", "pioneer", "from-redmine"}) + assert mapped["last_redmine_update"] is not None + + +def test_map_company_record(): + raw = { + "id": 8, + "is_company": True, + "company": "Acme Corp", + "email": "info@acme.example.com", + "updated_at": "2026-07-02T10:00:00Z", + } + mapped = map_contact(raw, base_url=BASE_URL) + assert mapped["full_name"] == "Acme Corp" + assert mapped["is_company"] is True + + +def test_enrich_fills_email_from_background(): + raw = { + "id": 9, + "first_name": "Linus", + "last_name": "Torvalds", + "background": "Reach me at linus@linux.org for kernel stuff.", + "updated_at": "2026-07-03T10:00:00Z", + } + mapped = map_contact(raw, base_url=BASE_URL) + assert mapped["emails"] == [] + enriched, meta = enrich(mapped, raw) + assert enriched["emails"] == [{"value": "linus@linux.org", "type": "work"}] + assert meta.email_source == "background" + assert enriched["enriched_at"] is not None + + +def test_enrich_derives_company_from_domain(): + raw = { + "id": 10, + "first_name": "Tim", + "last_name": "Berners-Lee", + "email": "tim@w3.org", + "updated_at": "2026-07-03T10:00:00Z", + } + mapped = map_contact(raw, base_url=BASE_URL) + mapped.pop("company", None) + enriched, meta = enrich(mapped, raw) + assert enriched["company"] == "W3 Inc" + assert meta.company_source == "domain_heuristic" + + +def test_crm_create_and_update_by_redmine_id(): + crm = CrmClient() + record = { + "redmine_contact_id": 1, + "full_name": "Ada Lovelace", + "emails": [{"value": "ada@example.com", "type": "work"}], + "last_redmine_update": "2026-07-01T00:00:00Z", + } + + result1 = crm.upsert(record) + assert result1.created is True + + result2 = crm.upsert(record) + assert result2.skipped_unchanged is True + + updated = dict(record) + updated["last_redmine_update"] = "2026-07-02T00:00:00Z" + result3 = crm.upsert(updated) + assert result3.updated is True + + +def test_crm_email_fallback_links_record(): + crm = CrmClient() + crm.upsert({ + "redmine_contact_id": 2, + "full_name": "Old Name", + "emails": [{"value": "match@example.com", "type": "work"}], + "last_redmine_update": "2026-07-01T00:00:00Z", + }) + + result = crm.upsert({ + "redmine_contact_id": 3, + "full_name": "New Name", + "emails": [{"value": "match@example.com", "type": "work"}], + "last_redmine_update": "2026-07-02T00:00:00Z", + }) + assert result.updated is True + + +@responses.activate +def test_run_sync_end_to_end(): + contacts = [ + { + "id": 1, + "first_name": "Grace", + "last_name": "Hopper", + "email": "grace@example.com", + "updated_at": "2026-07-04T12:00:00Z", + } + ] + responses.add( + responses.GET, + f"{BASE_URL}/contacts.json", + json={"contacts": contacts}, + status=200, + ) + + redmine = RedmineClient(BASE_URL, "secret") + crm = CrmClient() + crm._store.clear() + state = SyncState() + + stats = run_sync(redmine, crm, state, base_url=BASE_URL, batch_size=100) + + assert stats["processed"] == 1 + assert stats["created"] == 1 + assert stats["errors"] == 0 + + +def test_upsert_requires_redmine_id(): + crm = CrmClient() + with pytest.raises(CrmError): + crm.upsert({"full_name": "No ID"}) + + +def test_map_contact_notes_include_redmine_source_url(): + raw = { + "id": 42, + "first_name": "Marie", + "last_name": "Curie", + "updated_at": "2026-07-01T00:00:00Z", + } + mapped = map_contact(raw, base_url=BASE_URL) + assert re.search(r"Redmine source: https://redmine\.example\.com/contacts/42", mapped["notes"]) diff --git a/tests/test_attachments_cli.py b/tests/test_attachments_cli.py new file mode 100644 index 0000000..1b75f54 --- /dev/null +++ b/tests/test_attachments_cli.py @@ -0,0 +1,91 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +from redmine_ticket_attachments import ( + attachments_from_issue, + pick_attachment, + safe_filename, + upload_file_token, +) +from redmine_tickets import build_parser, inject_default_sync_command + + +class AttachmentHelpersTest(unittest.TestCase): + def test_pick_and_safe(self): + atts = [{"id": 1, "filename": "a.pdf"}, {"id": 2, "filename": "b.pdf"}] + self.assertEqual(2, pick_attachment(atts, 2)["id"]) + with self.assertRaises(LookupError): + pick_attachment(atts, None) + self.assertEqual(1, pick_attachment([atts[0]], None)["id"]) + self.assertEqual("foo_bar.pdf", safe_filename("foo/bar.pdf")) + + def test_attachments_from_issue(self): + self.assertEqual([], attachments_from_issue({})) + self.assertEqual(1, len(attachments_from_issue({"attachments": [{"id": 9}]}))) + + def test_cli_parse(self): + args = build_parser().parse_args( + inject_default_sync_command( + [ + "--project", + "customer-service", + "attachments", + "list", + "--issue-id", + "1", + "--refresh", + ] + ) + ) + self.assertEqual("attachments", args.command) + self.assertEqual("list", args.attachments_command) + self.assertTrue(args.refresh) + + args2 = build_parser().parse_args( + inject_default_sync_command( + [ + "attachments", + "upload", + "--issue-id", + "2", + "--file", + "/tmp/x.pdf", + "--apply", + ] + ) + ) + self.assertEqual("upload", args2.attachments_command) + self.assertTrue(args2.apply) + + def test_upload_token_mock(self): + with tempfile.TemporaryDirectory() as td: + path = Path(td) / "note.txt" + path.write_text("hello", encoding="utf-8") + + class FakeResp: + def read(self): + return b'{"upload":{"token":"tok.123","id":1}}' + + def __enter__(self): + return self + + def __exit__(self, *a): + return False + + with patch("redmine_ticket_attachments.urllib_request.urlopen", return_value=FakeResp()): + upload = upload_file_token( + base_url="http://192.168.50.170", + api_key="x", + path=path, + ) + self.assertEqual("tok.123", upload["token"]) + self.assertEqual("note.txt", upload["filename"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_browse_loop.py b/tests/test_browse_loop.py new file mode 100644 index 0000000..7761790 --- /dev/null +++ b/tests/test_browse_loop.py @@ -0,0 +1,39 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import unittest + +from redmine_ticket_browse import build_lines, issue_line, parse_id_from_line +from redmine_tickets import build_parser, inject_default_sync_command + + +class BrowseHelpersTest(unittest.TestCase): + def test_lines(self): + issue = { + "id": 42, + "status": {"name": "New"}, + "assigned_to": {"name": "Jason"}, + "subject": "Foo\tbar\nbaz", + "_control_tags": ["spammy"], + } + line = issue_line(issue) + self.assertTrue(line.startswith("42\t")) + self.assertIn("Jason", line) + self.assertIn("spammy", line) + self.assertNotIn("\n", line) + self.assertEqual(42, parse_id_from_line(line)) + self.assertEqual([line], build_lines([issue])) + + def test_cli(self): + args = build_parser().parse_args( + inject_default_sync_command( + ["browse", "--where", "assigned_to IS NULL", "--pager", "cat", "--once"] + ) + ) + self.assertEqual("browse", args.command) + self.assertTrue(args.once) + self.assertEqual("cat", args.pager) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_bugfix_qol_drafts.py b/tests/test_bugfix_qol_drafts.py new file mode 100644 index 0000000..8d5be45 --- /dev/null +++ b/tests/test_bugfix_qol_drafts.py @@ -0,0 +1,105 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import tempfile +import unittest +from argparse import Namespace +from pathlib import Path + +from redmine_ticket_drafts import KIND_INTERNAL_NOTE, add_draft +from redmine_tickets import ( + apply_runtime_config_to_argv, + build_parser, + inject_default_sync_command, + load_runtime_config, + print_draft_queued, + resolve_apply_base_url, +) + + +class ProdWritesParentFlagTest(unittest.TestCase): + def test_parent_allow_prod_writes_before_draft_apply(self): + args = build_parser().parse_args( + inject_default_sync_command( + [ + "--base-url", + "https://rm.ldrprep.com", + "--allow-prod-writes", + "draft", + "apply", + "deadbeefcafe", + ] + ) + ) + self.assertTrue(args.allow_prod_writes) + self.assertEqual("draft", args.command) + self.assertEqual("apply", args.draft_command) + + +class DraftBaseUrlTest(unittest.TestCase): + def test_resolve_prefers_draft_url(self): + args = Namespace(base_url="https://rm.ldrprep.com", allow_base_url_override=False) + draft = {"base_url": "http://192.168.50.170"} + # default CLI equals DEFAULT which is often prod; draft LAN should win when cli is "default-like" + # Here cli is explicit prod vs draft LAN — should refuse without override + with self.assertRaises(Exception): + resolve_apply_base_url(args, draft) + + args.allow_base_url_override = True + self.assertEqual("https://rm.ldrprep.com", resolve_apply_base_url(args, draft)) + + def test_resolve_uses_draft_when_cli_default(self): + from redmine_contacts import DEFAULT_BASE_URL + + args = Namespace(base_url=DEFAULT_BASE_URL, allow_base_url_override=False) + draft = {"base_url": "http://192.168.50.170"} + self.assertEqual("http://192.168.50.170", resolve_apply_base_url(args, draft)) + + +class ConfigAndDraftIdTest(unittest.TestCase): + def test_config_inject(self): + with tempfile.TemporaryDirectory() as td: + cfg = Path(td) / "c.json" + cfg.write_text( + '{"base-url":"http://192.168.50.170","project":"customer-service","cache-dir":"/tmp/x"}', + encoding="utf-8", + ) + out = apply_runtime_config_to_argv(["--config", str(cfg), "search", "--max-tickets", "1"]) + self.assertIn("--base-url", out) + self.assertIn("http://192.168.50.170", out) + self.assertIn("--project", out) + # explicit flag not overridden + out2 = apply_runtime_config_to_argv( + ["--project", "other", "--config", str(cfg), "search"] + ) + # project other should remain; config injects base-url/cache-dir only if missing + self.assertIn("other", out2) + + def test_print_draft_id(self): + with tempfile.TemporaryDirectory() as td: + path = add_draft( + Path(td), + issue_id=9, + kind=KIND_INTERNAL_NOTE, + body="hi", + base_url="https://rm.ldrprep.com", + ) + import io + import contextlib + + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + did = print_draft_queued( + path, + kind=KIND_INTERNAL_NOTE, + issue_id=9, + base_url="https://rm.ldrprep.com", + ) + text = buf.getvalue() + self.assertEqual(path.stem, did) + self.assertIn(f"draft_id={did}", text) + self.assertIn("base_url=https://rm.ldrprep.com", text) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_contact_enrichment.py b/tests/test_contact_enrichment.py new file mode 100644 index 0000000..4570c43 --- /dev/null +++ b/tests/test_contact_enrichment.py @@ -0,0 +1,276 @@ +import json +import time +import unittest +from unittest import mock + +import contact_enrichment + + +class EnrichmentMappingTest(unittest.TestCase): + def test_basic_contact_mapping(self): + redmine = { + "id": 42, + "first_name": "Alice", + "middle_name": "Marie", + "last_name": "Nguyen", + "company": "Acme Inc", + "job_title": "Engineer", + "email": "alice@acme.com", + "phone": "+1 555-123-4567", + "website": "https://acme.com", + "birthday": "1990-03-15", + "background": "VIP customer", + "tag_list": ["customer", "vip"], + "updated_at": "2026-07-01T12:00:00Z", + } + + result = contact_enrichment.enrich_contact(redmine) + e = result.enriched + + self.assertEqual(42, e["redmine_contact_id"]) + self.assertEqual("Alice", e["first_name"]) + self.assertEqual("Marie", e["middle_name"]) + self.assertEqual("Nguyen", e["last_name"]) + self.assertEqual("Alice Marie Nguyen", e["full_name"]) + self.assertEqual("Acme Inc", e["company"]) + self.assertEqual("Engineer", e["title"]) + self.assertEqual([{"value": "alice@acme.com", "type": "work"}], e["emails"]) + self.assertEqual([{"value": "+15551234567", "type": "work"}], e["phones"]) + self.assertEqual("https://acme.com", e["website"]) + self.assertEqual("1990-03-15", e["birthday"]) + self.assertEqual("redmine", e["source"]) + self.assertIn("customer", e["tags"]) + self.assertIn("vip", e["tags"]) + self.assertIn("from-redmine", e["tags"]) + self.assertIn("VIP customer", e["notes"]) + self.assertEqual("2026-07-01T12:00:00+00:00", e["last_redmine_update"]) + + def test_company_record_uses_company_as_name(self): + redmine = { + "id": 7, + "is_company": True, + "company": "Globex", + "first_name": "Globex", + "email": "info@globex.com", + } + + result = contact_enrichment.enrich_contact(redmine) + self.assertEqual("Globex", result.enriched["full_name"]) + self.assertEqual("Globex", result.enriched["first_name"]) + + def test_address_attributes_mapped(self): + redmine = { + "id": 3, + "first_name": "Bob", + "last_name": "Smith", + "address_attributes": { + "street": "123 Main St", + "city": "Springfield", + "state": "IL", + "zip": "62701", + "country": "USA", + }, + } + + result = contact_enrichment.enrich_contact(redmine) + address = result.enriched["address"] + self.assertEqual("123 Main St", address["street"]) + self.assertEqual("Springfield", address["city"]) + self.assertEqual("IL", address["region"]) + self.assertEqual("62701", address["postal_code"]) + self.assertEqual("USA", address["country"]) + + def test_email_extracted_from_background_when_missing(self): + redmine = { + "id": 9, + "first_name": "Carol", + "last_name": "Danvers", + "background": "Reach her at carol@shield.gov for details.", + } + + result = contact_enrichment.enrich_contact(redmine) + self.assertEqual([{"value": "carol@shield.gov", "type": "work"}], result.enriched["emails"]) + + def test_company_inferred_from_email_domain(self): + redmine = { + "id": 11, + "first_name": "Diana", + "last_name": "Prince", + "email": "diana@themyscira.io", + } + + result = contact_enrichment.enrich_contact(redmine) + self.assertEqual("Themyscira Inc", result.enriched["company"]) + + def test_custom_domain_to_company_mapping(self): + config = contact_enrichment.EnrichmentConfig( + domain_to_company={"themyscira.io": "Themyscira Labs"} + ) + redmine = { + "id": 12, + "first_name": "Diana", + "last_name": "Prince", + "email": "diana@themyscira.io", + } + + result = contact_enrichment.enrich_contact(redmine, config) + self.assertEqual("Themyscira Labs", result.enriched["company"]) + + def test_external_lookup_fills_company(self): + config = contact_enrichment.EnrichmentConfig( + external_lookup=lambda _cid: {"company": "Wayne Enterprises"} + ) + redmine = { + "id": 13, + "first_name": "Bruce", + "last_name": "Wayne", + "email": "bruce@wayne.com", + } + + result = contact_enrichment.enrich_contact(redmine, config) + self.assertEqual("Wayne Enterprises", result.enriched["company"]) + + def test_missing_contact_id_is_skipped(self): + result = contact_enrichment.enrich_contact({"first_name": "Ghost"}) + self.assertTrue(result.skipped) + self.assertIn("Missing Redmine contact id", result.reason) + + +class PhoneEnrichmentTest(unittest.TestCase): + def test_us_phone_normalized_to_e164(self): + redmine = { + "id": 20, + "first_name": "Eve", + "last_name": "Polastri", + "phone": "555.867.5309", + } + + result = contact_enrichment.enrich_contact(redmine) + self.assertEqual([{"value": "+15558675309", "type": "work"}], result.enriched["phones"]) + + def test_mobile_phone_classified(self): + redmine = { + "id": 21, + "first_name": "Frank", + "last_name": "Castle", + "phone": "mobile: 555-0100", + } + + result = contact_enrichment.enrich_contact(redmine) + self.assertEqual("mobile", result.enriched["phones"][0]["type"]) + + def test_short_phone_is_ignored(self): + redmine = { + "id": 22, + "first_name": "Grace", + "last_name": "Hopper", + "phone": "12345", + } + + result = contact_enrichment.enrich_contact(redmine) + self.assertEqual([], result.enriched["phones"]) + + +class NotesAndTagsTest(unittest.TestCase): + def test_notes_merge_background_and_custom_fields(self): + redmine = { + "id": 30, + "first_name": "Hank", + "last_name": "Pym", + "background": "Shrinking tech", + "custom_fields": [ + {"name": "LinkedIn", "value": "https://linkedin.com/in/hankpym"}, + {"name": "Source", "value": "referral"}, + ], + } + + result = contact_enrichment.enrich_contact(redmine) + notes = result.enriched["notes"] + self.assertIn("Shrinking tech", notes) + self.assertIn("LinkedIn: https://linkedin.com/in/hankpym", notes) + self.assertIn("Source: referral", notes) + self.assertIn("Redmine contact: 30", notes) + + def test_tags_are_lowercased_and_deduplicated(self): + redmine = { + "id": 31, + "first_name": "Irene", + "last_name": "Adler", + "tag_list": "Customer, VIP, customer", + "project_id": 1, + } + + result = contact_enrichment.enrich_contact(redmine) + self.assertEqual(["customer", "from-redmine", "project-1", "vip"], result.enriched["tags"]) + + +class RateLimitTest(unittest.TestCase): + def test_rate_limit_wrapper_respects_max_calls(self): + calls = [] + + def worker(x): + calls.append(x) + return x * 2 + + wrapped = contact_enrichment.with_rate_limit( + worker, max_calls=2, per_seconds=1.0, max_retries=0 + ) + start = time.monotonic() + self.assertEqual(2, wrapped(1)) + self.assertEqual(4, wrapped(2)) + self.assertEqual(6, wrapped(3)) + elapsed = time.monotonic() - start + self.assertGreaterEqual(elapsed, 0.45) + self.assertEqual([1, 2, 3], calls) + + def test_rate_limit_wrapper_retries_transient_errors(self): + source = mock.Mock(side_effect=[RuntimeError("boom"), RuntimeError("boom"), "ok"]) + + wrapped = contact_enrichment.with_rate_limit( + source, max_calls=10, per_seconds=1.0, max_retries=3, backoff_base=0.01 + ) + self.assertEqual("ok", wrapped()) + self.assertEqual(3, source.call_count) + + def test_rate_limit_error_propagates_without_retry(self): + def worker(): + raise contact_enrichment.RateLimitError("throttled") + + wrapped = contact_enrichment.with_rate_limit( + worker, max_calls=10, per_seconds=1.0, max_retries=3 + ) + with self.assertRaises(contact_enrichment.RateLimitError): + wrapped() + + +class BatchSyncTest(unittest.TestCase): + def test_build_sync_payload_skips_invalid_and_warns(self): + contacts = [ + {"id": 1, "first_name": "A", "last_name": "B", "email": "a@b.com"}, + {"first_name": "Invalid"}, + ] + payload = contact_enrichment.build_sync_payload(contacts) + self.assertEqual(1, payload["count"]) + self.assertEqual(1, len(payload["records"])) + self.assertEqual(1, len(payload["skipped"])) + self.assertEqual("Missing Redmine contact id", payload["skipped"][0]["reason"]) + + +class DateParsingTest(unittest.TestCase): + def test_birthday_formats(self): + for fmt, value in ( + ("%Y-%m-%d", "1990-03-15"), + ("%Y/%m/%d", "1990/03/15"), + ("%m/%d/%Y", "03/15/1990"), + ("%d/%m/%Y", "15/03/1990"), + ): + with self.subTest(fmt=fmt): + result = contact_enrichment._format_birthday(value) + self.assertEqual("1990-03-15", result) + + def test_invalid_birthday_returns_none(self): + self.assertIsNone(contact_enrichment._format_birthday("not-a-date")) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_contact_set_and_bridge.py b/tests/test_contact_set_and_bridge.py new file mode 100644 index 0000000..3fa7d1b --- /dev/null +++ b/tests/test_contact_set_and_bridge.py @@ -0,0 +1,85 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import tempfile +import unittest +from pathlib import Path +from unittest.mock import MagicMock + +from redmine_ticket_contacts import contact_id_from_issue +from redmine_ticket_drafts import KIND_CONTACT_FIELDS, add_draft, kind_label, load_draft, normalize_kind +from redmine_tickets import apply_draft_to_redmine, build_parser, inject_default_sync_command + + +class ContactBridgeTest(unittest.TestCase): + def test_contact_id_from_issue(self): + self.assertEqual( + 99, + contact_id_from_issue({"helpdesk_ticket": {"contact_id": 99}}), + ) + self.assertEqual( + 7, + contact_id_from_issue({"helpdesk_ticket": {"contact": {"id": 7, "email": "a@b.c"}}}), + ) + self.assertIsNone(contact_id_from_issue({"subject": "x"})) + + +class ContactDraftTest(unittest.TestCase): + def test_kind(self): + self.assertEqual(KIND_CONTACT_FIELDS, normalize_kind("contact_fields")) + self.assertIn("CONTACT UPDATE", kind_label(KIND_CONTACT_FIELDS)) + + def test_queue_and_apply_mock(self): + with tempfile.TemporaryDirectory() as td: + path = add_draft( + Path(td), + contact_id=42, + kind=KIND_CONTACT_FIELDS, + fields={"tag_list": ["maybe-customer"], "company": "Acme"}, + base_url="http://192.168.50.170", + ) + _p, draft = load_draft(Path(td), path.stem) + self.assertEqual(42, draft["contact_id"]) + self.assertIsNone(draft["issue_id"]) + client = MagicMock() + result = apply_draft_to_redmine( + client, + draft, + impersonate_user="rebot", + allow_customer_email=False, + ) + client.update_contact.assert_called_once() + self.assertEqual(42, result["contact_id"]) + self.assertFalse(result["emailed_customer"]) + + def test_cli_parse(self): + args = build_parser().parse_args( + inject_default_sync_command( + [ + "--project", + "customer-service", + "contact-set", + "--contact-id", + "14", + "--add-tag", + "maybe-customer", + "--company", + "Test Co", + ] + ) + ) + self.assertEqual("contact-set", args.command) + self.assertEqual(14, args.contact_id) + self.assertEqual(["maybe-customer"], args.add_tag) + + args2 = build_parser().parse_args( + inject_default_sync_command( + ["--project", "customer-service", "ticket-contact", "--issue-id", "41934", "--show"] + ) + ) + self.assertEqual("ticket-contact", args2.command) + self.assertTrue(args2.show) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_entities_export_browse.py b/tests/test_entities_export_browse.py new file mode 100644 index 0000000..859c3fc --- /dev/null +++ b/tests/test_entities_export_browse.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import tempfile +import unittest +from pathlib import Path + +from redmine_ticket_entities import extract_entities, extract_entities_from_issue +from redmine_ticket_control_export import export_control_tag_jsonl +from redmine_ticket_control_tags import add_tag, load_store +from redmine_tickets import KNOWN_COMMANDS, build_parser, inject_default_sync_command + + +class EntityExtractTest(unittest.TestCase): + def test_patterns(self): + text = ( + "Order 123-4567890-1234567 UPS 1Z999AA10123456784 " + "Item # SKU-ABC-99 USPS 9400111899223344556677" + ) + types = {e["type"] for e in extract_entities(text)} + self.assertIn("tracking_ups", types) + self.assertIn("tracking_usps", types) + self.assertIn("order_amazon", types) + self.assertIn("item_or_sku", types) + + def test_noise_suppressed(self): + htmlish = "order-mode responsive-title order-text" + hits = extract_entities(htmlish) + self.assertEqual([], hits) + + def test_from_issue(self): + issue = { + "subject": "RET 1ZY228K59066639253", + "description": "Order ID: 1013179 shipped", + "journals": [], + } + ents = extract_entities_from_issue(issue) + types = {e["type"] for e in ents} + self.assertIn("tracking_ups", types) + self.assertIn("order_id", types) + + +class ControlExportTest(unittest.TestCase): + def test_export(self): + with tempfile.TemporaryDirectory() as td: + cache = Path(td) + add_tag(cache, tag="spammy", issue_id=7, properties=["subject"], annotation="x") + store = load_store(cache) + out = cache / "out.jsonl" + n = export_control_tag_jsonl( + cache_dir=cache, + tags_payload=store, + issues_by_id={7: {"id": 7, "subject": "hello", "status": {"name": "New"}}}, + out_path=out, + ) + self.assertEqual(1, n) + line = out.read_text(encoding="utf-8").strip() + self.assertIn("spammy", line) + self.assertIn('"issue_id": 7', line) + + +class CliRegistrationTest(unittest.TestCase): + def test_known_commands(self): + self.assertIn("entities", KNOWN_COMMANDS) + self.assertIn("browse", KNOWN_COMMANDS) + args = build_parser().parse_args( + inject_default_sync_command(["entities", "show", "--issue-id", "1"]) + ) + self.assertEqual("entities", args.command) + self.assertEqual("show", args.entities_command) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_fields_and_control_tags.py b/tests/test_fields_and_control_tags.py new file mode 100644 index 0000000..5f239e8 --- /dev/null +++ b/tests/test_fields_and_control_tags.py @@ -0,0 +1,97 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import tempfile +import unittest +from pathlib import Path + +from redmine_ticket_control_tags import ( + add_tag, + filter_issues_by_control_tag, + list_tags, + remove_tag, + tags_for_issue, + tickets_for_tag, + wipe_all, +) +from redmine_ticket_drafts import KIND_FIELDS, add_draft, kind_label, load_draft, normalize_kind +from redmine_tickets import build_fields_payload_from_args, build_parser, inject_default_sync_command + + +class ControlTagsTest(unittest.TestCase): + def test_add_list_filter_remove(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + add_tag( + root, + tag="Bavaria", + issue_id=10, + properties=["subject", "helpdesk_ticket.contact.company"], + annotation="looks like BavariaTrachten", + ) + add_tag(root, tag="bavaria", issue_id=11, properties=["subject"]) + tags = list_tags(root) + self.assertEqual(1, len(tags)) + self.assertEqual("bavaria", tags[0]["tag"]) + self.assertEqual([10, 11], tags[0]["issue_ids"]) + self.assertEqual(["bavaria"], tags_for_issue(root, 10)) + issues = [{"id": 10}, {"id": 99}, {"id": 11}] + filtered = filter_issues_by_control_tag(issues, root, "bavaria") + self.assertEqual([10, 11], [i["id"] for i in filtered]) + self.assertTrue(remove_tag(root, tag="bavaria", issue_id=10)) + self.assertEqual([11], tickets_for_tag(root, "bavaria")) + wipe_all(root) + self.assertEqual([], list_tags(root)) + + def test_no_spaces(self): + with tempfile.TemporaryDirectory() as td: + with self.assertRaises(ValueError): + add_tag(Path(td), tag="bad tag", issue_id=1) + + +class FieldsDraftTest(unittest.TestCase): + def test_fields_kind(self): + self.assertEqual(KIND_FIELDS, normalize_kind("fields")) + self.assertIn("FIELD UPDATE", kind_label(KIND_FIELDS)) + with tempfile.TemporaryDirectory() as td: + p = add_draft( + Path(td), + issue_id=5, + kind=KIND_FIELDS, + fields={"status_id": 2, "assigned_to_id": None}, + ) + _path, data = load_draft(Path(td), p.stem) + self.assertEqual(KIND_FIELDS, data["kind"]) + self.assertFalse(data["emails_customer"]) + self.assertEqual(2, data["fields"]["status_id"]) + + def test_set_fields_cli_parse(self): + args = build_parser().parse_args( + inject_default_sync_command( + [ + "--project", + "customer-service", + "set-fields", + "--issue-id", + "1", + "--status-id", + "5", + "--assigned-to-id", + "0", + "--due-date", + "2026-08-01", + "--notes", + "closing", + "--private", + ] + ) + ) + payload = build_fields_payload_from_args(args) + self.assertEqual(5, payload["status_id"]) + self.assertIsNone(payload["assigned_to_id"]) + self.assertEqual("2026-08-01", payload["due_date"]) + self.assertTrue(payload["private_notes"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_lane_a_b_polish.py b/tests/test_lane_a_b_polish.py new file mode 100644 index 0000000..618cfbd --- /dev/null +++ b/tests/test_lane_a_b_polish.py @@ -0,0 +1,95 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +from redmine_ticket_entities import extract_entities +from redmine_tickets import ( + is_closed_status, + maybe_convert_to_textile, + resolve_body_text, + build_parser, + inject_default_sync_command, +) +import argparse + + +class OpenOnlyTest(unittest.TestCase): + def test_closed(self): + self.assertTrue(is_closed_status({"status": {"name": "Closed", "is_closed": True}})) + self.assertTrue(is_closed_status({"status": {"name": "Closed"}})) + self.assertFalse(is_closed_status({"status": {"name": "New"}})) + + +class EntityTightenTest(unittest.TestCase): + def test_item_no_and_noise(self): + text = "Order ID: 1013179 UPS 1ZT33T62A898848403 item 212345678 SKU-ABC-99 order-mode responsive-title" + ents = extract_entities(text) + types = {e["type"]: e["value"] for e in ents} + self.assertEqual("1ZT33T62A898848403", types.get("tracking_ups")) + self.assertIn("order_id", types) + self.assertEqual("1013179", types["order_id"]) + self.assertEqual("212345678", types.get("item_no")) + # HTML noise should not become order_id + values = [e["value"] for e in ents if e["type"] == "order_id"] + self.assertNotIn("responsive-title", values) + self.assertNotIn("lation-mode", values) + + +class BodyFileTextileTest(unittest.TestCase): + def test_body_file(self): + with tempfile.TemporaryDirectory() as td: + p = Path(td) / "note.md" + p.write_text("# Hello\n\nWorld", encoding="utf-8") + ns = argparse.Namespace(body=None, body_file=p, to_textile=False) + self.assertIn("Hello", resolve_body_text(ns)) + + def test_to_textile_without_pandoc(self): + with patch("shutil.which", return_value=None): + out = maybe_convert_to_textile("plain") + self.assertEqual("plain", out) + + +class CliFlagsTest(unittest.TestCase): + def test_flags(self): + args = build_parser().parse_args( + inject_default_sync_command( + [ + "search", + "--open-only", + "--where", + "assigned_to IS NULL", + "--max-tickets", + "5", + ] + ) + ) + self.assertTrue(args.open_only) + args2 = build_parser().parse_args( + inject_default_sync_command( + ["browse", "--open-only", "--no-preview", "--once"] + ) + ) + self.assertTrue(args2.open_only) + self.assertTrue(args2.no_preview) + args3 = build_parser().parse_args( + inject_default_sync_command( + [ + "note", + "--issue-id", + "1", + "--body-file", + "/tmp/x", + "--to-textile", + "--private", + ] + ) + ) + self.assertTrue(args3.to_textile) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_meta_and_contacts.py b/tests/test_meta_and_contacts.py new file mode 100644 index 0000000..57c3eae --- /dev/null +++ b/tests/test_meta_and_contacts.py @@ -0,0 +1,96 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import tempfile +import unittest +from pathlib import Path + +from redmine_ticket_meta import ( + normalize_name, + resolve_id_by_name, + users_from_memberships, +) +from redmine_ticket_contacts import ( + ContactCache, + contact_display_name, + filter_contacts, + primary_email, +) +from redmine_tickets import build_fields_payload_from_args, build_parser, inject_default_sync_command + + +class MetaResolveTest(unittest.TestCase): + def test_resolve_status(self): + statuses = [{"id": 1, "name": "New"}, {"id": 5, "name": "Closed"}] + self.assertEqual(5, resolve_id_by_name(statuses, "closed")) + self.assertEqual(5, resolve_id_by_name(statuses, "5")) + + def test_users_from_memberships(self): + users = users_from_memberships( + [ + {"user": {"id": 2, "name": "Jason T.", "login": "jason"}}, + {"user": {"id": 2, "name": "Jason T."}}, + {"group": {"id": 9, "name": "CS"}}, + ] + ) + self.assertEqual(1, len(users)) + self.assertEqual(2, resolve_id_by_name(users, "jason", name_keys=("name", "login"))) + + def test_set_fields_name_cli_parse(self): + args = build_parser().parse_args( + inject_default_sync_command( + [ + "--project", + "customer-service", + "set-fields", + "--issue-id", + "1", + "--status", + "Closed", + "--assigned-to", + "unassigned", + ] + ) + ) + self.assertEqual("Closed", args.status_name) + self.assertEqual("unassigned", args.assignee_name) + # build with resolved + payload = build_fields_payload_from_args( + args, + resolved_ids={"status_id": 5, "assigned_to_id": 0}, + ) + self.assertEqual(5, payload["status_id"]) + self.assertIsNone(payload["assigned_to_id"]) + + +class ContactsCacheTest(unittest.TestCase): + def test_write_filter(self): + with tempfile.TemporaryDirectory() as td: + cache = ContactCache(Path(td)) + cache.write_contact( + { + "id": 7, + "name": "Ada Lovelace", + "email": "ada@example.com", + "company": "Analytical Engines", + "updated_on": "2026-07-01T00:00:00Z", + "tag_list": ["customer", "vip"], + } + ) + loaded = cache.load_contact(7) + self.assertEqual("Ada Lovelace", contact_display_name(loaded)) + self.assertEqual("ada@example.com", primary_email(loaded)) + allc = list(cache.iter_contacts()) + hits = filter_contacts(allc, query="analytical", tag="customer") + self.assertEqual(1, len(hits)) + self.assertFalse(cache.should_fetch({"id": 7, "updated_on": "2026-07-01T00:00:00Z"})) + self.assertTrue(cache.should_fetch({"id": 7, "updated_on": "2026-07-02T00:00:00Z"})) + + +class NormalizeTest(unittest.TestCase): + def test_normalize(self): + self.assertEqual("jason t.", normalize_name(" Jason T. ")) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_nmi_runtime_profile.py b/tests/test_nmi_runtime_profile.py new file mode 100644 index 0000000..ddc6e11 --- /dev/null +++ b/tests/test_nmi_runtime_profile.py @@ -0,0 +1,71 @@ +"""NMI runtime-profile contract for the Redmine CLI/TUI.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +import redmine_ticket_nmi + + +def test_runtime_profile_honors_explicit_production_mode_and_env_file(tmp_path): + production_env = tmp_path / "production.env" + production_env.write_text("NMI_ENDPOINT_PRODUCTION=https://app.ldrprep.com/api.php\n") + + profile = redmine_ticket_nmi.resolve_runtime_profile( + environ={ + "NMI_MODE": "production", + "NMI_ENV_FILE": str(production_env), + } + ) + + assert profile.mode == "production" + assert profile.env_file == production_env + assert profile.ready is True + assert profile.problem is None + + +def test_runtime_profile_rejects_empty_env_file(tmp_path): + empty_env = tmp_path / "empty.env" + empty_env.touch() + + profile = redmine_ticket_nmi.resolve_runtime_profile( + environ={ + "NMI_MODE": "production", + "NMI_ENV_FILE": str(empty_env), + } + ) + + assert profile.mode == "production" + assert profile.ready is False + assert "empty" in profile.problem.lower() + + +def test_brief_refuses_unready_runtime_profile(monkeypatch, tmp_path): + empty_env = tmp_path / "empty.env" + empty_env.touch() + + monkeypatch.setattr( + redmine_ticket_nmi, + "resolve_runtime_profile", + lambda **_kwargs: redmine_ticket_nmi.NmiRuntimeProfile( + mode="production", + env_file=empty_env, + nmi_bin=Path("/tmp/nmi"), + endpoint=None, + ready=False, + problem="NMI env file is empty", + ), + ) + + payload = redmine_ticket_nmi.brief_for_issue( + {"id": 123, "subject": "item 200000001"}, + [{"type": "item_no", "value": "200000001", "confidence": "high"}], + mode="production", + env_file=str(empty_env), + ) + + assert payload["ok"] is False + assert payload["error"] == "nmi_runtime_not_ready" + assert "empty" in payload["runtime"]["problem"].lower() diff --git a/tests/test_note_vs_customer_email.py b/tests/test_note_vs_customer_email.py new file mode 100644 index 0000000..d1c6f61 --- /dev/null +++ b/tests/test_note_vs_customer_email.py @@ -0,0 +1,107 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import tempfile +import unittest +from pathlib import Path + +from redmine_ticket_drafts import ( + KIND_CUSTOMER_EMAIL, + KIND_INTERNAL_NOTE, + add_draft, + assert_write_target_allowed, + host_is_safe_for_writes, + kind_label, + list_drafts, + load_draft, + mark_draft_applied, + normalize_kind, +) + + +class DraftKindsTest(unittest.TestCase): + def test_normalize_legacy(self): + self.assertEqual(KIND_INTERNAL_NOTE, normalize_kind("note")) + self.assertEqual(KIND_CUSTOMER_EMAIL, normalize_kind("helpdesk_reply")) + + def test_customer_email_cannot_be_private(self): + with tempfile.TemporaryDirectory() as td: + with self.assertRaises(ValueError): + add_draft(Path(td), issue_id=1, kind=KIND_CUSTOMER_EMAIL, body="hi", private=True) + + def test_labels_are_explicit(self): + self.assertIn("NOT email", kind_label(KIND_INTERNAL_NOTE)) + self.assertIn("WILL email", kind_label(KIND_CUSTOMER_EMAIL)) + + def test_queue_and_list(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + p = add_draft(root, issue_id=42, kind=KIND_INTERNAL_NOTE, body="staff only", private=True) + self.assertTrue(p.is_file()) + items = list_drafts(root, status="pending") + self.assertEqual(1, len(items)) + self.assertFalse(items[0].get("emails_customer")) + p2 = add_draft(root, issue_id=43, kind=KIND_CUSTOMER_EMAIL, body="hello customer") + items = list_drafts(root, status="pending") + self.assertEqual(2, len(items)) + cust = [i for i in items if i["kind"] == KIND_CUSTOMER_EMAIL][0] + self.assertTrue(cust.get("emails_customer")) + mark_draft_applied(root, cust["id"], result={"ok": True}) + pending = list_drafts(root, status="pending") + self.assertEqual(1, len(pending)) + + def test_write_target_safety(self): + self.assertTrue(host_is_safe_for_writes("http://192.168.50.170/")) + self.assertTrue(host_is_safe_for_writes("http://retired.int.ldrprep.com")) + self.assertFalse(host_is_safe_for_writes("https://rm.ldrprep.com")) + with self.assertRaises(RuntimeError): + assert_write_target_allowed("https://rm.ldrprep.com", allow_prod_writes=False) + assert_write_target_allowed("https://rm.ldrprep.com", allow_prod_writes=True) + assert_write_target_allowed("http://192.168.50.170", allow_prod_writes=False) + + +class CliParseSafetyTest(unittest.TestCase): + def test_customer_email_requires_confirm_flag(self): + from redmine_tickets import build_parser, inject_default_sync_command + + parser = build_parser() + # Missing mandatory confirm flag should fail parse + with self.assertRaises(SystemExit): + parser.parse_args( + inject_default_sync_command( + [ + "--project", + "customer-service", + "customer-email", + "--issue-id", + "1", + "--body", + "hi", + ] + ) + ) + + def test_note_command_exists(self): + from redmine_tickets import build_parser, inject_default_sync_command + + args = build_parser().parse_args( + inject_default_sync_command( + [ + "--project", + "customer-service", + "note", + "--issue-id", + "1", + "--body", + "internal only", + "--private", + ] + ) + ) + self.assertEqual("note", args.command) + self.assertTrue(args.private) + self.assertFalse(args.apply) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_ocr_categories.py b/tests/test_ocr_categories.py new file mode 100644 index 0000000..0e14557 --- /dev/null +++ b/tests/test_ocr_categories.py @@ -0,0 +1,81 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import tempfile +import unittest +from pathlib import Path + +from redmine_ticket_ocr import ( + build_ocr_note, + extract_text_from_file, + looks_like_generic_scan_subject, + suggest_subject_from_text, +) +from redmine_tickets import KNOWN_COMMANDS, build_parser, inject_default_sync_command, is_closed_status + + +class OcrSuggestTest(unittest.TestCase): + def test_generic_subject(self): + self.assertTrue(looks_like_generic_scan_subject("Scanned document from HP")) + self.assertFalse(looks_like_generic_scan_subject("Invoice — Acme Corp")) + + def test_suggest_w2(self): + text = "Form W-2 Wage and Tax Statement\nName: Jane Doe\nCompany: Acme LLC\nJanuary 15, 2026" + subj = suggest_subject_from_text(text, original_subject="Scanned document") + self.assertIn("W-2", subj) + self.assertIn("Jane Doe", subj) + + def test_suggest_vendor_invoice_skips_fee_bullet(self): + text = ( + "- Return Check Fee up to $45.00\n" + "- A 1.5% Service Charge, 18% Annually ($1.00 Minimum) will be charged\n" + "Best Way Disposal\n" + "Invoice Number: 2404778\n" + "Invoice Due Date 9/25/2026\n" + "Total Due $307.52\n" + ) + subj = suggest_subject_from_text(text, original_subject="Scanned document from LDR Office") + self.assertIn("Invoice 2404778", subj) + self.assertIn("Best Way Disposal", subj) + self.assertNotIn("Return Check Fee", subj) + + def test_txt_extract(self): + with tempfile.TemporaryDirectory() as td: + p = Path(td) / "a.txt" + p.write_text("hello ocr world " * 5, encoding="utf-8") + text, method = extract_text_from_file(p) + self.assertEqual("read", method) + self.assertIn("hello", text) + + def test_note_builder(self): + note = build_ocr_note( + issue_id=1, + files=[{"filename": "a.pdf", "method": "pdftotext", "text": "body"}], + proposed_subject="Invoice — X", + ) + self.assertIn("proposed_subject: Invoice — X", note) + self.assertIn("a.pdf", note) + + +class CliRegistrationTest(unittest.TestCase): + def test_commands(self): + self.assertIn("ocr-rename", KNOWN_COMMANDS) + self.assertIn("categories", KNOWN_COMMANDS) + args = build_parser().parse_args( + inject_default_sync_command( + ["ocr-rename", "--issue-id", "1", "--no-queue-note", "--no-queue-subject"] + ) + ) + self.assertEqual("ocr-rename", args.command) + args2 = build_parser().parse_args( + inject_default_sync_command(["categories", "list", "--refresh"]) + ) + self.assertEqual("categories", args2.command) + args3 = build_parser().parse_args( + inject_default_sync_command(["walk", "--open-only", "--max-tickets", "3"]) + ) + self.assertTrue(args3.open_only) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_redmine_contacts.py b/tests/test_redmine_contacts.py new file mode 100644 index 0000000..a5471cb --- /dev/null +++ b/tests/test_redmine_contacts.py @@ -0,0 +1,364 @@ +import argparse +import json +import subprocess +import sys +import tempfile +from pathlib import Path +import unittest + +import redmine_contacts + + +class RedmineContactsBehaviorTest(unittest.TestCase): + class FakeLoopClient: + def __init__(self): + self.updated = [] + + def helpdesk_issues_by_contact(self, contact_id, limit): + return {"issues": [{"issue_id": 9001, "subject": "Sample ticket"}]} + + def issue_with_helpdesk(self, issue_id, message_limit=100): + return {"issue": {"id": issue_id}, "helpdesk_messages": {"messages": []}} + + def update_contact(self, contact_id, fields): + self.updated.append((contact_id, fields)) + + def test_filter_contacts_by_dates_and_tags(self): + contacts = [ + { + "id": 1, + "created_on": "2026-07-01T10:00:00Z", + "updated_on": "2026-07-03T12:00:00Z", + "tag_list": ["customer", "prep"], + }, + { + "id": 2, + "created_on": "2026-06-20T10:00:00Z", + "updated_on": "2026-06-25T12:00:00Z", + "tag_list": ["vendor"], + }, + { + "id": 3, + "created_on": "2026-07-02T10:00:00Z", + "updated_on": "2026-07-02T12:00:00Z", + "tag_list": [], + }, + ] + + filtered = redmine_contacts.filter_contacts( + contacts, + created_after="2026-07-01", + updated_before="2026-07-03", + with_tags=["customer"], + without_tags=["prep"], + no_tags=False, + ) + + self.assertEqual([], [contact["id"] for contact in filtered]) + + filtered_no_tags = redmine_contacts.filter_contacts( + contacts, + created_after=None, + updated_before=None, + with_tags=[], + without_tags=[], + no_tags=True, + ) + + self.assertEqual([3], [contact["id"] for contact in filtered_no_tags]) + + def test_filter_contacts_tag_matching_is_case_insensitive(self): + contacts = [{"id": 1, "tag_list": ["Customer", "Needs-Enriched"]}] + + filtered = redmine_contacts.filter_contacts( + contacts, + with_tags=["customer"], + without_tags=["NEEDS-ENRICHED"], + ) + + self.assertEqual([], [contact["id"] for contact in filtered]) + + filtered_required_only = redmine_contacts.filter_contacts( + contacts, + with_tags=["CUSTOMER"], + without_tags=[], + ) + self.assertEqual([1], [contact["id"] for contact in filtered_required_only]) + + def test_merge_tags_add_and_remove(self): + merged = redmine_contacts.merge_tags( + existing=["customer", "needs-enriched"], + add_tags=["needs-nmi-sync", "customer"], + remove_tags=["needs-enriched"], + ) + self.assertEqual(["customer", "needs-nmi-sync"], merged) + + def test_build_custom_field_patch_by_slug(self): + existing = [ + {"id": 70, "name": "Account Number", "value": "old"}, + {"id": 88, "name": "X/Twitter", "value": "@old"}, + ] + + patch = redmine_contacts.build_custom_field_values_patch( + existing, + {"account-number": "A-123", "x-twitter": "@new"}, + ) + + self.assertEqual({"70": "A-123", "88": "@new"}, patch) + + def test_build_custom_field_patch_rejects_unknown_slug(self): + with self.assertRaises(redmine_contacts.RedmineError): + redmine_contacts.build_custom_field_values_patch( + [{"id": 70, "name": "Account Number", "value": "old"}], + {"linkedin": "https://example.com/u/abc"}, + ) + + def test_has_contact_filters_detects_configured_filters(self): + args = argparse.Namespace( + created_after=None, + created_before=None, + updated_after="2026-07-01", + updated_before=None, + with_tag=[], + without_tag=[], + no_tags=False, + ) + self.assertTrue(redmine_contacts.has_contact_filters(args)) + + def test_has_contact_filters_false_without_filters(self): + args = argparse.Namespace( + created_after=None, + created_before=None, + updated_after=None, + updated_before=None, + with_tag=[], + without_tag=[], + no_tags=False, + ) + self.assertFalse(redmine_contacts.has_contact_filters(args)) + + def test_resolve_fetch_cache_mode_uses_filtered_when_filters_present_and_mode_auto(self): + args = argparse.Namespace( + cache_mode="auto", + created_after=None, + created_before=None, + updated_after="2026-07-01", + updated_before=None, + with_tag=[], + without_tag=[], + no_tags=False, + ) + self.assertEqual("filtered", redmine_contacts.resolve_fetch_cache_mode(args)) + + def test_resolve_fetch_cache_mode_uses_full_when_no_filters_and_mode_auto(self): + args = argparse.Namespace( + cache_mode="auto", + created_after=None, + created_before=None, + updated_after=None, + updated_before=None, + with_tag=[], + without_tag=[], + no_tags=False, + ) + self.assertEqual("full", redmine_contacts.resolve_fetch_cache_mode(args)) + + def test_build_update_fields_for_contact(self): + contact = { + "id": 42, + "tag_list": ["Customer"], + "custom_fields": [{"id": 70, "name": "Account Number", "value": "old"}], + } + args = argparse.Namespace( + set=["job_title=Buyer"], + add_tag=["needs-nmi-sync"], + remove_tag=["Customer"], + set_custom_field=["account-number=A-9"], + ) + + fields = redmine_contacts.build_update_fields_for_contact(contact, args) + + self.assertEqual("Buyer", fields["job_title"]) + self.assertEqual(["needs-nmi-sync"], fields["tag_list"]) + self.assertEqual({"70": "A-9"}, fields["custom_field_values"]) + + def test_select_batch_contacts_respects_filters_and_limit(self): + contacts = [ + {"id": 1, "tag_list": ["customer"], "updated_on": "2026-07-01T00:00:00Z"}, + {"id": 2, "tag_list": ["customer"], "updated_on": "2026-06-01T00:00:00Z"}, + {"id": 3, "tag_list": ["vendor"], "updated_on": "2026-05-01T00:00:00Z"}, + ] + args = argparse.Namespace( + created_after=None, + created_before=None, + updated_after=None, + updated_before=None, + with_tag=["customer"], + without_tag=[], + no_tags=False, + max_contacts=1, + order="as-is", + ) + + selected = redmine_contacts.select_batch_contacts(contacts, args) + self.assertEqual([1], [item["id"] for item in selected]) + + def test_select_batch_contacts_orders_oldest_updated_first(self): + contacts = [ + {"id": 10, "tag_list": ["customer"], "updated_on": "2026-07-03T00:00:00Z"}, + {"id": 11, "tag_list": ["customer"], "updated_on": "2026-04-01T00:00:00Z"}, + {"id": 12, "tag_list": ["customer"], "updated_on": "2026-06-01T00:00:00Z"}, + ] + args = argparse.Namespace( + created_after=None, + created_before=None, + updated_after=None, + updated_before=None, + with_tag=["customer"], + without_tag=[], + no_tags=False, + max_contacts=2, + order="oldest-updated-first", + ) + + selected = redmine_contacts.select_batch_contacts(contacts, args) + self.assertEqual([11, 12], [item["id"] for item in selected]) + + def test_select_batch_contacts_oldest_updated_first_puts_missing_timestamps_last(self): + contacts = [ + {"id": 21, "tag_list": ["customer"], "updated_on": None}, + {"id": 22, "tag_list": ["customer"], "updated_on": "2026-01-01T00:00:00Z"}, + {"id": 23, "tag_list": ["customer"], "updated_on": "2026-02-01T00:00:00Z"}, + ] + args = argparse.Namespace( + created_after=None, + created_before=None, + updated_after=None, + updated_before=None, + with_tag=["customer"], + without_tag=[], + no_tags=False, + max_contacts=3, + order="oldest-updated-first", + ) + + selected = redmine_contacts.select_batch_contacts(contacts, args) + self.assertEqual([22, 23, 21], [item["id"] for item in selected]) + + def test_load_loop_profile_normalizes_optional_defaults(self): + with tempfile.TemporaryDirectory() as tmpdir: + profile_path = Path(tmpdir) / "profile.json" + profile_path.write_text( + json.dumps( + { + "default_tag": "ai-checked", + "filter_1": {"type": "builtin", "name": "fixed-ticket-limit", "limit": 3}, + "filter_2": {"type": "command", "argv": ["python3", "noop.py"]}, + } + ), + encoding="utf-8", + ) + + profile = redmine_contacts.load_loop_profile(profile_path) + + self.assertEqual("ai-checked", profile["default_tag"]) + self.assertEqual([], profile["remove_tags"]) + self.assertEqual(False, profile["ticket_fetch"]["fetch_full_issues"]) + self.assertEqual(25, profile["ticket_fetch"]["message_limit"]) + + def test_run_filter1_builtin_fixed_ticket_limit_returns_integer_decision(self): + profile = { + "filter_1": {"type": "builtin", "name": "fixed-ticket-limit", "limit": 4}, + } + + decision = redmine_contacts.run_loop_filter1(profile, {"id": 123}) + + self.assertEqual(4, decision["result"]) + self.assertEqual("builtin fixed-ticket-limit", decision["reason"]) + + def test_run_loop_filter2_command_passes_bundle_and_dry_run_env(self): + with tempfile.TemporaryDirectory() as tmpdir: + script_path = Path(tmpdir) / "filter2.py" + script_path.write_text( + "import json, os, sys\n" + "payload = json.load(sys.stdin)\n" + "json.dump({\n" + " 'result': True,\n" + " 'contact_id': payload['contact']['id'],\n" + " 'dry_run': os.environ.get('REDMINE_LOOP_DRY_RUN'),\n" + "}, sys.stdout)\n", + encoding="utf-8", + ) + profile = { + "filter_2": {"type": "command", "argv": [sys.executable, str(script_path)]}, + } + + result = redmine_contacts.run_loop_filter2( + profile, + {"contact": {"id": 123}, "issues": []}, + runtime={"dry_run": True, "base_url": "http://redmine.test", "project": "customer-service"}, + ) + + self.assertEqual(True, result["result"]) + self.assertEqual(123, result["contact_id"]) + self.assertEqual("1", result["dry_run"]) + + def test_run_loop_for_contact_false_filter2_plans_default_tag_without_applying(self): + with tempfile.TemporaryDirectory() as tmpdir: + script_path = Path(tmpdir) / "filter2.py" + script_path.write_text( + "import json, sys\n" + "json.load(sys.stdin)\n" + "json.dump({'result': False, 'summary': 'noop'}, sys.stdout)\n", + encoding="utf-8", + ) + profile = { + "name": "test-loop", + "default_tag": "ai-checked", + "remove_tags": [], + "filter_1": {"type": "builtin", "name": "fixed-ticket-limit", "limit": 1}, + "ticket_fetch": {"fetch_full_issues": False, "message_limit": 25, "sanitize_text": True}, + "filter_2": {"type": "command", "argv": [sys.executable, str(script_path)]}, + } + client = self.FakeLoopClient() + + outcome = redmine_contacts.run_loop_for_contact( + client, + profile, + {"id": 123, "tag_list": ["needs-review"]}, + runtime={"dry_run": True, "base_url": "http://redmine.test", "project": "customer-service"}, + ) + + self.assertEqual("default-tag", outcome["action"]) + self.assertEqual({"tag_list": ["ai-checked", "needs-review"]}, outcome["fields"]) + self.assertEqual([], client.updated) + + def test_search_help_shows_examples_and_doc_path(self): + repo = Path(__file__).resolve().parents[1] + script = repo / "redmine_contacts.py" + search_help = subprocess.run( + [sys.executable, str(script), "search", "--help"], + cwd=repo, + capture_output=True, + text=True, + check=False, + ) + self.assertEqual(search_help.returncode, 0, search_help.stderr) + self.assertIn("docs/redmine_contacts_search.md", search_help.stdout) + self.assertIn("--with-tag promotional-spam", search_help.stdout) + self.assertIn("helpdesk_ticket.contact_id", search_help.stdout) + + root_help = subprocess.run( + [sys.executable, str(script), "--help"], + cwd=repo, + capture_output=True, + text=True, + check=False, + ) + self.assertEqual(root_help.returncode, 0, root_help.stderr) + self.assertIn("search --help", root_help.stdout) + self.assertIn("docs/redmine_contacts_search.md", root_help.stdout) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_redmine_ticket_query_and_view.py b/tests/test_redmine_ticket_query_and_view.py new file mode 100644 index 0000000..db5f7de --- /dev/null +++ b/tests/test_redmine_ticket_query_and_view.py @@ -0,0 +1,100 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import unittest +from datetime import datetime, timedelta, timezone + +from redmine_ticket_query import QueryError, compile_query, filter_by_query, issue_matches +from redmine_ticket_view import render_ticket_markdown + + +class PropertyQueryTest(unittest.TestCase): + def setUp(self): + self.issue = { + "id": 1, + "subject": "Please send verification code", + "updated_on": datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"), + "status": {"name": "New"}, + "assigned_to": None, + "helpdesk_ticket": { + "from_address": "same@example.com", + "contact": { + "id": 9, + "email": "same@example.com", + "company": "same@example.com", + "name": "Same", + }, + }, + } + + def test_includes(self): + self.assertTrue(issue_matches(self.issue, 'subject.includes("verification code")')) + self.assertFalse(issue_matches(self.issue, 'subject.includes("magic link")')) + + def test_path_eq_path(self): + expr = "helpdesk_ticket.contact.email == helpdesk_ticket.contact.company" + self.assertTrue(issue_matches(self.issue, expr)) + + def test_is_null(self): + self.assertTrue(issue_matches(self.issue, "assigned_to IS NULL")) + self.issue["assigned_to"] = {"name": "Jason T."} + self.assertFalse(issue_matches(self.issue, "assigned_to IS NULL")) + self.assertTrue(issue_matches(self.issue, 'assigned_to != "Other"')) + self.assertFalse(issue_matches(self.issue, 'assigned_to != "Jason T."')) + + def test_and_or_parens(self): + expr = ( + '(subject.includes("verification code") OR subject.includes("magic link")) ' + 'AND (assigned_to IS NULL)' + ) + self.assertTrue(issue_matches(self.issue, expr)) + + def test_ampersand_and(self): + self.assertTrue( + issue_matches(self.issue, 'subject.includes("verification") && status.name == "New"') + ) + + def test_since(self): + self.assertTrue(issue_matches(self.issue, 'updated_on.since("1 hour")')) + old = { + **self.issue, + "updated_on": (datetime.now(timezone.utc) - timedelta(days=10)) + .isoformat() + .replace("+00:00", "Z"), + } + self.assertFalse(issue_matches(old, 'updated_on.since("1 day")')) + + def test_filter_multiple_and(self): + issues = [self.issue, {**self.issue, "id": 2, "subject": "hello"}] + out = filter_by_query( + issues, + expressions=['subject.includes("verification")', 'status.name == "New"'], + ) + self.assertEqual([1], [i["id"] for i in out]) + + def test_bad_expr(self): + with self.assertRaises(QueryError): + compile_query("subject ~~ boom") + + +class MarkdownRenderTest(unittest.TestCase): + def test_render_contains_core_fields(self): + md = render_ticket_markdown( + { + "id": 42, + "subject": "Test", + "status": {"name": "New"}, + "project": {"name": "customer-service"}, + "description": "Body here", + "journals": [], + "updated_on": "2026-07-14T00:00:00Z", + "helpdesk_ticket": {"from_address": "a@b.c", "contact": {"id": 1}}, + } + ) + self.assertIn("# #42", md) + self.assertIn("Body here", md) + self.assertIn("a@b.c", md) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_redmine_tickets.py b/tests/test_redmine_tickets.py new file mode 100644 index 0000000..247ed20 --- /dev/null +++ b/tests/test_redmine_tickets.py @@ -0,0 +1,541 @@ +import tempfile +import unittest +from pathlib import Path + +import redmine_tickets + + +class RedmineTicketsCacheTest(unittest.TestCase): + def test_ticket_cache_skips_when_manifest_updated_on_matches(self): + with tempfile.TemporaryDirectory() as tmp: + cache = redmine_tickets.TicketCache(Path(tmp), "customer-service") + issue = {"id": 123, "updated_on": "2026-07-01T10:00:00Z"} + cache.write_issue(issue) + + self.assertFalse(cache.should_fetch({"id": 123, "updated_on": "2026-07-01T10:00:00Z"})) + self.assertTrue(cache.should_fetch({"id": 123, "updated_on": "2026-07-02T10:00:00Z"})) + + def test_manifest_records_action_idempotency(self): + with tempfile.TemporaryDirectory() as tmp: + cache = redmine_tickets.TicketCache(Path(tmp), "customer-service") + cache.record_action(123, "spam-check", "not-spam", {"score": 0}) + + self.assertTrue(cache.has_action(123, "spam-check")) + entry = cache.manifest["tickets"]["123"]["actions"]["spam-check"] + self.assertEqual("not-spam", entry["result"]) + self.assertEqual({"score": 0}, entry["metadata"]) + + def test_ticket_cache_can_use_alternate_state_store(self): + class MemoryStateStore: + def __init__(self): + self.states = {} + + def get_ticket_state(self, issue_id): + return self.states.get(str(issue_id)) + + def upsert_ticket_state(self, issue_id, fields): + self.states.setdefault(str(issue_id), {}).update(fields) + + def has_action(self, issue_id, action_name): + state = self.get_ticket_state(issue_id) or {} + return action_name in state.get("actions", {}) + + def record_action(self, issue_id, action_name, result, metadata=None): + state = self.states.setdefault(str(issue_id), {}) + state.setdefault("actions", {})[action_name] = { + "result": result, + "metadata": metadata or {}, + } + + with tempfile.TemporaryDirectory() as tmp: + store = MemoryStateStore() + cache = redmine_tickets.TicketCache(Path(tmp), "customer-service", state_store=store) + cache.write_issue({"id": 123, "updated_on": "2026-07-01T10:00:00Z"}) + cache.record_action(123, "spam-check", "spam", {"score": 2}) + + self.assertFalse(cache.should_fetch({"id": 123, "updated_on": "2026-07-01T10:00:00Z"})) + self.assertTrue(store.has_action(123, "spam-check")) + + +class RedmineTicketsBehaviorTest(unittest.TestCase): + def test_filter_issues_by_status_category_and_updated_since(self): + issues = [ + { + "id": 1, + "updated_on": "2026-07-01T10:00:00Z", + "status": {"name": "New"}, + "category": {"name": "Spam"}, + }, + { + "id": 2, + "updated_on": "2026-06-01T10:00:00Z", + "status": {"name": "Closed"}, + "category": {"name": "Spam"}, + }, + { + "id": 3, + "updated_on": "2026-07-02T10:00:00Z", + "status": {"name": "New"}, + "category": {"name": "Support"}, + }, + ] + + filtered = redmine_tickets.filter_issues( + issues, + updated_since="2026-07-01", + status="New", + category="Spam", + ) + + self.assertEqual([1], [issue["id"] for issue in filtered]) + + def test_select_issues_orders_and_limits(self): + issues = [ + {"id": 1, "updated_on": "2026-07-01T00:00:00Z"}, + {"id": 2, "updated_on": "2026-07-03T00:00:00Z"}, + {"id": 3, "updated_on": "2026-07-02T00:00:00Z"}, + ] + + selected = redmine_tickets.select_issues( + issues, + order="newest-updated-first", + max_tickets=2, + ) + + self.assertEqual([2, 3], [issue["id"] for issue in selected]) + + +class RedmineTicketsDecompositionTest(unittest.TestCase): + def test_decompose_repackify_ticket_plans_timeline_entity_and_contact_proposal(self): + issue = { + "id": 41424, + "subject": "Gaylord inquiry-RePackify", + "created_on": "2026-06-23T13:06:02Z", + "updated_on": "2026-06-23T13:06:02Z", + "description": "Hi there,\nWe’re interested in purchasing used Gaylord boxes and wanted to see if you have any available.\nSarah Johnson\nsarah@repackify.com", + "helpdesk_ticket": { + "from_address": "support@repackify.com", + "contact": {"id": 4470, "email": "support@repackify.com", "name": "support"}, + }, + } + + plan = redmine_tickets.decompose_issue_dry_run(issue) + + self.assertEqual(41424, plan["source"]["redmine_issue_id"]) + self.assertIn("partnership-buy-from-us", plan["buckets"]) + self.assertEqual("Timeline/2026-06/2026-06-23_repackify-gaylord-inquiry.md", plan["timeline_events"][0]["path"]) + self.assertEqual("entities/organizations/repackify.md", plan["entities"][0]["path"]) + self.assertIn("wiki/", "\n".join(plan["compatibility_notes"])) + self.assertEqual(4470, plan["redmine_contact_updates"][0]["contact_id"]) + self.assertIn("buy-from-us", plan["redmine_contact_updates"][0]["add_tags"]) + + def test_decompose_u2wear_forwarded_message_extracts_fulfillment_context(self): + issue = { + "id": 41448, + "subject": "Fwd: Have You Found a Suitable Returns Management Provider?", + "created_on": "2026-06-23T19:38:02Z", + "description": "---------- Forwarded message ---------\nFrom: The Whole Shebang \nSubject: Re: Have You Found a Suitable Returns Management Provider?\n\nHi Christian,\nI am now back in the US and ready to activate the winter fulfillment strategy for my brands, ELORA and u2wear.\nDoes your team still have the capacity to support the fulfillment and returns workflow we previously discussed?\n\nOn Mon, Mar 2, 2026 at 10:29 AM wrote:", + "helpdesk_ticket": { + "from_address": "sales@u2wear.com", + "contact": {"id": 3919, "email": "sales@u2wear.com", "name": "The Whole Shebang"}, + }, + } + + cleaned = redmine_tickets.clean_ticket_text(issue) + self.assertIn("winter fulfillment strategy", cleaned) + self.assertNotIn("Forwarded message", cleaned) + + plan = redmine_tickets.decompose_issue_dry_run(issue) + + self.assertIn("customer-new-project", plan["buckets"]) + self.assertIn("meaningful-event", plan["buckets"]) + self.assertIn("fulfillment/returns opportunity", plan["analysis"]["summary"]) + self.assertTrue(any("returns management" in concept["name"] for concept in plan["concepts"])) + self.assertNotIn("rate-sheet-gap", plan["tags"]) + + def test_decompose_snuggle_puppy_ticket_detects_rate_sheet_gap(self): + issue = { + "id": 41504, + "subject": "3PL & FBA Returns", + "created_on": "2026-06-29T14:11:51Z", + "updated_on": "2026-07-06T19:53:44Z", + "description": "From: Justine Micks \nSubject: 3PL & FBA Returns\n\nMessage Body:\nHello, we are looking for new potential partners for warehousing & FBA returns. I would be interested in capabilities and pricing.\n\n--\nThis e-mail was sent from a contact form on (https://ldrprep.com)", + "helpdesk_ticket": { + "from_address": "justine.micks@snugglepuppy.com", + "contact": {"id": 4476, "email": "justine.micks@snugglepuppy.com", "name": "justine micks"}, + }, + "journals": [ + {"created_on": "2026-07-06T18:29:02Z", "user": {"name": "Anonymous"}, "notes": "That link doesn't seem to calculate anything. Could you send me a rate sheet?"}, + {"created_on": "2026-07-06T19:53:44Z", "user": {"name": "Matthew C."}, "notes": "We do not have a sheet for returns pricing. Rates are typically as follows."}, + ], + } + + cleaned = redmine_tickets.clean_ticket_text(issue) + self.assertNotIn("Message Body:", cleaned) + self.assertNotIn("This e-mail was sent from", cleaned) + self.assertIn("Could you send me a rate sheet?", cleaned) + + plan = redmine_tickets.decompose_issue_dry_run(issue) + + self.assertIn("customer-new-project", plan["buckets"]) + self.assertIn("meaningful-event", plan["buckets"]) + self.assertIn("rate-sheet-gap", plan["tags"]) + self.assertEqual("Timeline/2026-07/2026-07-06_snuggle-puppy-3pl-fba-returns.md", plan["timeline_events"][0]["path"]) + self.assertEqual("entities/organizations/snuggle-puppy.md", plan["entities"][0]["path"]) + self.assertIn("rate sheet", plan["analysis"]["summary"].lower()) + self.assertTrue(any("calculator" in point.lower() for point in plan["analysis"]["key_points"])) + self.assertTrue(plan["review_required"]) + + +class RedmineTicketsSyncTest(unittest.TestCase): + class FakeClient: + def __init__(self): + self.full_fetches = [] + + def list_issues(self, **kwargs): + return [ + {"id": 100, "updated_on": "2026-07-01T00:00:00Z", "subject": "new"}, + {"id": 101, "updated_on": "2026-07-02T00:00:00Z", "subject": "newer"}, + ] + + def get_issue_with_includes(self, issue_id, includes): + self.full_fetches.append((issue_id, tuple(includes))) + return {"id": issue_id, "updated_on": f"2026-07-0{issue_id - 99}T00:00:00Z", "helpdesk_ticket": {"id": issue_id + 1000}} + + def test_sync_fetches_only_missing_or_updated_tickets(self): + with tempfile.TemporaryDirectory() as tmp: + cache = redmine_tickets.TicketCache(Path(tmp), "customer-service") + cache.write_issue({"id": 100, "updated_on": "2026-07-01T00:00:00Z"}) + client = self.FakeClient() + + result = redmine_tickets.sync_tickets( + client, + cache, + updated_since=None, + updated_before=None, + status=None, + category=None, + max_tickets=None, + order="as-is", + apply=True, + include_journals=True, + ) + + self.assertEqual(2, result.seen) + self.assertEqual(1, result.skipped_cached) + self.assertEqual(1, result.fetched) + self.assertEqual([(101, ("journals", "helpdesk"))], client.full_fetches) + self.assertTrue(cache.issue_path(101).is_file()) + + +class RedmineTicketsSpamCheckTest(unittest.TestCase): + def test_score_ticket_spam_flags_promotional_helpdesk_ticket(self): + issue = { + "id": 41789, + "subject": "Up to 20% Off Select Fans & Blowers", + "helpdesk_ticket": { + "from_address": "offers@example.com", + "contact": {"email": "offers@example.com"}, + }, + "journals": [{"notes": "Limited time sale and discount coupon."}], + } + + result = redmine_tickets.score_ticket_spam(issue) + + self.assertTrue(result.is_spam) + self.assertEqual("spam-promotional", result.category) + self.assertGreaterEqual(result.score, 3) + self.assertIn("promotional language", "; ".join(result.reasons)) + + def test_score_ticket_classifies_sales_pitch_as_promotional_spam(self): + issue = { + "id": 50001, + "subject": "Commercial cleaning services for your warehouse", + "helpdesk_ticket": {"from_address": "sales@sparkleclean.example"}, + "journals": [{"notes": "We would like to offer janitorial cleaning services for your facility."}], + } + + result = redmine_tickets.score_ticket_spam(issue) + + self.assertEqual("spam-promotional", result.category) + self.assertTrue(result.is_spam) + + def test_score_ticket_classifies_fba_notification_as_operational_not_promotional(self): + issue = { + "id": 41753, + "subject": "Automated unfulfillable FBA inventory removal notification", + "helpdesk_ticket": {"from_address": "fba-noreply@amazon.com"}, + "journals": [{"notes": "This is an automated notification about your FBA inventory."}], + } + + result = redmine_tickets.score_ticket_spam(issue) + + self.assertEqual("automated-operational", result.category) + self.assertFalse(result.is_spam) + + def test_score_ticket_classifies_receipts_bounces_and_security_as_operational(self): + examples = [ + ("USPS eReceipt", "donotreply@ereceipt.usps.gov", "Money Order Fee Total"), + ("Undelivered Mail Returned to Sender", "bounces-noreply@mandrillapp.com", "message could not be delivered"), + ("Critical security alert", "no-reply@accounts.google.com", "saved passwords were found online"), + ("Your call transcription from 240-521-4036", "noreply@voip.ms", "This email contains the transcription of the following call"), + ("New voicemail in mailbox 1", "noreply@voipinterface.net", "You have a new message in mailbox"), + ] + for subject, sender, body in examples: + with self.subTest(subject=subject): + issue = { + "id": 90001, + "subject": subject, + "helpdesk_ticket": {"from_address": sender}, + "description": body, + } + + result = redmine_tickets.score_ticket_spam(issue) + + self.assertEqual("automated-operational", result.category) + self.assertFalse(result.is_spam) + + def test_score_ticket_does_not_close_established_or_inbound_business_inquiries(self): + examples = [ + ( + "Fwd: Have You Found a Suitable Returns Management Provider?", + "sales@u2wear.com", + "Does your team still have the capacity to support the fulfillment and returns workflow we previously discussed?", + ), + ( + "Gaylord inquiry-RePackify", + "support@repackify.com", + "We’re interested in purchasing used Gaylord boxes and wanted to see if you have any available.", + ), + ] + for subject, sender, body in examples: + with self.subTest(subject=subject): + issue = { + "id": 90002, + "subject": subject, + "helpdesk_ticket": {"from_address": sender}, + "description": body, + } + + result = redmine_tickets.score_ticket_spam(issue) + + self.assertEqual("unknown-review", result.category) + self.assertFalse(result.is_spam) + + def test_score_ticket_does_not_treat_ldr_reply_to_real_inquiry_as_spam(self): + issue = { + "id": 41504, + "subject": "3PL & FBA Returns", + "description": "Hello, do you process FBA returns for brands?", + "helpdesk_ticket": {"from_address": "justine.micks@snugglepuppy.com"}, + "journals": [ + {"user": {"name": "Matthew C."}, "notes": "Rates are typically as follows. We advertise prep pricing separately."} + ], + } + + result = redmine_tickets.score_ticket_spam(issue) + + self.assertEqual("unknown-review", result.category) + self.assertFalse(result.is_spam) + + def test_score_ticket_does_not_treat_customer_seo_context_as_sales_spam(self): + issue = { + "id": 41237, + "subject": "RE: New message from our team may require attention", + "description": "They are both the same. We just use different titles for SEO. Please ensure all repackaged outfits include pants.", + "helpdesk_ticket": {"from_address": "manager@sonisha.com"}, + "journals": [], + } + + result = redmine_tickets.score_ticket_spam(issue) + + self.assertEqual("unknown-review", result.category) + self.assertFalse(result.is_spam) + + def test_run_spam_check_skips_already_checked_ticket(self): + with tempfile.TemporaryDirectory() as tmp: + cache = redmine_tickets.TicketCache(Path(tmp), "customer-service") + cache.write_issue( + { + "id": 1, + "updated_on": "2026-07-01T00:00:00Z", + "subject": "Up to 20% Off", + "helpdesk_ticket": {"from_address": "deals@example.com"}, + "journals": [{"notes": "coupon discount"}], + } + ) + cache.record_action(1, "spam-check", "spam", {"score": 4}) + + results = redmine_tickets.run_spam_check(cache, apply=True) + + self.assertEqual(0, len(results)) + + +class RedmineTicketsCloseIfSpamTest(unittest.TestCase): + class FakeClient: + def __init__(self): + self.updated = [] + self.contact_lookups = [] + self.contacts = {} + self.contact_updates = [] + + def get_contact(self, contact_id): + return self.contacts.get(contact_id, {"id": contact_id, "tag_list": []}) + + def update_contact(self, contact_id, fields, impersonate_user=None): + self.contact_updates.append((contact_id, fields, impersonate_user)) + + def recent_issues_for_contact(self, contact_id, limit): + self.contact_lookups.append((contact_id, limit)) + return [ + { + "id": 10, + "subject": "Up to 20% Off Select Fans & Blowers", + "helpdesk_ticket": {"from_address": "reply@globalindustrial.com"}, + "journals": [], + }, + { + "id": 11, + "subject": "Limited time sale", + "helpdesk_ticket": {"from_address": "reply@globalindustrial.com"}, + "journals": [], + }, + ] + + def update_issue(self, issue_id, fields, impersonate_user=None): + self.updated.append((issue_id, fields, impersonate_user)) + + def test_close_if_spam_dry_run_builds_payload_and_contact_review(self): + with tempfile.TemporaryDirectory() as tmp: + cache = redmine_tickets.TicketCache(Path(tmp), "customer-service") + cache.write_issue( + { + "id": 41788, + "updated_on": "2026-07-07T20:38:01Z", + "subject": "Up to 20% Off Select Fans & Blowers", + "helpdesk_ticket": { + "from_address": "reply@globalindustrial.com", + "contact": {"id": 321, "email": "reply@globalindustrial.com"}, + }, + "journals": [], + } + ) + cache.record_action(41788, "spam-check", "spam", {"score": 2}) + client = self.FakeClient() + + plans = redmine_tickets.run_close_if_spam( + client, + cache, + close_status_id=5, + contact_ticket_limit=10, + apply=False, + ) + + self.assertEqual(1, len(plans)) + self.assertEqual(41788, plans[0].issue_id) + self.assertEqual({"status_id": 5, "notes": "Closed as likely spam by redmine_tickets.py close-if-spam."}, plans[0].payload) + self.assertEqual(321, plans[0].contact_id) + self.assertTrue(plans[0].contact_review["likely_spammer"]) + self.assertEqual([(321, 10)], client.contact_lookups) + self.assertEqual([], client.updated) + self.assertFalse(cache.has_action(41788, "close-if-spam")) + + def test_close_if_spam_apply_updates_issue_and_records_idempotency(self): + with tempfile.TemporaryDirectory() as tmp: + cache = redmine_tickets.TicketCache(Path(tmp), "customer-service") + cache.write_issue({"id": 1, "updated_on": "2026-07-01T00:00:00Z", "helpdesk_ticket": {}}) + cache.record_action(1, "spam-check", "spam", {"score": 2}) + client = self.FakeClient() + + redmine_tickets.run_close_if_spam(client, cache, close_status_id=5, apply=True) + second = redmine_tickets.run_close_if_spam(client, cache, close_status_id=5, apply=True) + + self.assertEqual( + [(1, {"status_id": 5, "notes": "Closed as likely spam by redmine_tickets.py close-if-spam."}, "rebot")], + client.updated, + ) + self.assertEqual([], second) + self.assertTrue(cache.has_action(1, "close-if-spam")) + + def test_close_if_spam_apply_can_override_impersonation_user(self): + with tempfile.TemporaryDirectory() as tmp: + cache = redmine_tickets.TicketCache(Path(tmp), "customer-service") + cache.write_issue({"id": 2, "updated_on": "2026-07-01T00:00:00Z", "helpdesk_ticket": {}}) + cache.record_action(2, "spam-check", "spam", {"score": 2}) + client = self.FakeClient() + + redmine_tickets.run_close_if_spam( + client, + cache, + close_status_id=5, + apply=True, + impersonate_user="alternate-agent", + ) + + self.assertEqual("alternate-agent", client.updated[0][2]) + + def test_close_if_spam_skips_automated_operational_ticket_by_default(self): + with tempfile.TemporaryDirectory() as tmp: + cache = redmine_tickets.TicketCache(Path(tmp), "customer-service") + cache.write_issue( + { + "id": 41753, + "updated_on": "2026-07-01T00:00:00Z", + "subject": "Automated unfulfillable FBA inventory removal notification", + "helpdesk_ticket": {"from_address": "fba-noreply@amazon.com", "contact": {"id": 51}}, + } + ) + cache.record_action(41753, "spam-check", "automated-operational", {"score": 2, "category": "automated-operational"}) + client = self.FakeClient() + + plans = redmine_tickets.run_close_if_spam(client, cache, close_status_id=5, apply=False) + + self.assertEqual([], plans) + self.assertEqual([], client.contact_lookups) + + def test_close_if_spam_skips_contact_tagged_customer(self): + with tempfile.TemporaryDirectory() as tmp: + cache = redmine_tickets.TicketCache(Path(tmp), "customer-service") + cache.write_issue( + { + "id": 500, + "updated_on": "2026-07-01T00:00:00Z", + "subject": "Up to 20% Off", + "helpdesk_ticket": {"from_address": "deals@example.com", "contact": {"id": 777}}, + } + ) + cache.record_action(500, "spam-check", "spam", {"score": 2, "category": "spam-promotional"}) + client = self.FakeClient() + client.contacts[777] = {"id": 777, "tag_list": ["customer"]} + + plans = redmine_tickets.run_close_if_spam(client, cache, close_status_id=5, apply=False) + + self.assertEqual([], plans) + self.assertEqual([], client.updated) + self.assertEqual([], client.contact_lookups) + + def test_tag_maybe_customer_contacts_adds_review_tag_for_inbound_inquiry(self): + with tempfile.TemporaryDirectory() as tmp: + cache = redmine_tickets.TicketCache(Path(tmp), "customer-service") + cache.write_issue( + { + "id": 501, + "updated_on": "2026-07-01T00:00:00Z", + "subject": "Gaylord inquiry-RePackify", + "description": "We’re interested in purchasing used Gaylord boxes.", + "helpdesk_ticket": {"from_address": "support@repackify.com", "contact": {"id": 888}}, + } + ) + cache.record_action( + 501, + "spam-check", + "not-spam", + {"score": 0, "category": "unknown-review", "reasons": ["inbound or established business inquiry"]}, + ) + client = self.FakeClient() + client.contacts[888] = {"id": 888, "tag_list": ["vendor"]} + + results = redmine_tickets.run_tag_maybe_customer_contacts(client, cache, apply=True) + + self.assertEqual([888], [result["contact_id"] for result in results]) + self.assertEqual([(888, {"tag_list": ["maybe-customer", "vendor"]}, "rebot")], client.contact_updates) diff --git a/tests/test_redmine_tickets_remote_filters.py b/tests/test_redmine_tickets_remote_filters.py new file mode 100644 index 0000000..d003aee --- /dev/null +++ b/tests/test_redmine_tickets_remote_filters.py @@ -0,0 +1,104 @@ +#!/usr/bin/env python3 +"""Unit tests for server-side remote filter helpers in redmine_tickets.py.""" + +from __future__ import annotations + +import unittest +from datetime import datetime, timezone +from pathlib import Path +import sys + +ROOT = Path(__file__).resolve().parents[1] +if str(ROOT) not in sys.path: + sys.path.insert(0, str(ROOT)) + +from redmine_tickets import ( # noqa: E402 + build_parser, + build_remote_updated_on_value, + coerce_time_value, + format_redmine_time, + inject_default_sync_command, +) +from redmine_contacts import RedmineError # noqa: E402 + + +class RemoteFilterHelpersTest(unittest.TestCase): + def test_format_redmine_time_date_only(self): + dt = datetime(2026, 6, 14, 12, 30, tzinfo=timezone.utc) + self.assertEqual("2026-06-14", format_redmine_time(dt, date_only=True)) + self.assertEqual("2026-06-14T12:30:00Z", format_redmine_time(dt, date_only=False)) + + def test_coerce_absolute_date_is_date_only(self): + dt, date_only = coerce_time_value("2026-06-14") + self.assertTrue(date_only) + self.assertEqual(dt.date().isoformat(), "2026-06-14") + + def test_coerce_relative_days(self): + now = datetime(2026, 7, 14, 12, 0, 0, tzinfo=timezone.utc) + dt, date_only = coerce_time_value("30d", now=now) + self.assertFalse(date_only) + self.assertEqual(dt.isoformat().replace("+00:00", "Z"), "2026-06-14T12:00:00Z") + + def test_coerce_relative_days_ago_phrase(self): + now = datetime(2026, 7, 14, 12, 0, 0, tzinfo=timezone.utc) + dt, date_only = coerce_time_value("2 days ago", now=now) + self.assertFalse(date_only) + self.assertEqual(dt.isoformat().replace("+00:00", "Z"), "2026-07-12T12:00:00Z") + + def test_build_remote_updated_on_raw(self): + self.assertEqual( + ">=2014-01-02T08:12:32Z", + build_remote_updated_on_value(raw=">=2014-01-02T08:12:32Z"), + ) + + def test_build_remote_updated_on_bare_timestamp_implies_gte(self): + self.assertEqual( + ">=2026-06-14", + build_remote_updated_on_value(raw="2026-06-14"), + ) + + def test_build_remote_updated_on_since_date(self): + self.assertEqual( + ">=2026-06-14", + build_remote_updated_on_value(since="2026-06-14"), + ) + + def test_build_remote_updated_on_range(self): + self.assertEqual( + "><2026-01-01|2026-06-01", + build_remote_updated_on_value(since="2026-01-01", before="2026-06-01"), + ) + + def test_build_remote_updated_on_rejects_mixed_raw_and_convenience(self): + with self.assertRaises(RedmineError): + build_remote_updated_on_value(raw=">=2026-01-01", since="30d") + + def test_cli_accepts_remote_flags_without_explicit_sync_word(self): + raw = [ + "--base-url", + "https://rm.ldrprep.com", + "--project", + "customer-service", + "--cache-dir", + "/tmp/cache", + "--remote-updated-since", + "30d", + "--remote-status-id", + "*", + ] + norm = inject_default_sync_command(raw) + args = build_parser().parse_args(norm) + self.assertEqual(args.command, "sync") + self.assertEqual(args.remote_updated_since, "30d") + self.assertEqual(args.status_id, "*") + value = build_remote_updated_on_value( + raw=args.remote_updated_on, + since=args.remote_updated_since, + before=args.remote_updated_before, + now=datetime(2026, 7, 14, 12, 0, 0, tzinfo=timezone.utc), + ) + self.assertEqual(value, ">=2026-06-14T12:00:00Z") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_sync_redmine_contacts.py b/tests/test_sync_redmine_contacts.py new file mode 100644 index 0000000..c11d7af --- /dev/null +++ b/tests/test_sync_redmine_contacts.py @@ -0,0 +1,178 @@ +"""End-to-end tests for the Redmine -> CRM sync pipeline.""" + +from __future__ import annotations + +import json +import os +import tempfile +from pathlib import Path +from unittest import mock + +import pytest +import responses + +import contact_enrichment +import sync_redmine_contacts +from redmine_crm.crm_client import CrmClient +from redmine_crm.redmine_client import RedmineClient + +BASE_URL = "https://redmine.example.com" + + +@responses.activate +def test_pipeline_full_run_creates_contact(): + contacts = [ + { + "id": 1, + "first_name": "Grace", + "last_name": "Hopper", + "email": "grace@example.com", + "updated_at": "2026-07-04T12:00:00Z", + } + ] + responses.add( + responses.GET, + f"{BASE_URL}/contacts.json", + json={"contacts": contacts, "total_count": 1}, + status=200, + ) + + with tempfile.TemporaryDirectory() as tmpdir: + state_file = Path(tmpdir) / "state.json" + redmine = RedmineClient(BASE_URL, "secret", project_id=1) + crm = CrmClient() + crm._store.clear() + state = sync_redmine_contacts.FileSyncState(state_file) + config = contact_enrichment.EnrichmentConfig() + + result = sync_redmine_contacts.run_pipeline( + redmine=redmine, + crm=crm, + enrichment_config=config, + state=state, + batch_size=100, + dry_run=False, + ) + + assert result["metrics"]["fetched"] == 1 + assert result["metrics"]["enriched"] == 1 + assert result["metrics"]["created"] == 1 + assert result["metrics"]["errors"] == 0 + assert state.last_sync_timestamp is not None + + +@responses.activate +def test_dry_run_does_not_write_crm(): + contacts = [ + { + "id": 2, + "first_name": "Ada", + "last_name": "Lovelace", + "email": "ada@example.com", + "updated_at": "2026-07-04T12:00:00Z", + } + ] + responses.add( + responses.GET, + f"{BASE_URL}/contacts.json", + json={"contacts": contacts, "total_count": 1}, + status=200, + ) + + with tempfile.TemporaryDirectory() as tmpdir: + state_file = Path(tmpdir) / "state.json" + redmine = RedmineClient(BASE_URL, "secret", project_id=1) + crm = CrmClient() + crm._store.clear() + state = sync_redmine_contacts.FileSyncState(state_file) + + result = sync_redmine_contacts.run_pipeline( + redmine=redmine, + crm=crm, + state=state, + batch_size=100, + dry_run=True, + ) + + assert result["dry_run"] is True + assert result["metrics"]["created"] == 0 + assert not crm._store + + +@responses.activate +def test_pipeline_uses_last_sync_timestamp(): + contacts = [ + { + "id": 3, + "first_name": "Alan", + "last_name": "Turing", + "email": "alan@example.com", + "updated_at": "2026-07-05T12:00:00Z", + } + ] + responses.add( + responses.GET, + f"{BASE_URL}/contacts.json", + json={"contacts": contacts, "total_count": 1}, + status=200, + ) + + with tempfile.TemporaryDirectory() as tmpdir: + state_file = Path(tmpdir) / "state.json" + state_file.write_text(json.dumps({"last_sync_timestamp": "2026-07-04T00:00:00Z"}), encoding="utf-8") + redmine = RedmineClient(BASE_URL, "secret", project_id=1) + crm = CrmClient() + crm._store.clear() + state = sync_redmine_contacts.FileSyncState(state_file) + + result = sync_redmine_contacts.run_pipeline( + redmine=redmine, + crm=crm, + state=state, + batch_size=100, + dry_run=False, + ) + + assert result["metrics"]["fetched"] == 1 + request = responses.calls[0].request + assert "updated_on" in (request.url or "") + + +def test_cli_requires_api_key(): + env = {"REDMINE_API_KEY": ""} + with mock.patch.dict(os.environ, env, clear=True): + with pytest.raises(RuntimeError, match="REDMINE_API_KEY"): + sync_redmine_contacts._redmine_client_from_args( + sync_redmine_contacts.build_argument_parser().parse_args(["--base-url", BASE_URL, "--dry-run"]) + ) + + +def test_file_sync_state_persists_and_reads(): + with tempfile.TemporaryDirectory() as tmpdir: + state_file = Path(tmpdir) / "state.json" + state = sync_redmine_contacts.FileSyncState(state_file) + metrics = sync_redmine_contacts.SyncMetrics() + metrics.fetched = 5 + state.record_run("2026-07-05T08:00:00Z", metrics) + + new_state = sync_redmine_contacts.FileSyncState(state_file) + assert new_state.last_sync_timestamp == "2026-07-05T08:00:00Z" + assert new_state.last_metrics is not None + assert new_state.last_metrics["fetched"] == 5 + + +def test_metrics_aggregate_counts(): + metrics = sync_redmine_contacts.SyncMetrics() + metrics.fetched = 10 + metrics.enriched = 9 + metrics.skipped = 1 + metrics.created = 5 + metrics.updated = 3 + metrics.unchanged = 1 + metrics.errors = 0 + metrics.finished_at = sync_redmine_contacts._utc_now_iso() + + d = metrics.to_dict() + assert d["fetched"] == 10 + assert d["enriched"] == 9 + assert d["created"] + d["updated"] + d["unchanged"] == 9 diff --git a/tests/test_tui_dispatch_regression.py b/tests/test_tui_dispatch_regression.py new file mode 100644 index 0000000..2931fec --- /dev/null +++ b/tests/test_tui_dispatch_regression.py @@ -0,0 +1,133 @@ +"""Regression: `rtl tui` must reach its TUI launcher instead of falling through to exit 1.""" + +from __future__ import annotations + +import redmine_tickets + + +def test_tui_command_dispatches_to_tui_launcher(monkeypatch, tmp_path): + called = {} + + class DummyCache: + def __init__(self, cache_dir, project): + self.cache_dir = cache_dir + self.project = project + + def fake_tui_launcher(args, cache): + called["command"] = args.command + called["project"] = cache.project + return 73 + + monkeypatch.setattr(redmine_tickets, "TicketCache", DummyCache) + monkeypatch.setattr(redmine_tickets, "run_tui_cli", fake_tui_launcher) + + config = tmp_path / "live.json" + config.write_text( + '{"base-url":"https://rm.ldrprep.com","project":"customer-service",' + '"cache-dir":"/tmp/redmine-cache"}', + encoding="utf-8", + ) + + rc = redmine_tickets.main( + ["--config", str(config), "tui", "--open-only", "--max-tickets", "5"] + ) + + assert rc == 73 + assert called == {"command": "tui", "project": "customer-service"} + + +class DummyCache: + def __init__(self, cache_dir, project): + self.cache_dir = cache_dir + self.project = project + + +def test_ocr_rename_command_reaches_handler_not_silent_exit(monkeypatch, tmp_path): + """ocr-rename used to be nested under search after `return 0` and fell through.""" + called = {} + + def fake_load(cache, issue_id): + called["issue_id"] = issue_id + raise RuntimeError("ocr-rename handler reached") + + monkeypatch.setattr(redmine_tickets, "TicketCache", DummyCache) + monkeypatch.setattr(redmine_tickets, "load_cached_issue", fake_load) + + try: + rc = redmine_tickets.main( + [ + "--project", + "hr", + "--cache-dir", + str(tmp_path), + "ocr-rename", + "--issue-id", + "43476", + "--json", + ] + ) + except RuntimeError as exc: + assert "ocr-rename handler reached" in str(exc) + assert called == {"issue_id": 43476} + return + raise AssertionError(f"ocr-rename fell through without reaching handler (rc={rc})") + + +def test_attachments_list_reaches_handler_not_silent_exit(monkeypatch, tmp_path): + called = {} + + def fake_load(cache, issue_id): + called["issue_id"] = issue_id + raise RuntimeError("attachments handler reached") + + monkeypatch.setattr(redmine_tickets, "TicketCache", DummyCache) + monkeypatch.setattr(redmine_tickets, "load_cached_issue", fake_load) + + try: + rc = redmine_tickets.main( + [ + "--project", + "hr", + "--cache-dir", + str(tmp_path), + "attachments", + "list", + "--issue-id", + "43476", + ] + ) + except RuntimeError as exc: + assert "attachments handler reached" in str(exc) + assert called == {"issue_id": 43476} + return + raise AssertionError(f"attachments fell through without reaching handler (rc={rc})") + + +def test_view_command_reaches_handler_not_silent_exit(monkeypatch, tmp_path): + called = {} + + def fake_load(cache, issue_id): + called["issue_id"] = issue_id + raise RuntimeError("view handler reached") + + monkeypatch.setattr(redmine_tickets, "TicketCache", DummyCache) + monkeypatch.setattr(redmine_tickets, "load_cached_issue", fake_load) + + try: + rc = redmine_tickets.main( + [ + "--project", + "hr", + "--cache-dir", + str(tmp_path), + "view", + "--issue-id", + "43476", + "--print", + ] + ) + except RuntimeError as exc: + assert "view handler reached" in str(exc) + assert called == {"issue_id": 43476} + return + raise AssertionError(f"view fell through without reaching handler (rc={rc})") diff --git a/tests/test_tui_tty_guard.py b/tests/test_tui_tty_guard.py new file mode 100644 index 0000000..3fcddf1 --- /dev/null +++ b/tests/test_tui_tty_guard.py @@ -0,0 +1,28 @@ +"""Curses launch must fail clearly outside a real terminal.""" + +from __future__ import annotations + +from pathlib import Path + +import redmine_ticket_tui + + +def test_tui_refuses_non_tty_with_clear_message(monkeypatch, capsys): + class Args: + cache_dir = Path("/tmp/redmine-cache") + project = "customer-service" + base_url = "https://rm.ldrprep.com" + allow_prod_writes = False + apply = False + + class Cache: + cache_dir = Args.cache_dir + project = Args.project + + monkeypatch.setattr(redmine_ticket_tui.os, "isatty", lambda _fd: False) + + rc = redmine_ticket_tui.main_cli(Args(), Cache()) + + captured = capsys.readouterr() + assert rc == 1 + assert "requires an interactive terminal" in captured.err