feat: comma substitution, paired dash detection, conjunction handling #1

Merged

jez merged 1 commit from feat/comma-substitution-paired-dash into main

2026-08-25 13:04:22 +00:00

BarnacleBoy commented

2026-08-25 12:56:53 +00:00

Owner

Summary

Three changes to the em dash substitution engine:

1. Comma substitution path

New _WEAK_STARTERS set: that, which, who, whom, whose, where, when → , instead of : or ; .
New _COORDINATING_STARTERS set: but, yet, so, nor → , instead of ; (semicolon before a conjunction was redundant).

2. Paired em dash detection

Detects — text — patterns and substitutes appropriately:

Non-restrictive clauses (starting with which/who/that/etc.) → , commas,
Removable asides and appositives → (parentheses)

3. Bug fix: between-text extraction

Fixed a bug where _is_paired_dash used dash_positions[start] + 1 which landed inside the em dash regex match. Now uses re.Match.end() and .start() for accurate position tracking.

What this fixes

The original MFT response that motivated this change:

"psychological systems — "foundations" — that are present" → "psychological systems (\"foundations\") that are present" (was : ;)
"moral disagreements — especially between liberals and conservatives — are" → "moral disagreements (especially between liberals and conservatives) are" (was : ;)
"it was hard — but we did it" → "it was hard, but we did it" (was ; but)

Tests

56 tests across 3 suites, all passing
New test_new_features.py with 22 tests covering all new behaviors
Existing tests updated for corrected expectations (non-restrictive clauses now get commas, not parentheses)

## Summary Three changes to the em dash substitution engine: ### 1. Comma substitution path New `_WEAK_STARTERS` set: `that`, `which`, `who`, `whom`, `whose`, `where`, `when` → `, ` instead of `: ` or `; `. New `_COORDINATING_STARTERS` set: `but`, `yet`, `so`, `nor` → `, ` instead of `; ` (semicolon before a conjunction was redundant). ### 2. Paired em dash detection Detects `— text —` patterns and substitutes appropriately: - Non-restrictive clauses (starting with `which`/`who`/`that`/etc.) → `, commas,` - Removable asides and appositives → `(parentheses)` ### 3. Bug fix: between-text extraction Fixed a bug where `_is_paired_dash` used `dash_positions[start] + 1` which landed inside the em dash regex match. Now uses `re.Match.end()` and `.start()` for accurate position tracking. ### What this fixes The original MFT response that motivated this change: - `"psychological systems — "foundations" — that are present"` → `"psychological systems (\"foundations\") that are present"` (was `: ;`) - `"moral disagreements — especially between liberals and conservatives — are"` → `"moral disagreements (especially between liberals and conservatives) are"` (was `: ;`) - `"it was hard — but we did it"` → `"it was hard, but we did it"` (was `; but`) ### Tests - 56 tests across 3 suites, all passing - New `test_new_features.py` with 22 tests covering all new behaviors - Existing tests updated for corrected expectations (non-restrictive clauses now get commas, not parentheses)

BarnacleBoy added 1 commit

2026-08-25 12:56:53 +00:00

feat: add comma substitution, paired dash detection, and conjunction handling 9514ef0500

- Paired em dashes: non-restrictive clauses (which/who/that) → commas,
  removable asides and appositives → parentheses
- New _WEAK_STARTERS set: that, which, who, whom, whose, where, when → comma
- New _COORDINATING_STARTERS set: but, yet, so, nor → comma
- Paired dash detection uses re.Match objects for accurate between-text extraction
- Removed unused _is_short_appositive helper
- 56 tests across 3 suites all passing
- README updated with new examples