Making a mature NLP toolkit fast, correct, and ready for modern pipelines.

One arc, in three movements.

The work reads as a single arc rather than a pile of unrelated patches. The first movement made the toolkit trustworthy. The inference classes were made safe to share across threads, the deep-learning components stopped leaking native memory and producing NaN scores, and error messages started naming what actually went wrong instead of throwing an opaque exception.

The second movement gave OpenNLP a text foundation it did not have before: a dependency-free, offset-preserving Unicode normalization engine and a Unicode-conformant word tokenizer. The invariant that runs through it is that every normalization step is alignment-preserving, so any annotation made later can be mapped back to an exact span in the original text.

The third movement builds on that foundation: emoji handled as first-class text, full Unicode case folding, per-language normalization profiles, and the stack of new features below. Read end to end, it is the NLP layer a distributed semantic search engine needs, built inside the project that will maintain it.

The research branch map.

Work reaches Apache OpenNLP through its normal review, one pull request at a time. The map below shows the current state: what has already merged into the project's main branch, the pull requests open against it, and the research branches staged behind them. Merged work lands in apache main at the top and drops out of the research line once it is there.

flowchart LR
  main([apache main])
  merged["Merged into apache main, July 2026:
case folding, emoji normalization, UCD whitespace,
de-regex normalizers, hot-path performance, stemmer factory,
loading hardening, pattern precompile"]
  main --- merged

  main --> p1177["#1177 emoji annotations"]
  main --> p1182["#1182 document container (foundation)"]
  main --> d1154["#1154 gazetteer + geocoder"]
  main --> d1155["#1155 wordnet knowledge base"]
  d1155 --> d1167["#1167 lexical expansion"]
  main --> d1166["#1166 light stemmers"]
  main --> d1165["#1165 SentencePiece"]
  d1165 --> d1152["#1152 static embeddings"]

  main --> depp["depparse (neural tier)"]
  main --> fftag["ff-postagger"]
  main --> bilstm["bilstm-tagger"]
  main --> cjk["#1191 CJK lattice tokenization"]
  main --> huns["#1190 hunspell stemmer"]

  p1182 --> glos["glossary"]
  p1182 --> pii["pii"]
  p1182 --> coref["coref"]
  p1182 --> num["numeric"]
  p1182 --> tart["text-artifacts"]
  p1182 --> asset["embedded-assets"]
  asset --> noiz["noise scoring"]
  depp --> deppa["depparse-annotator"]
  deppa --> rel["relation"]
  d1152 --> emb["embedding-annotator"]
  num --> rvote["region-vote"]
  rvote --> geo["geocode-annotator"]
  geo --> hier["hierarchy-annotator"]

  classDef mergedC fill:#f3f8f4,stroke:#4a7c59,color:#1c1e23;
  classDef ready fill:#eef4ee,stroke:#4a7c59,color:#1c1e23,stroke-width:2px;
  classDef draft fill:#faf7ea,stroke:#b8860b,color:#1c1e23;
  classDef foundation fill:#eef1f4,stroke:#34495e,color:#1c1e23,stroke-width:3px;
  classDef staged fill:#ffffff,stroke:#c8c8c2,color:#43474f;
  class merged mergedC;
  class p1177,d1165 ready;
  class d1152,d1154,d1155,d1166,d1167,huns,cjk draft;
  class p1182 foundation;
  class depp,fftag,bilstm,glos,pii,coref,num,tart,asset,noiz,deppa,rel,geo,hier,rvote,emb staged;
                    

Green is merged into apache main. Blue is the document container that the annotators build on. Amber is an open draft. White is a research branch staged on the fork, tested at its tip, not yet proposed. The full, dated map lives in the research fork.

New features being proposed for 3.x.

Each of these is a distinct new capability. Some are ready for review, some are open drafts, and some are staged on the research fork and tested at their tips before being proposed. The status is marked on each. The measured numbers are the project's own figures from its manuals and tests.

Merged into Apache OpenNLP.

The work already in the project's main branch, grouped by what it set out to fix. The full history, with line counts and dates, is on the project's pull request list.

Thread safety across the toolkit

The major inference classes were made safe to share across threads, so one loaded model can serve concurrent requests instead of being duplicated per call. That covers the maxent tokenizer, sentence detector, tagger, and lemmatizer, the ONNX deep-learning components, and the stemmers, which also gained a factory and per-thread caching. Existing single-threaded use is unchanged; high-throughput pipelines can now reuse instances.

The deep-learning components, from demo to production

A run of fixes took the ONNX components from fragile to dependable. The sentence-vectors component had been sending an all-zero attention mask, so the encoder attended to nothing; that and inverted token-type ids were corrected to the standard BERT convention. Tensors that leaked native memory on every call now release deterministically. A softmax that overflowed to infinity and produced NaN scores was replaced with the numerically stable form. The name finder, which had only ever decoded person tags, was generalized to every label a model emits. A proper BERT tokenizer replaced the ad-hoc one, with Unicode-aware punctuation splitting that matches the reference.

A Unicode-correct, offset-preserving text foundation

The largest piece of merged work is a text-normalization engine and a Unicode-conformant word tokenizer, delivered as a review-split stack. The engine classifies characters from the Unicode Character Database with a cursor rather than regular expressions, and every normalization step keeps a bidirectional alignment so an offset in the cleaned text maps back to an exact span in the original. The tokenizer passes the official Unicode word-break conformance suite in full. On top of it sits a layered term model that projects a single token through an ordered stack of forms, from the original through normalization, case folding, accent folding, and on to stem and lemma, keeping every intermediate form and its source span. Full Unicode case folding was added as an offset-aware step, catching the expansions plain lower-casing misses. A separate pass audited every whitespace test in the codebase and either moved it onto the Unicode property or froze it with a documented reason. The four legacy regex normalizers were rewritten as single forward scans, byte-for-byte identical to the originals, each change guarded by characterization tests and randomized differential checks first.

Emoji, and a long-open bug

Emoji and emoticons became reversible, offset-aware normalization steps driven by a project-authored mapping table with per-row provenance. Separately, a sentence-detector bug open since 2011 was closed by moving the mapping from end-of-sentence positions to spans out of the detection loop, fixing a latent misalignment when whitespace-only candidates were dropped.

Build, test, and licensing hygiene

Some of the most useful fixes were unglamorous. An evaluation-tests module had been compiling its sources and silently running zero tests; two lines made it execute again. Stemmer tests that had been excluded from the build were turned back on and their stale expectations corrected. A missing license section for the bundled stopword lists was added to the binary distribution.

The current 3.x research.

This sits alongside the broader Pipestream AI research on distributed semantic search. The NLP features here are the annotation and tokenization layer that search pipeline draws on.