Open-source software that assists with human understanding.
Pipestream AI is a small collaboration of engineers working on distributed semantic search, document understanding, and the open infrastructure that connects them.
We build in the open and contribute upstream. Everything we release is licensed under Apache 2.0 or MIT. Where we can make an existing project faster, safer, or more capable, we send the work back to it rather than fork away.
Public packages
Everything below is public and open source — libraries you can depend on, tools you can run, and the research engine — under the ai-pipestream organization.
Distributed semantic search with collaborative HNSW KNN across shards.
Quarkus Mutiny gRPC stubs generated with Buf, cross-platform.
JVM-only, self-contained protoc codegen for Quarkus.
Pure-Java Protobuf toolkit that runs protoc as WebAssembly.
A Java API for Docling, including a gRPC service client.
All-in-one shaded Tika 4 jar for quick Java integration.
Runnable multi-language examples for the Docling gRPC interface.
A gRPC transport definition for the Model Context Protocol.
Draft RFC for a QUIC-based pipeline transport.
distributed-search
A gRPC distributed semantic search engine with collaborative HNSW nearest-neighbour search across shards. It is the reference implementation for the search research below, and the reason the other search work exists.
Vector search scales by sharding an index across nodes, but the standard distributed contract makes every shard run its search to full depth while a coordinator merges and discards most of the results. distributed-search exists to test a better contract: shards that collaborate during the search, sharing a running bound so each can stop chasing candidates the rest of the cluster has already beaten. The core mechanism is being upstreamed to Apache Lucene as a sandbox collector.
More projects
Other open-source repositories we build and maintain.
quarkus-buf-grpc-generator
Generates Quarkus Mutiny gRPC stubs using Buf, so a Quarkus build can consume proto workspaces without a native protoc toolchain. Cross-platform. → repo
tika4-shaded
An all-in-one shaded build of Apache Tika 4 — one jar you can drop into a Java application for text and metadata extraction without wrestling dependency conflicts. → repo
docling-grpc-examples
Runnable examples, in several languages, for the gRPC document-conversion interface being proposed to the Docling project. → repo
pipestream-quic-protocol-rfc
A draft RFC exploring a QUIC-based transport for the pipeline. → repo
Upstream contributions
Functional work merged into or proposed to established open-source projects. Each links to the pull request so the claim can be read against the code.
-
OpenNLPApache · merged
An ongoing effort to make OpenNLP's core faster and safer for concurrent use: making tokenization and normalization hot paths run in a fraction of their previous time, moving the stemmer and other stateful components toward thread-safety, and correcting Unicode handling against the Unicode Character Database. Alongside the performance work we've been adding capability — a lexical knowledge base, a gazetteer and geocoder for place-name understanding, and a static embedding engine.
- mergedThread-safe stemmers,
StemmerFactory, per-thread stem caching - mergedNon-breaking normalization & tokenization hot-path speedups
- mergedLexical knowledge-base seam — WN-LMF/WNDB readers, Morphy lemmatizer
- mergedGazetteer & geocoder API with a bundled Natural Earth implementation
- mergedStatic embedding engine for distilled embedding tables
- …and more — full history on GitHub
- mergedThread-safe stemmers,
-
TikaApache · merged + open
A
MarkdownParserfortika-parser-text-module. Before it,.mdfiles were detected astext/markdownbut no parser claimed them, so they fell through to plain-text extraction and their headings, tables, and code fences collapsed into flat text. This walks the Markdown AST into structured, lossless XHTML — and upstream has since built follow-on work on it. Alongside it, a strongly-typedDocumentparse contract fortika-grpc, so services can consume Tika output as a schema rather than by parsing HTML. -
LuceneApache · open
A shared-floor kNN collector in the sandbox module — the building block for coordinating an HNSW nearest-neighbour search across independent shards without a central rank pass. Read how the mechanism works →
-
QuarkusQuarkus · merged
Fixes and improvements to the gRPC stack — including a correction to the Stork load-balancer subchannel lifecycle — with further bug fixes and documentation improvements being catalogued.
-
Jandex Gradle Pluginkordamp · merged
Made the plugin compatible with Gradle's configuration cache — the build-caching feature it previously broke on. The task was refactored off non-serializable Gradle types (
Project,Configuration,SourceSet) onto the lazyProperty/ProviderAPI, with correct input/output declarations for up-to-date checking and functional tests covering the cached path. -
gRPC ZeroQuarkiverse · merged
Sustained work on the JVM-only, self-contained protoc codegen for Quarkus: an input-hash cache that skips regeneration when the proto inputs haven't changed, descriptor-FQN validation that catches duplicate proto types, generator and post-processor SPIs for extensibility, clearer codegen failure diagnostics, and the project's documentation.
-
Apicurio RegistryApicurio · merged
Hardening the registry's protobuf handling. Binary descriptor uploads whose identifiers violate the protobuf grammar were being stored intact and carried into downstream consumers; we now validate identifiers and reject conflicting type definitions at the upload boundary. The two follow-on fixes keep consumers alive when schema resolution fails, by falling back to direct protobuf parsing rather than stalling the consumer group.
-
protobuf4jroastedroot · merged
Memory-safety and diagnostics work on this JVM Protobuf toolkit, which runs
protoccompiled to WebAssembly: wrapping nativemalloc/freein anAutoCloseablebuffer so WASM memory can't leak, and surfacing native-pluginstderrin exceptions instead of swallowing it. -
Micronaut gRPCMicronaut · merged
Created the
protobuf-json-supportmodule — a JSON ⇄ Protocol Buffers bridge that lets Micronaut gRPC services speak JSON over a standard proxy, so a gRPC API can be consumed as REST/JSON without hand-written translation. It now ships as a first-class module on the framework's 5.0.x line. -
Docling & MarkItDownDocling / Microsoft · merged + open
Work toward a common gRPC document-conversion interface across the leading open document-parsing tools. In
docling-javawe implemented thedocling-serve-grpcservice client and brought the JavaDoclingDocumentmodel to a complete, wire-accurate match. In flight upstream: a gRPC server with PDF/PPTX streaming for MarkItDown, and native gRPC conversion plus shared protobuf definitions for Docling.
Dedicated pages for OpenNLP, document parsing, gRPC, Quarkus, the Jandex Gradle plugin, and Apicurio are in progress.
Pipestream AI Public Research
Open research on distributed vector search. We publish the work so it can be reviewed, reproduced, and improved on.
Collaborative distributed HNSW search
HNSW graphs give excellent approximate nearest-neighbour recall on a single node, but a naïve distributed search queries every shard to full depth and merges afterward — paying the full cost on every shard. We are investigating a collaborative approach where shards share a running floor value, letting each shard prune candidates that cannot enter the global top-k. The Lucene shared-floor kNN collector is the first upstream piece of this work.
- distributed-search — reference engine for collaborative KNN across shards.
- Shared-floor kNN collection — proposed to Apache Lucene (sandbox module).
- Distributed Lucene index — a shard-native index format, in progress.
Background
The mission — software that assists with human understanding — has a long root.
Pipestream AI's founder began in open source on Linux desktop accessibility: the AccessX keyboard and pointer features — StickyKeys, SlowKeys, BounceKeys, and MouseKeys — that let someone operate a computer without a conventional keyboard or mouse. That work was done around 1999 through the rehabilitation-services program at the University of Illinois at Urbana‑Champaign, with Ralph Johnson. Making information usable by people has been the point ever since.
The through-line since then has been search: an NLP search engine in 1999, and later production work on relevance and live-updating indexes — rebuilding Etsy's search in 2010 for live updates and modernized relevance, and the search at Adorama. distributed-search and the collaborative-HNSW research are the current chapter of the same problem.