Open-source infrastructure for semantic search and document parsing.

Public packages

Everything below is public and open source — libraries you can depend on, tools you can run, and the research engine — under the ai-pipestream organization.

distributed-search

A gRPC distributed semantic search engine with collaborative HNSW nearest-neighbour search across shards. It is the reference implementation for the search research below, and the reason the other search work exists.

Vector search scales by sharding an index across nodes, but the standard distributed contract makes every shard run its search to full depth while a coordinator merges and discards most of the results. distributed-search exists to test a better contract: shards that collaborate during the search, sharing a running bound so each can stop chasing candidates the rest of the cluster has already beaten. The core mechanism is being upstreamed to Apache Lucene as a sandbox collector.

gRPC document parsers

Four services that turn documents into structured protobuf and stream the results back as they parse, without writing anything to disk.

A real pipeline runs into every format at once: scanned PDFs and images, Word and PowerPoint and their OpenDocument cousins, and spreadsheets in every Excel flavour. Rather than force all of them through one engine, each service wraps the library that already handles its format well and puts it behind the same protobuf contract. gRParse runs OCR and layout on the GPU and also coordinates the others, grpc-libreoffice renders and reads office documents through LibreOffice, grPOIc pulls content and metadata with Apache POI, and grpc-calamine streams spreadsheets cell by cell in Rust.

ProtoMolt

A modular Java toolkit for Protocol Buffers at runtime, and a schema registry whose storage is a plain Git repository behind the Confluent protocol. It is the schema layer underneath Pipestream, maintained as its own project.

Every code path operates on descriptors rather than generated classes: gather .proto sources from Git, Maven, jars, or other registries; validate messages against rules carried on the schema (the protovalidate dialect passes the full conformance suite, 2,872 of 2,872 cases, re-scored in CI on every push); diff schemas with a typed compatibility engine; and register versions through compatibility-gated writes, one Git commit each. gRPC consumers fetch compiled binary descriptor sets directly.

An MCP server makes the whole toolkit agent-operable: an AI agent can reflect a live gRPC server, invoke its methods with no generated stubs, and generate native clients in eight languages. The OpenVINO tutorial walks the full path against a real inference server.

The same descriptor model reaches past schemas into data and integration work: projecting messages into Lucene, OpenSearch, Solr, and Microsoft Graph search; landing them in Apache Iceberg tables as committed snapshots written by its own Hadoop-free Parquet emitter; reading and writing OneDrive and SharePoint files and metadata over the Microsoft Graph API; masking fields by declared sensitivity; and rendering schemas as knowledge bundles. Twenty-three verbs run identically over gRPC, REST, and MCP.

More projects

Other open-source repositories we build and maintain.

quarkus-buf-grpc-generator

Generates Quarkus Mutiny gRPC stubs using Buf, so a Quarkus build can consume proto workspaces without a native protoc toolchain. Cross-platform. → repo

tika4-shaded

An all-in-one shaded build of Apache Tika 4 — one jar you can drop into a Java application for text and metadata extraction without wrestling dependency conflicts. → repo

docling-grpc-examples

Runnable examples, in several languages, for the gRPC document-conversion interface being proposed to the Docling project. → repo

pipestream-quic-protocol-rfc

A draft RFC exploring a QUIC-based transport for the pipeline. → repo

Upstream contributions

Functional work merged into or proposed to established open-source projects. Each links to the pull request so the claim can be read against the code.

• OpenNLPApache · merged

  An ongoing effort to make OpenNLP's core faster and safer for concurrent use: making tokenization and normalization hot paths run in a fraction of their previous time, moving the stemmer and other stateful components toward thread-safety, and correcting Unicode handling against the Unicode Character Database. Alongside the performance work we've been adding capability — a lexical knowledge base, a gazetteer and geocoder for place-name understanding, and a static embedding engine.

  • mergedThread-safe stemmers, StemmerFactory, per-thread stem caching
  • mergedNon-breaking normalization & tokenization hot-path speedups
  • openLexical knowledge-base seam: WN-LMF and WNDB readers, Morphy lemmatizer
  • openGazetteer and geocoder API with a bundled Natural Earth implementation
  • openStatic embedding engine for distilled embedding tables
  • Read the full OpenNLP write-up, merged work and new features →
• TikaApache · merged + open

  A MarkdownParser for tika-parser-text-module. Before it, .md files were detected as text/markdown but no parser claimed them, so they fell through to plain-text extraction and their headings, tables, and code fences collapsed into flat text. This walks the Markdown AST into structured, lossless XHTML — and upstream has since built follow-on work on it. Alongside it, a strongly-typed Document parse contract for tika-grpc, so services can consume Tika output as a schema rather than by parsing HTML.

  • mergedMarkdown parser with structured, lossless XHTML output
  • openTyped Document parse contract for tika-grpc
• LuceneApache · open

  A shared-floor kNN collector in the sandbox module — the building block for coordinating an HNSW nearest-neighbour search across independent shards without a central rank pass. Read how the mechanism works →

  • openAdd shared-floor kNN collection to the sandbox module
• QuarkusQuarkus · merged

  Fixes and improvements to the gRPC stack — including a correction to the Stork load-balancer subchannel lifecycle — with further bug fixes and documentation improvements being catalogued.

  • mergedFix gRPC Stork load-balancer subchannel lifecycle
• Jandex Gradle Pluginkordamp · merged

  Made the plugin compatible with Gradle's configuration cache — the build-caching feature it previously broke on. The task was refactored off non-serializable Gradle types (Project, Configuration, SourceSet) onto the lazy Property/Provider API, with correct input/output declarations for up-to-date checking and functional tests covering the cached path.

  • mergedMake the Jandex Gradle Plugin compatible with the Gradle configuration cache
• gRPC ZeroQuarkiverse · merged

  Sustained work on the JVM-only, self-contained protoc codegen for Quarkus: an input-hash cache that skips regeneration when the proto inputs haven't changed, descriptor-FQN validation that catches duplicate proto types, generator and post-processor SPIs for extensibility, clearer codegen failure diagnostics, and the project's documentation.

  • mergedInput-hash cache to skip codegen when proto inputs are unchanged
  • mergedDetect duplicate proto types by descriptor FQN
  • mergedAdd GrpcZeroGenerator and GrpcZeroPostProcessor SPIs
  • mergedImprove proto codegen failure diagnostics
• Apicurio RegistryApicurio · merged

  Hardening the registry's protobuf handling. Binary descriptor uploads whose identifiers violate the protobuf grammar were being stored intact and carried into downstream consumers; we now validate identifiers and reject conflicting type definitions at the upload boundary. The two follow-on fixes keep consumers alive when schema resolution fails, by falling back to direct protobuf parsing rather than stalling the consumer group.

  • mergedValidate identifiers & reject conflicting schemas at upload
  • mergedOpt-in fallback to direct protobuf parsing when schema resolution fails
  • mergedCatch SchemaException in ProtobufSchemaParser.parseSchema()
• protobuf4jroastedroot · merged

  Memory-safety and diagnostics work on this JVM Protobuf toolkit, which runs protoc compiled to WebAssembly: wrapping native malloc/free in an AutoCloseable buffer so WASM memory can't leak, and surfacing native-plugin stderr in exceptions instead of swallowing it.

  • mergedWrap WASM malloc/free in an AutoCloseable buffer
  • mergedInclude native-plugin stderr in the thrown exception
• Micronaut gRPCMicronaut · merged

  Created the protobuf-json-support module — a JSON ⇄ Protocol Buffers bridge that lets Micronaut gRPC services speak JSON over a standard proxy, so a gRPC API can be consumed as REST/JSON without hand-written translation. It now ships as a first-class module on the framework's 5.0.x line.

  • mergedgRPC protobuf-JSON support
  • mergedJSON Protocol Buffer proxy support
  • mergedBring protobuf-JSON support to 4.10.x
• Docling & MarkItDownDocling / Microsoft · merged + open

  Work toward a common gRPC document-conversion interface across the leading open document-parsing tools. In docling-java we implemented the docling-serve-grpc service client and brought the Java DoclingDocument model to a complete, wire-accurate match. In flight upstream: a gRPC server with PDF/PPTX streaming for MarkItDown, and native gRPC conversion plus shared protobuf definitions for Docling.

  • mergeddocling-java: implement the docling-serve-grpc service client
  • mergeddocling-java: DoclingDocument model with correct wire formats
  • openMarkItDown: gRPC server + PDF/PPTX streaming
  • openDocling: gRPC protobuf definitions + conversion utility
  • openDocling-serve: native gRPC converter

Dedicated write-ups: the OpenNLP work and the gRPC document parsers. Pages for Quarkus, the Jandex Gradle plugin, and Apicurio are in progress.

Pipestream AI Public Research

Open research on distributed vector search. We publish the work so it can be reviewed, reproduced, and improved on.

Background

The through-line is search. An NLP search engine in 1999, then production work on relevance and live-updating indexes, including rebuilding Etsy's search in 2010 for live updates and modernized relevance, and the search at Adorama. The distributed-search engine and the collaborative-HNSW research are the current chapter of the same problem.

Further back, the same hands wrote the AccessX keyboard and pointer features for Linux in 1999 (StickyKeys, SlowKeys, BounceKeys, and MouseKeys), the first time those accessibility features shipped on the platform, where they went out across the Linux desktops. It was freeware, so the attribution sits in the dark corners of the internet, but AccessX still gets referenced.