Pipestream AI / Parsers

Spreadsheets · Rust · calamine

Workbooks that stream out cell by cell.

grpc-calamine reads Excel and OpenDocument workbooks with the Rust calamine crate and sends the cells back as they are parsed. You upload a workbook once and then run as many reads against it as you like. The bytes stay in memory the whole time and never touch the disk.

xls, xlsx, xlsm, xlsb, ods Rust and tonic Nothing written to disk Apache-2.0

Upload once, read many times.

The service is handle based. One upload returns a workbook id and its metadata, and every read after that works against the id.

A client streams the file up in chunks. The server parses it in memory, hands back the id, the format it detected, and the list of sheets and defined names. From there a caller can stream a sheet's values, stream its formulas, pull the VBA project, or fetch embedded pictures, and it can do several of those at once against the same workbook.

Reads do not block each other. Each one builds its own calamine reader over the same shared, reference counted bytes, so many clients can stream the same workbook in parallel with no lock on the read path. The cost of that choice is honest and stated in the code: each reader re-reads the workbook's shared structures, which is the price of keeping reads lock free.

See a sheet fill in as it parses.

The Node demo puts a browser in front of the service. Drop a workbook and the rows appear as the Rust server reads them, straight from gRPC through server-sent events into the page.

Live demo

The demo bridge streams the upload straight into the gRPC call and holds no copy of the file. The same repository ships command line clients in Node, Python, and Java against the identical contract.

Two parse paths, one wire contract.

How a sheet streams depends on what calamine can do with the format, but the events a client sees are the same either way.

For xlsx and xlsb, calamine exposes an incremental cell reader, so rows go out as the parser walks the sheet. For xls and ods, calamine only offers whole-range parsing, so the range is parsed and then streamed row by row. A caller cannot tell the two apart from the events. Both send one header describing the sheet and its dimensions, then one row at a time, densely filled from the first column.

Backpressure is real here. The parser pushes rows into a small bounded channel, so a client that reads slowly slows the parser down instead of making the server buffer the whole sheet.

  • RangeStarted Sent once, first. Carries the resolved sheet name, the range dimensions, and the total cell count so a client can size a grid before any row arrives.
  • WorksheetRow One per row, in order, with the absolute row index and its dense cell values.
  • StreamError An in-band, typed error for a recoverable problem, so one bad sheet does not kill the rest of the stream.

The protobuf model mirrors calamine exactly.

The contract was drawn one to one against calamine's own public types, so nothing is guessed and nothing is lost.

A cell value is a protobuf oneof with the same shape as calamine's own value type: integers, floats, strings, shared strings, booleans, dates, ISO date and duration text, typed error values, and an explicit empty. An empty cell is its own case, never a missing field, so a client never has to guess whether a blank was intended.

Dates carry the detail spreadsheets usually lose. Excel stores a date as a serial number counting from one of two epochs, and the choice of epoch is a property of the workbook rather than the cell. The service reads that epoch once when the workbook opens and stamps it onto every date it streams, so a client can convert a serial to a real date without guessing which calendar it came from. That value comes straight from calamine's own workbook level reader, so the build runs against the published crate with no fork.

More than values.

The same handle reaches the parts of a workbook a plain cell reader skips.

Formulas

Stream a sheet's formula strings with the same header-then-rows shape as values. A sheet with no formulas says so rather than returning an empty grid.

VBA projects

Stream the macro project as a header of references and module names, then one module at a time. The raw module bytes are delivered as calamine returns them, so decoding stays a client choice.

Embedded pictures

Stream the images embedded in a workbook, each with its anchor position and file type, one event per picture.

Metadata and defined names

Read the sheet list, sheet visibility and type, and the workbook's defined names from the open response or on demand later.

Bounded on purpose.

The limits are compile-time constants, chosen so a hostile or oversized upload fails cleanly.

Max workbook upload512 MiB, refused with a resource-exhausted status past that
Max gRPC frame32 MiB, on both send and receive
Stream channel64 events of backpressure between the parser and the client
RuntimeMulti-threaded tokio, with all calamine parsing on the blocking pool

Read it, run it, test it.

The tests spin up a real server on an ephemeral port and stream real workbook files, comparing every streamed cell against calamine's own output. The repository carries the protobuf contract, the server, and the Node, Python, and Java demos.