Package ai.pipeline.connector.intake.service

Class ChunkAssembler

java.lang.Object

ai.pipeline.connector.intake.service.ChunkAssembler

@ApplicationScoped
public class ChunkAssembler
extends Object

Service for assembling streaming chunks into complete documents.

Maintains in-memory state of partial chunks and assembles them when complete.

Constructor Summary

Constructors

Constructor	Description
ChunkAssembler()

Method Summary

Modifier and Type	Method	Description
void	cleanupStaleChunks()	Clean up stale chunk buffers that have not been updated recently.
byte[]	processChunk(ai.pipestream.connector.intake.StreamingChunk chunk)	Deprecated. Use StreamingChunkProcessor with header/raw/footer protocol instead

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ChunkAssembler

public ChunkAssembler()

Method Details

processChunk

@Deprecated
public byte[] processChunk(ai.pipestream.connector.intake.StreamingChunk chunk)

Deprecated.

Use StreamingChunkProcessor with header/raw/footer protocol instead

Process a streaming chunk belonging to a logical document.

This method is deprecated in favor of the new header/raw/footer protocol. The new protocol streams data directly to S3 without buffering.

Parameters:

chunk - The chunk to process

Returns:

the complete document bytes if this was the last chunk; otherwise null

Throws:

IllegalArgumentException - if chunk numbers are out of order for a given documentRef

cleanupStaleChunks

public void cleanupStaleChunks()

Clean up stale chunk buffers that have not been updated recently.

Iterates over in-memory chunk states and removes entries whose lastUpdate exceeds CLEANUP_TIMEOUT_MS. Intended to be run periodically by a scheduler. Concurrency: safe to invoke concurrently with processChunk; the underlying map is concurrent. Side effects: drops partial data; clients must resend if chunks are purged.