Package ai.pipeline.connector.intake.service

Class DocumentProcessor

java.lang.Object

ai.pipeline.connector.intake.service.DocumentProcessor

@ApplicationScoped
public class DocumentProcessor
extends Object

Service for processing individual documents.

Handles validation, rate limiting, chunk assembly, metadata enrichment, and storage.

Constructor Summary

Constructors

Constructor	Description
DocumentProcessor()

Method Summary

Modifier and Type	Method	Description
void	cleanup()	Cleanup method to properly shut down the worker thread pool.
io.smallrye.mutiny.Uni<ai.pipestream.connector.intake.DocumentResponse>	processDocument(CrawlSession session, ai.pipestream.connector.intake.ConnectorConfig config, ai.pipestream.connector.intake.DocumentData document)	Process a document.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DocumentProcessor

public DocumentProcessor()

Method Details

processDocument

public io.smallrye.mutiny.Uni<ai.pipestream.connector.intake.DocumentResponse> processDocument(CrawlSession session,
 ai.pipestream.connector.intake.ConnectorConfig config,
 ai.pipestream.connector.intake.DocumentData document)

Process a document.

Validates metadata, enforces rate limits, assembles chunked content when needed, enriches metadata, and persists the document via repository-service. When the session is configured to track documents, a record is written to the local database linking source IDs to stored document IDs. Reactive semantics:

• Returns a Uni<DocumentResponse> that completes asynchronously; persistence and remote gRPC calls are executed in a non-blocking fashion.
• Errors are recovered into a success=false response with an error message.

Side effects:

• May write to repository-service (remote) and the local DB (tracking table).
• Uses an in-memory rate limiter per connector.

Parameters:

session - The crawl session

config - The connector configuration

document - The document data

Returns:

a Uni emitting the DocumentResponse indicating success and document ID if applicable

cleanup

@PreDestroy
public void cleanup()

Cleanup method to properly shut down the worker thread pool.