Literary Artifacts

Simply embedding source text is often not enough. Broad questions may require generated support material that is indexed for retrieval without being treated as primary source evidence.

This chapter is where the RAG pipeline starts to become a product instead of a generic document search demo. The raw retrieval system can find passages, but literary artifacts shape the system around the expectations of a book-club user. They give the application its domain behavior: this is what makes it a book-club chat instead of a travel-info chat, a legal-document chat, or a generic PDF chatbot.

The important design question is not only "What text did we index?" It is also "What will real users ask, and what supporting knowledge should exist so the system can answer well?" For a book-club assistant, users often ask about characters, themes, setting, motivations, symbolism, and discussion prompts. Those questions may require synthesis across the whole book, not just one nearby paragraph.

To improve this, the worker generates extra derived chunks:

• a book-club profile;
• a name/entity profile.

These are created by RAG.Core/Services/LiteraryArtifactGenerator.cs and the configured ILiteraryAnalysisProvider.

The artifact generator reads representative source chunks and asks the selected AI provider to create structured, searchable summaries for the domain. These summaries are not a replacement for source chunks. They are additional retrieval targets that make broad, user-centered questions easier to answer.

The generated artifacts are embedded and stored in Qdrant like normal chunks, but their chunkType identifies them:

literary_book_club_profile
literary_name_profile

This is an important RAG lesson: the vector database can store both source material and generated support material. The support material should be designed around expected user questions.

In a real project, this is where product analysis matters. A useful RAG system starts with the user's workflow, not only the database schema or model choice. If the product were a travel-info chat, the generated artifacts might focus on destinations, opening hours, transit options, weather, accessibility, and itinerary constraints. If it were a legal assistant, the artifacts might focus on parties, obligations, dates, clauses, risks, and definitions. The retrieval layer should reflect the job the user is trying to get done.

For this MVP, the expected domain is book-club discussion, so the generated profile focuses on:

• likely protagonists;
• major characters;
• setting;
• plot overview;
• character arcs;
• themes;
• motifs;
• discussion questions;
• evidence notes.

Generated Artifact Provenance

The code keeps generated artifacts honest with ChunkProvenance. Source chunks are not marked generated. The deterministic name profile records Provider = "RAGPipeline", Model = "deterministic-name-extractor", and PromptVersion = "deterministic-name-profile-v1". The LLM book-club profile records the configured AI provider, chat model, prompt version, generation timestamp, and the source chunk and page indexes used to build the profile.

public sealed record ChunkProvenance(
    bool IsGenerated,
    string? ArtifactKind,
    string? Provider,
    string? Model,
    string? PromptVersion,
    DateTimeOffset? GeneratedAtUtc,
    IReadOnlyList<int>? SourceChunkIndexes,
    IReadOnlyList<int>? SourcePageNumbers);

That provenance is the difference between using generated summaries responsibly and pretending they are primary evidence. Generated artifacts improve retrieval, but citations and UI labels must still tell readers when a chunk came from generated support material.