Most RAG demos only show the happy path. in production, the failures are timeouts, retries that loop, tools with too much access, and no record of what the agent actually did

packt publishing is running a hands on workshop on building the harness that handles this, using an AI data analyst as the example. what gets covered:

• timeouts, retries, re planning and execution budgets
• structured traces of every step, so you can debug a bad answer
• permissioned tool access
• a fault injected test suite that breaks things on purpose

led by Bruno Gonçalves, 4 hours online, Saturday Oct 31, 9:30 AM to 1:30 PM ET. you run it on a laptop with a mock model, no GPU or API key needed

Full details and the agenda are here

happy to answer any questions about what gets covered

I’m building a multi-agent research assistant as part of my thesis, and the system is designed as a nine-agent pipeline where each agent handles a specific stage of the research process.

Agent 1 – Query Processing Agent: Takes the user’s research query and processes it into a structured representation that the downstream agents can use. Its job is to make sure the research request is clear and organized before the workflow begins.

Agent 2 – Literature Retrieval Agent: Searches for relevant research papers based on the processed query. It retrieves candidate papers and manages the retrieval results, helping establish the literature foundation for the rest of the pipeline.

Agent 3 – PDF Analysis Agent: Works with the retrieved research papers and extracts structured information from their PDF contents. This turns unstructured research documents into usable analytical data for later agents.

Agent 4 – Evidence Agent: Extracts scientific evidence from the analyzed papers. It organizes evidence by paper and tracks the supporting information, helping ensure that later conclusions can be traced back to the research sources.

Agent 5 – Method and Limitation Agent: Analyzes the methods used in the papers and identifies their limitations. The goal is to understand not only what previous studies found, but also how they conducted their research and where their approaches may fall short.

Agent 6 – Cross-Paper Comparison Agent: Compares information across multiple papers, bringing together evidence and methodological profiles to identify similarities, differences, and patterns across the literature.

Agent 7 – Research Synthesis Agent: Synthesizes the findings from the preceding analysis into a broader understanding of the research landscape. It helps connect individual paper-level insights into a coherent picture of the existing work.

Agent 8 – Research Gap Discovery Agent: Identifies potential research gaps from the synthesized observations. It validates candidate gaps against their supporting observations and provenance, checks upstream completeness, and preserves usable candidates even when the result is partial rather than treating incomplete inputs as a complete analysis.

Agent 9 – Gap Validation Agent: Evaluates whether the proposed research gaps are sufficiently supported. It checks candidate references against the observations and their source lineage, verifies relevant artifacts and source chunks, invokes a reviewer, and can use bounded adversarial retrieval to challenge the candidates. It then produces structured verdicts and an overall workflow validation status.

One of the most challenging parts of this project is not simply making nine agents work independently. It is making them work together reliably. Each stage depends on the quality and completeness of the previous stages, so I’m also building a shared memory architecture with state management, persistent storage, provenance tracking, and explicit status handling.

The bigger goal is to build a research assistant that does more than retrieve papers or generate summaries. I want it to follow a traceable process from a research question to evidence-based analysis and, ultimately, research gaps that can be critically evaluated.

Still a work in progress, but building and debugging this system has been a great learning experience in agentic AI, orchestration, and reliable multi-agent system design.

The biggest challenge I’m facing right now is LLM reliability. A multi-agent research workflow is a thinking-intensive task, and the quality of the underlying models matters a lot. But I’m running into provider-specific issues with Hugging Face, where model/provider incompatibilities sometimes block agents. What makes it particularly frustrating is that the same setup can work once and then fail on another run. Sometimes one agent works then fails in another run that same agent.

For those of you who have built multi-agent systems with Hugging Face Inference Providers: Are there provider-specific rate limits, usage quotas, or availability restrictions even on paid plans? Is this more likely to be a model–provider compatibility issue, temporary provider availability, or something related to inference limits?

I’d also love to hear how you handle this in production. Do you pin a specific provider, use automatic provider selection with fallbacks, or switch to dedicated inference endpoints? I’m trying to make the pipeline reliable without constantly fighting infrastructure issues instead of focusing on the actual research problem.

Hey everyone,

I've been working on an experimental multimodal RAG system for research PDFs, mainly because I wanted to explore a problem that most basic RAG tutorials tend to ignore: what happens when the answer is inside a chart, table, or diagram rather than plain text?

The difficult part was extracting evidence from complex documents while preserving enough information to verify where it actually came from.

What I built:

• Document ingestion: Docling for layout-aware parsing, selective OCR and local vision models for visual evidence processing.
• Multimodal evidence: Charts, tables and diagrams retain references to their original PDF, page and source region. Uncertain extractions require review before becoming searchable evidence.
• Hybrid retrieval: PostgreSQL + pgvector, full-text search, reciprocal rank fusion and MiniLM reranking.
• Local inference: Ollama with Qwen3-8B for generation and Qwen3-VL-4B for visual processing.
• Citations: Source-aware references with validation against the selected evidence.
• Operations: Access controls, durable jobs, SSE event replay, observability and reproducible evaluations.

Some benchmark results:

• 50/51 correct documents retrieved in the top 5 on an exact document-heading lookup benchmark.
• nDCG@10: 0.9587 on that same evaluation.
• 29/30 known-positive evidence targets reached the final six contexts.

Important distinction: these measure specific retrieval tasks, not overall answer accuracy. The evaluation labels are limited, and I haven't established broad multimodal correctness.

What still needs work:

Complex visual regions still require review, and end-to-end answer quality needs broader evaluation. I wouldn't call this production-ready.

The default workspace serves 27 research papers, with a separate 51-paper candidate used for broader retrieval experiments.

I've published the code, architecture, benchmark artifacts and known limitations here:

https://github.com/miguelCyclone/multimodal-rag-showcase

One thing I'm particularly interested in is how others handle visual evidence in RAG. Do you extract charts and tables automatically and trust the output, or use some kind of confidence/review process before indexing?

A few weeks ago I posted an eval here on whether a retrieved record stops an LLM from blending neighbouring fashion styles. This time I ran the same aesthetics against three public assistants (signed out, one fresh chat each), to see whether the problem shows up in production too. It does.

• Daydream (fashion shopping agent): asked for Quiet Luxury, it returned 105 products and none from Loro Piana, Brunello Cucinelli, The Row or Zegna. Asked for Brunello Cucinelli by name, it returned 11 products. The catalog has the right items; the step from aesthetic to brand before retrieval is what fails. City Boy came back as Ivy/preppy (J.Crew, Ralph Lauren, Brooks Brothers), and “Opium” was refused outright, probably a drug filter.
• Encore (AI secondhand search): Goth Ninja came back as Techwear (hoodie, cargos, balaclava). Amekaji and Ura-Hara came back with no brands at all, which matters because brands are the search keys on resale sites.
• ChatGPT (signed out, same prompt as the eval): it made the same two errors the eval measured, Red Wing for Amekaji and Massimo Dutti for Quiet Luxury. It built Goth Ninja on Techwear brands (ACRONYM, Y-3). Ura-Hara was perfect, with all 5 core brands.

Disclosure: I made free long-form breakdowns on YouTube about AI to help people get the theoretical vision of the technology. The goal is helping people understand better the technology.

If you are building LLM features in 2026, the gap I see is people jumping from "hello world RAG" straight to agents without fixing document extraction and retrieval quality first. Most production pain is boring infrastructure, not prompt wording.

I recorded two free walkthroughs that mirror how I actually teach this:

1. Multimodal RAG, bottlenecks in order

Parsing → chunking → embedding → retrieval → reranking → generation. Where hello-world tutorials silently fail (PDF tables, scanned pages, multilingual, multi-vector).

Link for the video in the first comment!

1. Agents in a real product (not a demo script)

Mental model: LLM + tools + memory + stop conditions. Structured output/tool calls, evals before "multi-agent," failure modes I keep seeing in prod.

Link for the video in the first comment as well!

Before both: Andrej Karpathy's intro material still beats most paid courses for "how models fail." Watch that first if you are new.

What I would build first (exercise): one agent, three tools (search, calculator, read a local file). No crew, no swarm. Measure pass/fail on 10 fixed scenarios before adding RAG.

If you watch either video and get stuck on a step (parsing vs rerank vs tool schema), drop the timestamp and your doc type (PDF, PPTX, API JSON). I will reply with specifics.

Hope this helps people getting into the domain !

Ps, i am working on databricks platform for data engineering needs and querying unstructured docs alongside live op data is crushing our latency (Text-to-SQL adds 800ms+) , and separate vector stores is causeing constant sync drifts.. Can unified serverless Postgres +pgvector via Lakebase hold up under load, or is Databricks Genie actually viable for low-latency end-user RAG?

LinkedIn posts seem useful as a source for organizational RAG, especially when announcements and explanations never make it into the main documentation but some posts only make sense alongside information stored somewhere else

Consider a professional association publishing these updates:

“Registration closes friday”

“We’ve updated the certification requirements”

“This expands on the position we outlined last month”

Retrieving the right post wouldnt necessarily give an assistant enough information to answer, the first needs a publication date and an identifiable event, the second probably needs the linked requirements page, the third depends on finding another post, theres a concrete implementation worth discussing here: customgpt.ai linkedIn connector lets an agent use public profile or company page posts as sources, with citations back to the original posts, this is their explanation of how the posts are added and kept updated https://customgpt.ai/introducing-linkedin-connector/

That raises an ingestion question for any system using this kind of source: when should a post be treated as a complete retrieval unit and when should it be treated as a pointer to more evidence?

My starting approach would be to preserve the author, publication date and original url with each post then keep explicit relationships to any linked documents, at answer time I’d want the system to distinguish between a post announcing a change and a document explaining that change

For evaluation I’d include questions where retrieving the announcement is only the first step “did the requirements change?” might be answerable from the post “am I still eligible?” would need the actual requirements, a useful failure case would be an assistant answering both from the announcement alone. The tradeoff is how far to follow those dependencies before a small collection of posts becomes a much larger crawl, for people using short social posts in rag: do you resolve linked context during ingestion, retrieve it on demand or leave the post searchable and have the assistant acknowledge when it lacks the underlying details??

Traditional RAG has a fatal architectural flaw that nobody likes to admit: vector retrieval always returns something, even when the answer does not exist. If you ask a vector database about a person or product that was never mentioned in your corpus, cosine distance simply grabs the least distant noise chunks, feeds them to the generator, and guarantees a hallucination. Today, after six months of development, I released Hillock v1.0.0 to replace probabilistic RAG with a deterministic neuro-symbolic engine.

Instead of chopping documents into arbitrary paragraphs and storing dense float32 vectors, Hillock decouples memory from generation. It extracts clean fact triples into a structured knowledge graph and encodes concepts into 10,000-dimensional hypervectors. When a query comes in, our late-interaction MaxSim gate tests whether the query tokens and graph relations actually align above a strict mathematical threshold. If the fact is missing or unsupported, the gate slams shut before the LLM can hallucinate.

On our benchmark of answerable versus unanswerable test queries, standard RAG frequently leaks hallucinations on negative controls, while Hillock maintained a 0% hallucination leak rate. It also avoids context window bloat by passing only verified facts and synaptic priming traces to the renderer, saving massive prompt token costs.

This release represents half a year of non-stop work on Vector Symbolic Architectures and knowledge graphs. It is completely open-source, and I would love to hear feedback from folks who are pushing production RAG systems to their limits. Please check out the project and drop a star if you find the approach interesting: https://github.com/roandejager/Hillock

​If you use LLMs for writing, you know the telltale signs: corporate filler, overly formal transitions, and repetitive phrasing like "in today's fast-paced digital landscape."

​Most default prompts push AI models toward a generic, overly polished tone that reads like a press release. To get natural, engaging content, you have to actively prompt against those robotic default habits.

​The key comes down to a few core rules: enforce sentence variety, explicitly ban AI buzzwords, assign a clear perspective, and direct the model to write at a specific reading level. When you structure your instructions around how humans actually communicate, the quality of the output improves drastically.

​What specific rules or negative prompts have you found most effective for making AI draft naturally?

​Here is the complete guide and prompt framework: https://interconnectd.com/blog/6/the-masteprompting-playbook-make-ai-write-like-a-human-not-a-robot/

1. Can irrelevant information in a long context make it harder for an LLM to identify and use the information needed to answer a question correctly?
2. How does the Transformer's self-attention mechanism (Query, Key, and Value) influence how an LLM identifies and uses relevant information? Does adding more tokens affect how attention is distributed across the context?
3. I've noticed that providing too much information to ChatGPT sometimes leads to incorrect answers, even when the correct information is present in the context. Can someone explain why this happens and how self-attention relates to it? When should we use RAG instead of providing the full context to an LLM? If the entire document collection fits within an LLM's context window, when would it be better to provide the full context rather than use RAG

Most RAG problems are not model problems. They are retrieval problems, and chunking is where a lot of them start.

Fixed-window chunking is simple, so everyone uses it. But it does not respect structure. A table gets cut right in the middle of a row.

Then you get two bad chunks. The first one has the header and half a row. The second one has the rest of the row, but no header, so on its own it means almost nothing.

The retriever brings back one of them, the model fills the gap confidently, and the user gets a wrong answer that sounds right.

I saw the same document give a different answer to the same question, just because the split landed in a different place.

What actually helped was chunking by structure, keeping a table together where possible, and testing retrieval itself, not just the final answer. Ask questions that need a full row, and see if the right chunk even comes back.

How are you chunking tables right now? Fixed window, or something structure-aware?

​

I’ve been thinking about a problem that seems easy to underestimate: when an AI coding session produces an unexpected result, how much of the session do you need to reconstruct what actually happened?

A transcript alone may not tell the whole story. Depending on the platform, useful evidence could include:

- The sequence of user and assistant messages

- Tool calls, their inputs, outputs, and timestamps

- Visible reasoning summaries and status changes

- Code or artifact revisions across the session

- Retries, regenerated responses, and execution branches

- The relationship between an action and the resulting change

I’m exploring whether a session exporter or telemetry layer that preserves these events in a structured, inspectable format would be genuinely useful for debugging, incident investigation, or evaluating AI coding agents.

I’m not assuming that capturing more data automatically makes an audit better, or that hidden internal reasoning needs to be exposed. The question is what evidence is actually necessary.

Three questions for people who build or use AI coding agents:

1. When an agent makes a mistake, what information is usually missing when you investigate it?

2. Are existing logs, traces, and session exports sufficient, or do you still have to reconstruct events manually?

3. Would a structured, replayable session record solve a real problem for you? If so, what would it need to preserve to be worth using?

Concrete examples from real workflows would be especially helpful. I’m trying to understand the actual gap before deciding what, if anything, is worth building.

Say you ask a RAG system a question and it gives you the wrong answer.

The first thing I'd check is whether the correct information was even in the retrieved chunks.

If it wasn't, there's not much point changing the prompt. The issue could be the search query, chunking, or filtering.

If the correct information was retrieved, try giving the model just that passage and asking the same question again.

If it answers correctly this time, the problem might be how it's handling the full retrieved context rather than retrieval itself.

Seems like a useful way to narrow things down before changing a bunch of things at once.

If your RAG app uses Chroma Cloud: writes stop on Nov 22, reads on Dec 22.

I built pgchroma, a server that implements Chroma's HTTP API on Postgres + pgvector. Your chromadb / LangChain / LlamaIndex code keeps working; you change CloudClient to HttpClient. pgchroma migrate --from-cloud copies your collections, resumes if interrupted, and verifies counts, records and search results.

It also runs what single-node Chroma can't: Chroma Cloud's Search API and sparse vectors (BM25 / SPLADE hybrid search), and it swaps Chroma Cloud's hosted Qwen and SPLADE embedding models for the same open models running locally.

Compatibility is tested by running the official client against a real Chroma server and pgchroma side by side (75 scenarios + the JS client). Tested on Neon; works with Supabase, RDS or a local Postgres.

Repo: https://github.com/TheJustas228/pgchroma Migration guide: https://thejustas228.github.io/pgchroma/migrate-from-chroma-cloud/

Disclosure: I built it with Claude Code (the commits say so), so the tests against a real Chroma server are the thing to judge it by.

Feedback and bug reports welcome, especially from anyone moving a large Chroma Cloud deployment.

Hi! :B. ive been building RAG systems for a while, and i can count on one hand the times ive heard or seen anyone mention what actually happens in production. The "security" aspects are often left aside. I come from cybersec/pentesting, so i really never understood why its ignored in most courses and tutorials i see. Its always "build a vector store, plug an llm, done".

Im sharing a course on Data Eng, ML, DL, and AI where the last capstone consists of building a production RAG Agent, and then attacking it. Prompt injection, OWASP LLM Top 10, red teaming your own guardrails.

This is just my way of seeing it, but i think we need to understand how the system fails, otherwise you are just "hoping" ...

Anyway, the course itself is free :B, it has 173 lessons, and covers everything from python basics to training a small LLM. But since this is a RAG community, these might be the ones that are interest:

• Part VI (AI Engineering) covers retrieval, RAG architecture, and agents.
• Part VIII (MLOps) goes into evaluation and monitoring.
• The final capstone is where you build and break the agent.

So here we go :B : https://github.com/BlueRingsLabs/no-shortcuts

Hope its useful for somebody here :)

Oh and while we're on the topic, i'm curious how you guys handle automating security evals in CI. manual red teaming is everywhere but automated prompt injection tests are super rare. do u use PyRIT/Garak or build your own adversarial datasets?

The usual faithfulness check for a RAG answer is a second LLM as judge: another API call per answer, and possibly the same blind spots as the generator.

We tried something smaller: split the answer into sentences, pair each with the retrieved passages on the same topic, and run a small NLI cross-encoder (DeBERTa-v3-small, CPU) over each pair. Does the passage support the sentence, contradict it, or say nothing?

One small worked example, not a benchmark. The failures were the useful part.

Setup

Question: "What is the refund policy for annual plans?" Retrieval returned 5 passages, one off-topic ("The support office is closed on public holidays."). Answer A is a typical LLM draft (6 sentences), Answer B a careful one (2 sentences).

1. Only compare sentence/passage pairs with cosine similarity above 0.45.
2. Score each pair with NLI (entailment, contradiction).
3. Label: SUPPORTED, CONTRADICTED, UNPROVEN (nothing on-topic supports or contradicts it), or DISPUTED (one passage entails it above 0.6, another contradicts it).
4. Gate: any CONTRADICTED = BLOCK, all SUPPORTED = SHIP, else REVIEW.

Answer A

Answer A: 1 of 6 supported, faithfulness 0.17, BLOCK. Answer B: 2 of 2, faithfulness 1.0, SHIP.

What worked

• Caught the reversal. Sentence 2 vs "After 30 days, annual plans are refunded pro rata for the unused full months": 0.999 contradiction.
• Unsourced sentences don't pass silently. 5 and 6 came back UNPROVEN.
• The similarity floor kept the off-topic passage out (best similarity 0.235). NLI models will call two unrelated sentences a confident "contradiction": in a separate test, the same model scored "learning to play the cello" as contradicting "office is in Denver" at 0.997.
• You get the sentence and the passage that decided it, so you know what to cut.

What didn't

• Scope. Sentence 1 is correct, and passage 1 entails it at 0.998. But "after 30 days... pro rata" scores 0.995 contradiction against it. The model misses that the rules cover different time windows. Rewording the passage didn't help (0.995 again). That's why DISPUTED exists: blocking on any contradiction blocks correct answers.
• Numbers. "Within 24 hours" vs "within 5 to 10 business days" scored 0.930 entailment. Sentence 4 was only flagged because another passage happened to contradict it.
• Support isn't truth. Stale passages pass stale answers.
• Retrieval limits what can be checked. A true sentence whose passage wasn't retrieved comes back UNPROVEN.

Caught vs missed

The 0.45 floor sits between the off-topic passage (0.235) and the weakest passage the check needed (0.561). That's one dataset; score a labelled sample of your own first.

Where we landed: NLI as the cheap first gate on every answer, DISPUTED and UNPROVEN to a human or an LLM judge, and numbers and dates compared as typed values.

How are you handling numbers and dates in faithfulness checks?

Disclosure: I work on SynapCores, the database we ran this in (embeddings and NLI run inside it). Full SQL in a comment.

I build document ingestion tools, and I tested them on real public documents: Federal Reserve releases, Census tables, policy PDFs. Nothing crashed. That was the problem. Everything looked fine and was quietly wrong.

WHAT BROKE

1. Tables lost their row labels. Federal Reserve and Census tables came back as columns of bare numbers because table detection skipped the label column. A number without its label is useless to a retriever.
2. Some chunks started in the middle of a word.
3. Scanned PDFs came back empty when OCR was off, and the pipeline treated empty as success.
4. Blank pages counted as real pages.
5. Password-protected and damaged files failed with raw parser errors nobody could act on.

WHAT FIXED IT

Row labels now get attached from the table's own row positions. Chunks only start on word boundaries. Scanned and blank pages get flagged instead of passed along, and bad files get a plain error message.

The edge in RAG isn't the model. It's whether your chunks still mean something.

WHAT I BUILT

Three tools on the Apify Store that all output the same chunk format: PDFs with table coordinates; Word, Excel, PowerPoint, HTML and images; and audio or video with timestamps. The free, open-source examples repo includes a LlamaIndex example and goes from a fresh clone to a cited answer in about 14 seconds, no account needed.

Disclosure: I built these. They're pay per page or per minute (a third of a cent per PDF page), and failed or empty files are never charged.
https://github.com/landoCommandoz/rag-ingestion-examples

Happy to answer questions about any document type. I've broken most of them at this point.

I would treat embedding specificity as a retrieval signal, not as permission to trust a memory. A detailed record can still be outdated, superseded, or outside the current user's scope.

Hyperbolic embeddings make the distinction interesting. In the tested image-and-text model, direction represents semantic content while distance from the center carries information about how broad or specific it is. That could be useful when deciding whether a query needs a general concept or a particular instance. The reported experiments concern image and text retrieval, though; they do not establish better agent-memory behavior.

The mechanics do not require a separate hyperbolic database. For the Lorentz representation described, flipping the query's first coordinate lets inner-product search recover the hyperbolic ordering. Centering the stored documents' first coordinate can improve how an approximate index organizes them. Vector databases like Milvus can provide that candidate-retrieval path.

The unresolved part is what happens after retrieval. I would still carry writer, user scope, timestamp, and supersession metadata separately. A larger radius is not a confidence score, and the image and text radii in the experiment occupy different ranges, so comparing them as one universal specificity scale would be misleading.

I would probably test this on paired general and specific memory requests while holding access filters fixed. Then I would add an older, detailed record that conflicts with a newer correction. The question is whether geometry helps find the right level of detail without letting specificity override the system's authority rules.

I would accept a slightly larger compressed index if it kept the retrieval recall I actually need. A smaller memory footprint is only useful if recovering the missing neighbors does not consume the latency budget.

For vector databases like Milvus, the interesting distinction is between the representation used while traversing an index and the representation used to check the surviving candidates. SVS LeanVec reduces dimensionality and quantizes the primary vectors, then reranks candidates against a full-dimensional secondary representation. That secondary copy is still quantized; it is not a hidden float32 copy.

This changes what I would measure. Comparing file sizes alone misses the cost of widening the search until two configurations reach the same recall. I would compare memory, throughput, and tail latency at a fixed recall target, then check whether that target actually preserves the evidence needed by the RAG application.

There is an operational catch. The published tests used Intel Xeon 6 machines, and LeanVec benefits from AVX-512. SVS also requires a source build with support enabled rather than the default container image. Those are deployment requirements, not details I would bury beneath a throughput chart.

I would probably start with the existing index as the baseline and sweep search settings on the intended hardware. A gain that survives equal recall, the real concurrency level, and the build process is much more convincing than a smaller index measured at a different quality point.

I'm James Luan, CTO at Zilliz, the company behind Milvus. Milvus is an open-source vector database used to store, index, and search embeddings over unstructured data. Vector Lakebase is our next lake-native expansion around that serving path.

The first problem we ran into with GPTCache was the difference between a similar request and a reusable answer. Repeated questions make caching appealing, but similarity leaves out conditions that an agent or downstream program may depend on.

A request for code only can retrieve an answer containing working code plus two paragraphs of explanation. The subject matches; the output contract does not. If another component expects to parse code directly, the extra text becomes a failure at the handoff.

Some differences are even smaller. Asking for sentences ending in “Gaming” is not satisfied by sentences ending in “gaming.” A regional restriction also changes the answer: a question about medications available in the UK cannot simply inherit a generic answer about those medications.

An identical request is not enough either. If the cached response ends at “Here are the steps,” exact matching only returns the same incomplete answer more confidently.

We tried a reranker that compared the current and cached questions. That could improve relevance scoring, but it did not inspect whether the answer was complete or obeyed the required format, region, or version. The missing information was in the answer and its relationship to the current request.

For an agent workflow, this puts the cache boundary before the next component consumes the response. Retrieval can nominate a candidate. A separate check has to decide whether that particular answer meets the current contract. Increasing similarity confidence does not perform that second job.

I'm James Luan, CTO at Zilliz, the company behind Milvus. Milvus is an open-source vector database used to store, index, and search embeddings over unstructured data. Vector Lakebase is our next lake-native expansion around that serving path.

When we built GPTCache in 2023, we wanted to avoid generating the same answer repeatedly. The hard part was not retrieving a similar question. It was deciding whether the cached answer actually satisfied the new request. A code-only request can fail on extra explanation; a regional restriction can change which answer is usable.

Jev made an extra check economically interesting. Its listed price in my experiment was $0.042 per million input tokens, with no output charge. I kept retrieval and added a gate over the current request, cached request, cached answer, and available time information. It checked task, constraints, output, freshness, and context, using the lowest score as the reuse decision. This evaluates suitability for reuse, not independent factual correctness.

I used 1,536 examples, reserving 384 for calibration and 1,152 for a held-out test. In the initial comparison, Jev reached 99.7% precision with one incorrect reuse. Separately, a threshold of 0.80 selected on the calibration set accepted 227 reusable answers and no observed incorrect ones on the held-out set. Those are different evaluation stages, and neither guarantees zero future false hits.

Prompt rules mattered too. An early freshness rule rejected a timeless hypothetical because it lacked a timestamp. Checking whether the answer depends on changing facts before requiring time information fixed that example without retraining.

The tradeoff still includes missed valid hits and latency: the remote API's median was 0.79 seconds in my tests. That is easier to justify before an expensive reasoning run than before a very short generation. It also suggests other experiments, such as deciding whether to continue searching or interpreting operational signals. Permissions and resource limits still belong in code.

I put the full evaluation and reasoning here:

https://milvus.io/blog/jevs-cheap-decisions-make-semantic-caching-worth-another-try.md

I'd love to hear your thoughts on where a reuse gate would fit in your latency budget.

Built a RAG Evaluation Pipeline at HYPNEXIS’26 | Problem Statement AI04

We are excited to share a project our team developed during HYPNEXIS’26, a 36-hour hackathon hosted by Manipal University Jaipur.

Our project, RAG Evaluation Pipeline, focuses on evaluating and improving Retrieval-Augmented Generation (RAG) systems by measuring how effectively they retrieve relevant information and support accurate answers.

What does our project do?

Our pipeline aims to make RAG evaluation more systematic through metrics and diagnostic techniques, including:

Recall@5: Measures whether relevant information appears among the top five retrieved results.

Precision@5: Measures the relevance of the retrieved results.

MRR (Mean Reciprocal Rank): Evaluates how highly the first relevant result is ranked.

The goal is to help developers assess retrieval quality, identify weaknesses, and make informed improvements to their RAG pipelines.

Project Links

Live Demo:

https://jitendravishwakarma746-rag-evaluation-pipeline-app-ob2mg0.streamlit.app/

GitHub Repository:

https://github.com/Jitendravishwakarma746/RAG-Evaluation-pipeline

Our Team — CodeCrafter

Prabhu

Gyan Ganga College of Technology

GitHub: https://github.com/ThePaleBlueDot405

Instagram: @hellofromprabhu

Jitendra Vishwakarma

Gyan Ganga College of Technology

LinkedIn: https://www.linkedin.com/in/jitendra-vishwakarma-93283939b/

GitHub: https://github.com/Jitendravishwakarma746

Instagram: @jitendra_vishwakarma_18

Raj Chaturvedi

Shri Ram Institute of Technology

Instagram: @Raj_chaturvedi_304

Pawan Gupta

Gyan Ganga College of Technology

Instagram: @pawan_gupta_1294

Although we couldn't make it to the final round, participating in HYPNEXIS’26 was a valuable learning experience. Working under time constraints challenged us to collaborate, solve technical problems, and turn our ideas into a working application.

We are grateful for this opportunity and look forward to improving our skills, refining our project, and building more meaningful solutions together.

#HYPNEXIS26 #Hackathon #RAG #ArtificialIntelligence #MachineLearning #InformationRetrieval #Python #Streamlit #TeamCodeCrafter #GyanGangaCollegeOfTechnology #ShriRamInstituteOfTechnology

Hi everyone,

Some of you know me from the RAG_Techniques repo, or from the RAG book I shared here in April. For the last two years I've also maintained the GenAI Agents and Agents Towards Production repositories, and I just finished a book on AI agents.

The chapter most relevant to this sub is retrieval as a tool. Its rule: keep retrieval a fixed step until one search stops being enough. Then give the model a search tool, a test for when it has enough, and a rule to cite or refuse. It also goes through what breaks when you do: the search that never comes back empty, the endless investigation, the invented citation, the right quote from the wrong document, and the retrieved passage that gives the agent orders.

Every chapter answers the same four questions about one pattern: how it works, what breaks, what it costs, and when it is overkill. The 28 chapters are in seven parts:

• Before You Build: whether you need an agent at all, what one actually is, and the fixed workflows to try first.
• Giving the Model Hands: tool calling, designing tools the model can use, retrieval, the web, running code, standard protocols, and skills.
• Making It Reliable: context engineering, evals, tracing, guardrails and prompt injection, humans in the loop, and what an agent really costs.
• Planning and Reasoning: reason-act-observe, plan-then-execute, reflection, and thinking budgets.
• State and Memory: runs that survive a crash, and what an agent should keep between sessions.
• Many Agents: whether to split the work at all, orchestrators and workers, and handoffs.
• Shipping: deployment, the anatomy of a coding agent, and how agents fail.

Full disclosure: I'm the author. I want the community that helped build the repos to actually be able to get it, so the Kindle version is $0.99 for the next 24 hours (the floor Amazon allows).

Happy to answer any technical questions about the patterns in the guide or the repos!

Here's the link: https://www.amazon.com/dp/B0HNXRSRH2

I have been working on Limma for a year now with the goal of building a proactive solution to the reliability issues that plague practical AI and agentic use in companies with RAG.

It is a single MCP that exists for the sole purpose of ensuring that the right information is retrieved with every query and no information is skipped and or missed at any scale for total precision and recall.

We do this by building a map and structuring all data-points within unstructured data as well as federating structured data for addressability so that agents don't need to search through chunks directly and or solely rely on ranking in vector search and instead only get precisely the right information .

We wanted this to be extremely easy to use so that anyone can set it up and try it within minutes so that the precision issues (especially with large quantities of data) that LLMs struggle with can finally be remedied. We use self-hosted models (primarily Qwen models as of current) so that the system can be deployed on one's own server if needed, and also to optimize for speed.

It handles OCR and we have data connectors for various resources to make it easy to connect, and it can be used with agents or just for normal Q&A!

I have a version for anyone to try out on my website limma.ai and would love any feedback you have!