Skip to content
Rayyan Ahemad
← All projects
RAG + AgentShipped

EU AI Act Assistant

Cited question answering over the 144-page EU AI Act, with hybrid search, local re-ranking, an agent with human-approved actions and measured injection defences.

Explore the architecture ↓See it running ↓Download the deck ↓
Recall@5 on 44 eval questions
0.87
Recall@1 with re-ranking, up from 0.37
0.63
prompt-injection attacks resisted (was 7/10)
10/10

Architecture

How it works

  1. Clean the question
  2. FAISS + BM25 → RRF
  3. bge-reranker (local GPU)
  4. Relevance gate ≥ 0.05
  5. Trimmed context
  6. Answer + page citations
  • Cited answer + confidence score
  • “I don't have information”
Agent mode swaps the fixed path for a tool loop: six tools, a 6-step cap, and a code guard so a message can only reach the human approval gate when the user asks for it.

  1. Index

    The 144-page regulation is extracted with PyMuPDF, split into 1,000-character chunks with 200 overlap, embedded (text-embedding-3-small, or bge-base on the local GPU) into FAISS, and indexed again for BM25 keyword search.

  2. Clean the question

    Before anything runs, sentences that give the assistant orders and text posing as context or citations are stripped out, so a question can't smuggle in fake passages. That alone took injection resistance from 7/10 to 10/10.

  3. Hybrid search

    FAISS (meaning) and BM25 (exact terms like “Article 6(1)” or “Annex III”) each return their top 15, merged by Reciprocal Rank Fusion. A question that names an article or annex also gets that unit's own text by its heading.

  4. Re-rank and gate

    A bge-reranker cross-encoder scores up to 30 candidates on the local GPU. If the best passage scores below 0.05, the question is refused without calling the model at all.

  5. Answer with citations

    At most 5 passages go to the model, trimmed to their relevant sentences (29% fewer tokens). Every claim cites source and page; a check flags any cited page the model was never given, and each answer gets a confidence score.

  6. Agent mode

    The model picks from six tools (search, calculator, exact quote, definitions, date gaps and a send_message draft), capped at 6 steps. Sending is guarded in code and needs a person to confirm, so a poisoned document can't make it act.

  7. Evaluate everything

    44 questions with expected pages drive recall@k, MRR, an ablation per retrieval stage, a refusal-threshold table and a regression script that fails on any regression. Every API call is priced, with a hard $5 budget cap.

Screenshots

A cited answer with its confidence score and the live cost table. (opens full-size image)
A cited answer with its confidence score and the live cost table.
The clickable How it works flowchart: RAG pipeline next to the agent loop. (opens full-size image)
The clickable How it works flowchart: RAG pipeline next to the agent loop.
Routing check: does the agent pick the right tool for each kind of question? (opens full-size image)
Routing check: does the agent pick the right tool for each kind of question?
Retrieval trace: every step of a question, with what happened and how long it took. (opens full-size image)
Retrieval trace: every step of a question, with what happened and how long it took.

Information

The EU AI Act is 144 pages of legal text: 113 articles, 13 annexes and 180 recitals. Answers about it have to be exact (fine amounts, dates, article numbers), so the assistant answers only from the regulation, cites the page, and refuses when the answer isn't there.

EPITA · Team ParisAI · Team of 3, with Sai Prasad Bandari & Prodipta Paul

Recall@5 on 44 eval questions
0.87
Recall@1 with re-ranking, up from 0.37
0.63
prompt-injection attacks resisted (was 7/10)
10/10
poisoned sends reached the approval gate
0/3
wrong refusals on answerable questions
0%
per question in fully local mode
$0

What I built

  • Built the first version: FAISS retrieval, gpt-4o-mini answers with page citations, Gradio UI
  • Moved re-ranking onto the local GPU and added query and retrieval caches
  • Added api, hybrid and fully local run modes, streamed answers and smaller prompts
  • Built the prompt-injection defences and test suite: 7/10 → 10/10 attacks resisted
  • Wrote the eval runner: recall@k, MRR, ablation and threshold tables, regression checks
  • Added the get_definition and date_diff agent tools, the send guard and the action-injection test
  • Built the retrieval trace, the How it works, Routing and Evaluation pages, and the presentation deck
  • Python
  • FAISS
  • BM25
  • bge-reranker
  • gpt-4o-mini
  • Qwen2.5-3B
  • Gradio
  • uv

Download the deck (.pptx)