🔬 AI Research Paper Intelligence System

An industry-grade Retrieval-Augmented Generation (RAG) microservice designed to ingest, semantically query, summarize, and structurally analyze over 117,000+ scientific documents in sub-millisecond execution windows.

This backend engine bridges modern Natural Language Processing (NLP) pipelines with high-speed vector indexing, converting vast unstructured text into an interactive, intelligence-driven API.

---

🚀 Core Features & Business Value

• Sub-Millisecond Semantic Search: Explores high-dimensional conceptual meanings instead of raw text keywords.
• Context-Aware Generative Summarization: Condenses dense scientific texts into digestible executive summaries natively on the GPU.
• Granular Metadata Extraction (NER): Auto-identifies critical named entities (Organizations, Locations, Concepts) embedded within text bodies.
• Automated Architectural Caching: Processes massive matrix operations in safe, resume-ready checkpoint coordinates.

---

📊 The Data Foundation: ML-ArXiv-Papers

This system utilizes the ML-ArXiv-Papers dataset (comprising 117,592 records).

Why ArXiv?

Scientific research text presents one of the toughest challenges for standard NLP models. ArXiv data contains highly dense, domain-specific terminology, complex mathematical notation, and nuanced academic jargon[cite: 3]. Building a system that successfully parses, indexes, and understands this dataset proves the architecture can survive real-world, high-complexity enterprise data.

---

⚙️ The Technical Pipeline (Process Involved)

The machine learning lifecycle of this system flows across four unified operational steps:

1. NLP Text Preprocessing

Raw titles and abstracts are concatenated into a singular corpus[cite: 3], normalized to strip out breaking line characters[cite: 3], and stripped into optimized text streams to match deep learning tokenizer context window boundaries.

2. High-Dimensional Text Embeddings

Instead of viewing words as plain text strings, the system uses all-MiniLM-L6-v2 to transform long textual sentences into dense 384-dimensional floating-point vectors. These mathematical coordinates map the exact semantic intent and context of the writing.

3. High-Performance Vector Indexing (Semantic Search)

Standard relational databases stall when computing vector similarities across 117,000+ dimensions. We utilize Facebook AI Similarity Search (FAISS) configured with an Inner Product matrix (IndexFlatIP). By normalizing our embeddings, the system runs mathematical Cosine Similarity rankings at lightning speed to surface contextually matching documents instantaneously.

4. Downstream Intelligence Layer (RAG)

When a user queries the API, the system fetches the top matches and pipelines them through dual specialized transformer architectures:

• Summarization Layer: BART-Large-CNN extracts the core findings into a custom abstract.
• Named Entity Recognition Layer: BERT-NER isolates critical technical keywords, institutions, and core entities.

---

📡 API Architecture Blueprint

POST /api/v1/search

Executes semantic retrieval, abstract generation, and NER parsing inside a single unified endpoint response.

Sample Request JSON Payload:

{
  "query": "transformer models in computer vision",
  "top_k": 3
}

🛠️ Technology Ecosystem

• Core Backend: FastAPI, Uvicorn, Pydantic, AsyncIO

• Vector Engine: FAISS (Facebook AI Similarity Search)

• Deep Learning Frameworks: PyTorch, Hugging Face Transformers, Sentence-Transformers

• Data Science Suite: Pandas, NumPy, Hugging Face Datasets[cite: 3]

👨‍💻 Project Maintainer

Kislay Anand

B.Tech Computer Science student at IIMT College of Engineering