---
title: SPJIMR Topic Modelling Submission App
sdk: gradio
sdk_version: "5.25.2"
app_file: app.py
pinned: true
license: mit
short_description: Topic modelling with council validation
---

# SPJIMR Topic Modelling Submission App

This app generates the final topic modelling submission outputs requested in the SPJIMR Management Minor feedback.

## What It Does

- Loads a Scopus CSV.
- Builds one paper-level text field from `Title + Abstract + DOI`.
- Attempts SPECTER2/SPECTER transformer embeddings first, then falls back to TF-IDF + SVD so the app still runs end to end if transformer dependencies are unavailable.
- Runs UMAP + HDBSCAN parameter optimization toward 15-25 crisp clusters with 5-100 papers per cluster.
- Falls back to KMeans only if density clustering cannot meet the required range.
- Labels each cluster through an in-app council:
  - three Mistral council personas when `MISTRAL_API_KEY` is configured,
  - deterministic keyword/PAJAIS/local semantic fallback when no key is available, so the app still runs.
- Exports validation evidence, top three paper titles per cluster, PAJAIS mapping, optimizer logs, and TCCM/computational-technique validation for the top-cited 100 papers.
- Provides a Compliance tab showing PASS / CONFIG_REQUIRED / INPUT_REQUIRED / MANUAL_REQUIRED for each professor requirement.
- Provides a TCCM Dual Validation tab for NotebookLM extraction CSV plus a second LLM extraction CSV from full-text PDFs.

## Main Deliverables

- `outputs/comparison.csv`
- `outputs/taxonomy_map.json`
- `outputs/topic_model_report.md`
- `outputs/narrative.txt`
- `outputs/cluster_optimization_log.csv`
- `outputs/llm_council_validation.csv`
- `outputs/tccm_validation.csv`
- `outputs/tccm_dual_validation.csv`
- `outputs/compliance_checklist.csv`
- `outputs/run_metadata.json`
- `outputs/combined_labels.json`

## Run Locally

```bash
pip install -r requirements.txt
python app.py
```

Then open the Gradio URL and click **Run Complete Pipeline** after uploading the Scopus CSV.

For command-line generation:

```bash
python run_pipeline.py
```

## Optional LLM Setup

Set `MISTRAL_API_KEY` to use Mistral as one council labeler. The app still runs without it.