README.md

metadata

title: SPJIMR Topic Modelling Submission App
sdk: gradio
sdk_version: 5.25.2
app_file: app.py
pinned: true
license: mit
short_description: Topic modelling with council validation

SPJIMR Topic Modelling Submission App

This app generates the final topic modelling submission outputs requested in the SPJIMR Management Minor feedback.

What It Does

• Loads a Scopus CSV.
• Builds one paper-level text field from Title + Abstract + DOI.
• Attempts SPECTER2/SPECTER transformer embeddings first, then falls back to TF-IDF + SVD so the app still runs end to end if transformer dependencies are unavailable.
• Runs UMAP + HDBSCAN parameter optimization toward 15-25 crisp clusters with 5-100 papers per cluster.
• Falls back to KMeans only if density clustering cannot meet the required range.
• Labels each cluster through an in-app council:
  • three Mistral council personas when MISTRAL_API_KEY is configured,
  • deterministic keyword/PAJAIS/local semantic fallback when no key is available, so the app still runs.
• Exports validation evidence, top three paper titles per cluster, PAJAIS mapping, optimizer logs, and TCCM/computational-technique validation for the top-cited 100 papers.
• Provides a Compliance tab showing PASS / CONFIG_REQUIRED / INPUT_REQUIRED / MANUAL_REQUIRED for each professor requirement.
• Provides a TCCM Dual Validation tab for NotebookLM extraction CSV plus a second LLM extraction CSV from full-text PDFs.

Main Deliverables

• outputs/comparison.csv
• outputs/taxonomy_map.json
• outputs/topic_model_report.md
• outputs/narrative.txt
• outputs/cluster_optimization_log.csv
• outputs/llm_council_validation.csv
• outputs/tccm_validation.csv
• outputs/tccm_dual_validation.csv
• outputs/compliance_checklist.csv
• outputs/run_metadata.json
• outputs/combined_labels.json

Run Locally

pip install -r requirements.txt
python app.py

Then open the Gradio URL and click Run Complete Pipeline after uploading the Scopus CSV.

For command-line generation:

python run_pipeline.py

Optional LLM Setup

Set MISTRAL_API_KEY to use Mistral as one council labeler. The app still runs without it.