claw-semantic-sim

Why this exists

If you ask ChatGPT to "measure research neglect for diseases," it will:

• Not know which embedding model to use for biomedical text
• Hallucinate metrics that sound plausible but have no methodological grounding
• Skip quality filtering (year coverage, abstract coverage, minimum papers)
• Not handle MPS acceleration or checkpointed batch processing
• Produce a single scatter plot with no disease classification

This skill encodes the correct methodological decisions:

• Uses PubMedBERT (the gold-standard biomedical language model)
• Fetches from PubMed with exponential backoff and NCBI rate limiting
• Quality filters: year coverage >= 70%, abstract coverage >= 95%, minimum 50 papers
• Batch embedding with Apple MPS acceleration and CPU fallback
• Checkpointed processing (resume after interruption)
• HDF5 storage with gzip compression and SHA-256 checksums
• Classification against WHO NTD list and Global South priority diseases
• Statistical significance testing (Welch's t-test, Cohen's d)

Key Finding

Neglected tropical diseases (NTDs) are significantly more semantically isolated than other conditions (P < 0.001, Cohen's d = 0.8+). They exist in knowledge silos with limited cross-disciplinary research bridges. The 25 most isolated diseases are disproportionately Global South priority conditions.

Demo (works out of the box)

The demo uses pre-computed embeddings and metrics for 175 GBD diseases and generates the full 4-panel figure instantly.