Unstructured Medical Text Miner

Unstructured Medical Text Miner (ID: 213)

When to Use

Key Features

See ## Features above for related details.

Dependencies

pandas>=1.3.0
spacy>=3.4.0
scispacy>=0.5.1
radlex (for radiology terminology)
negspacy (for negation detection)

Example Usage

See ## Usage above for related details.

cd "20260318/scientific-skills/Evidence Insight/unstructured-medical-text-miner"
python -m py_compile scripts/main.py
python scripts/main.py --help

Example run plan: 1. Confirm the user input, output path, and any required config values. 2. Edit the in-file CONFIG block or documented parameters if the script uses fixed settings. 3. Run python scripts/main.py with the validated inputs. 4. Review the generated output and return the final artifact with any assumptions called out.

Implementation Details

See ## Workflow above for related details.

Quick Check

Use this command to verify that the packaged script entry point can be parsed before deeper execution.

python -m py_compile scripts/main.py

Audit-Ready Commands

Use these concrete commands for validation. They are intentionally self-contained and avoid placeholder paths.

python -m py_compile scripts/main.py
python scripts/main.py --help
python scripts/main.py -h

Workflow

1. Confirm the user objective, required inputs, and non-negotiable constraints before doing detailed work. 2. Validate that the request matches the documented scope and stop early if the task would require unsupported assumptions. 3. Use the packaged script path or the documented reasoning path with only the inputs that are actually available. 4. Return a structured result that separates assumptions, deliverables, risks, and unresolved items. 5. If execution fails or inputs are incomplete, switch to the fallback path and state exactly what blocked full completion.

Overview

Mine "text data" that has been long overlooked in MIMIC-IV, extracting unstructured diagnostic logic, order details, and progress notes.

Purpose

The MIMIC-IV database contains large amounts of structured data (vital signs, laboratory results, etc.), but its true clinical value is often hidden in unstructured text:

This Skill provides a complete text mining toolchain to transform raw medical text into analyzable structured insights.

Features

1. Text Extraction

2. Information Extraction

3. Clinical Logic Parsing

4. Structured Output

Usage

from skills.unstructured_medical_text_miner.scripts.main import MedicalTextMiner
Initialize miner
miner = MedicalTextMiner()
Load MIMIC-IV note data
miner.load_notes(notes_path="path/to/noteevents.csv")
Extract all text records for a specific patient
patient_texts = miner.get_patient_texts(subject_id=10000032)
Execute complete information extraction
insights = miner.extract_insights(
    text=patient_texts,
    extract_entities=True,
    extract_relations=True,
    extract_timeline=True
)

Input

Data Sources

Field Requirements

Output

Entity Extraction Results

{
  "entities": [
    {
      "text": "acute myocardial infarction",
      "type": "DISEASE",
      "start": 156,
      "end": 183,
      "confidence": 0.94
    },
    {
      "text": "aspirin 81mg",
      "type": "MEDICATION",
      "start": 245,
      "end": 257,
      "attributes": {
        "dose": "81mg",
        "frequency": "daily"
      }
    }
  ]
}

Clinical Logic Graph

{
  "clinical_logic": {
    "presenting_complaint": "chest pain",
    "differential_diagnoses": ["ACS", "PE", "aortic dissection"],
    "workup": ["ECG", "troponin", "CTA chest"],
    "final_diagnosis": "STEMI",
    "treatment_plan": ["PCI", "dual antiplatelet"]
  }
}

Temporal Events

{
  "timeline": [
    {
      "time": "2020-03-15 08:30",
      "event": "admission",
      "description": "presented with chest pain"
    },
    {
      "time": "2020-03-15 09:15",
      "event": "ECG",
      "description": "ST elevation in V1-V4"
    }
  ]
}

Configuration

config.yaml
extraction:
  entity_types: ["DISEASE", "SYMPTOM", "MEDICATION", "PROCEDURE", "ANATOMY"]
  relation_types: ["TREATS", "CAUSES", "CONTRAINDICATED_WITH"]
  enable_negation_detection: true
  
models:
  ner_model: "en_core_sci_lg"  # or "en_core_sci_scibert"
  relation_model: "custom_relation_extractor"
  
output:
  format: "json"  # json/fhir/kg
  include_raw_text: false

CLI Usage

Process single file
python -m skills.unstructured_medical_text_miner.scripts.main \
  --input notes.csv \
  --output extracted.json \
  --extract all
Process specific patient
python -m skills.unstructured_medical_text_miner.scripts.main \
  --subject-id 10000032 \
  --db-path mimic_iv.db \
  --output patient_insights.json

References

1. MIMIC-IV Clinical Database: https://physionet.org/content/mimiciv/ 2. scispacy: https://allenai.github.io/scispacy/ 3. NegEx/negspacy for negation detection 4. FHIR Clinical Document specifications

Author

Skill ID: 213 Category: Medical Data Mining Complexity: Advanced

Risk Assessment

| Risk Indicator | Assessment | Level | |----------------|------------|-------| | Code Execution | Python/R scripts executed locally | Medium | | Network Access | No external API calls | Low | | File System Access | Read input files, write output files | Medium | | Instruction Tampering | Standard prompt guidelines | Low | | Data Exposure | Output files saved to workspace | Low |

Security Checklist

Prerequisites

Python dependencies
pip install -r requirements.txt

Evaluation Criteria

Success Metrics

Test Cases

Output Requirements

Every final response should make these items explicit when they are relevant:

Error Handling

Input Validation

This skill accepts requests that match the documented purpose of unstructured-medical-text-miner and include enough context to complete the workflow safely.

Do not continue the workflow when the request is out of scope, missing a critical input, or would require unsupported assumptions. Instead respond:

> unstructured-medical-text-miner only handles its documented workflow. Please provide the missing required inputs or switch to a more suitable skill.

References

Response Template

Use the following fixed structure for non-trivial requests:

1. Objective 2. Inputs Received 3. Assumptions 4. Workflow 5. Deliverable 6. Risks and Limits 7. Next Checks

If the request is simple, you may compress the structure, but still keep assumptions and limits explicit when they affect correctness.

from skills.unstructured_medical_text_miner.scripts.main import MedicalTextMiner
Initialize miner
miner = MedicalTextMiner()
Load MIMIC-IV note data
miner.load_notes(notes_path="path/to/noteevents.csv")
Extract all text records for a specific patient
patient_texts = miner.get_patient_texts(subject_id=10000032)
Execute complete information extraction
insights = miner.extract_insights(
    text=patient_texts,
    extract_entities=True,
    extract_relations=True,
    extract_timeline=True
)

config.yaml
extraction:
  entity_types: ["DISEASE", "SYMPTOM", "MEDICATION", "PROCEDURE", "ANATOMY"]
  relation_types: ["TREATS", "CAUSES", "CONTRAINDICATED_WITH"]
  enable_negation_detection: true
  
models:
  ner_model: "en_core_sci_lg"  # or "en_core_sci_scibert"
  relation_model: "custom_relation_extractor"
  
output:
  format: "json"  # json/fhir/kg
  include_raw_text: false