Skip to content

Environment Setup & Run Commands

From the repository root, enter topic_segmentation/ and run the commands below:

cd topic_segmentation

Installation

# uv (once)
curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"

# Create and activate the locked environment, repeat in every new shell
source scripts/activate.sh

# Punkt sentence splitter for IDEA-seg and MIND; activate.sh points NLTK to this folder
python -m nltk.downloader -d data/resources/nltk punkt

News, MIND, and clinical training use cache/tmp/ unless TMPDIR is set. C4 preprocessing keeps its temporary Arrow files in cache/c4/.

Data and weights

IDEA-seg, the MIND article bodies, and the trained thesis models are not distributed with the repository. The public models are downloaded below; C4 is downloaded during preparation; MIMIC-IV-Note requires credentialed PhysioNet access.

Models

hf download answerdotai/ModernBERT-base --local-dir checkpoints/modernbert_base
hf download google-bert/bert-base-uncased --local-dir checkpoints/bert_base
hf download FacebookAI/roberta-base --local-dir checkpoints/roberta_base
hf download allenai/longformer-base-4096 --local-dir checkpoints/longformer_base

# Chonkie baselines
hf download minishlab/potion-base-32M --local-dir checkpoints/baselines/chonkie/semantic-potion-base-32m
hf download mirth/chonky_modernbert_base_1 --local-dir checkpoints/baselines/chonkie/neural-modernbert-base

Data

IDEA-seg

The source articles are available in the IDEA dataset. IDEA-seg adds the segmentation labels and splits used in this thesis.

  • data/news/: train/val/test files, split into paragraphs.
  • data/news_sent/: the same articles, split into sentences.
  • data/news/raw/: saved pages, article list, and source dataset.

The pipeline uses the supplied splits in data/news/ and data/news_sent/. To rebuild IDEA-seg separately from saved pages:

python scripts/prepare_data.py news

The rebuilt dataset and splits go in runs/news_construction/<time>/.

MIND

Download the original dataset from the MIND website. The download does not include article bodies; these must be supplied separately (see the official data description).

Put MIND articles with full text in data/mind/full/news.csv. Columns:

news_id,category,subvert,title,abstract,url,entity,ab_entity,body

Standalone preparation writes predicted boundaries to data/mind/full/ and training splits to data/mind/prepared/. The pipeline writes both under its MIND run's data/ folder. Subset sizes are 2k, 5k, 10k, and 25k articles.

C4 realnewslike

Preparation downloads realnewslike from allenai/c4 on Hugging Face. Tokenized data with paragraph-boundary labels goes in data/c4/tokenized_modernbert/.

MIMIC-IV-Note 2.2

Download MIMIC-IV-Note 2.2 from PhysioNet after completing the required training, obtaining credentialed access, and signing the data use agreement.

  • Raw notes: data/mimic/raw/mimic-iv-note-deidentified-free-text-clinical-notes-2.2/.
  • Prepared evaluation and training data: data/mimic/.

Check

With IDEA-seg and the four backbones in place, check versions, data, and tokenizers offline:

python scripts/checkenv.py

Run Commands

These are independent examples using configured inputs. Use run_ts_pipeline.sh to chain newly trained checkpoints automatically.

Stage scripts resolve relative paths from topic_segmentation/ and accept --dry-run (print commands) and --help.

# ── 1. Data ───────────────────────────────────────────────────────────────────
# MIND comes after stage 3: its silver labels need the news teacher (see below)
python scripts/prepare_data.py c4 mimic

# ── 2. Boundary pre-training on C4 ────────────────────────────────────────────
# Smoke test on 100k examples
python scripts/run_boundary_pretraining.py --smoke-test --gpus 0

# Full run
python scripts/run_boundary_pretraining.py --gpus 0 1

# ── 3. News fine-tuning ───────────────────────────────────────────────────────
# Experiment groups: backbones, boundary_pretraining, granularity, objectives, clinical_initialization
python scripts/run_news_finetuning.py --experiment backbones --gpu 0
python scripts/run_news_finetuning.py --experiment objectives --gpu 0

# One thesis run
python scripts/run_news_finetuning.py --run modernbert_c4_1024_cssl --gpu 0

# Any setting: --backbone bert|roberta|longformer|modernbert|modernbert_c4, --length 512|1024,
# --objective ts|ts_da|cssl|cssl_da|tssp|tssp_da|full_noda|full, --data news|news_sent
python scripts/run_news_finetuning.py --backbone roberta --length 512 --objective ts_da --gpu 0
python scripts/run_news_finetuning.py --backbone modernbert_c4 --length 1024 --objective full --data news_sent --gpu 0

# ── MIND data with the configured teacher; --teacher-model selects a new one ──
python scripts/prepare_data.py mind --gpu 0

# ── 4. MIND adaptation ────────────────────────────────────────────────────────
# Experiment groups: scale, final, objective_matrix, matched_steps, clinical_initialization
python scripts/run_mind_adaptation.py --experiment final --gpu 0

# One thesis run (reruns the adaptation it starts from)
python scripts/run_mind_adaptation.py --run mind_25k_full_news_ts --gpu 0

# One thesis run, starting from the saved adaptation
python scripts/run_mind_adaptation.py --run mind_25k_full_news_ts --recorded-adaptation --gpu 0

# Any setting: --size 2k|5k|10k|25k, --objective ts|ts_da|full, --matched-steps, --news-objective ts|ts_da|full
python scripts/run_mind_adaptation.py --size 10k --objective full --matched-steps --news-objective ts --gpu 0

# ── 5. Long-document inference ────────────────────────────────────────────────
# Experiment groups: strides, stage_matrix, matched_steps, baselines
python scripts/run_long_document_inference.py --experiment strides --gpu 0

# Any model and window setting
python scripts/run_long_document_inference.py --model runs/mind_adaptation/mind_25k_full_news_ts \
    --window 1024 --stride 256 512 768 1024 --no-protect --gpu 0
python scripts/run_long_document_inference.py --model runs/mind_adaptation/mind_25k_full_news_ts \
    --window 1024 --stride 512 --protect --gpu 0

# Chonkie baselines
python scripts/run_long_document_inference.py --baseline semantic --gpu 0
python scripts/run_long_document_inference.py --baseline neural --gpu 0

# ── 6. Clinical transfer ──────────────────────────────────────────────────────
# Train the news initializations first
python scripts/run_news_finetuning.py --experiment clinical_initialization --gpu 0
python scripts/run_mind_adaptation.py --experiment clinical_initialization --gpu 0

# Zero-shot: --models news_pretrain news_ft chonkie_semantic chonkie_neural, --levels block subheading
python scripts/run_clinical_transfer.py zero-shot --gpus 0
python scripts/run_clinical_transfer.py zero-shot --models news_ft --levels block --gpus 0

# Clinical fine-tuning on growing training sets: --initialization base news_pretrain news_ft
python scripts/run_clinical_transfer.py scaling --gpus 0 1
python scripts/run_clinical_transfer.py scaling --initialization base --gpus 0

# ── All stages in thesis order ────────────────────────────────────────────────
# Set GPU IDs and experiment groups in the script
bash scripts/run_ts_pipeline.sh --dry-run
bash scripts/run_ts_pipeline.sh

Using reproduced checkpoints

run_ts_pipeline.sh passes the new C4, news, and MIND checkpoints to their downstream stages. It uses one timestamp across runs/<stage>/reproductions/<time>/ and trains each shared model once.

MIND sentences, teacher predictions, subset files, and prepared splits go in the current MIND run's data/ folder. Clinical predictions and fine-tuning results go in zero_shot/ and scaling/ under the current clinical run.

For individual stage commands, the options below select reproduced checkpoints. Example output directories:

c4="$PWD/runs/boundary_pretraining/reproductions/my_run"
news="$PWD/runs/news_finetuning/reproductions/my_run"
mind="$PWD/runs/mind_adaptation/reproductions/my_run"
clinical="$PWD/runs/clinical_transfer/reproductions/my_run"

After boundary pre-training

The C4 run directory contains the final model and tokenizer. Use it for news fine-tuning and MIND adaptation with --c4-model:

python scripts/run_boundary_pretraining.py --gpus 0 1 --output-dir "$c4"
python scripts/run_news_finetuning.py --experiment backbones boundary_pretraining granularity objectives clinical_initialization \
    --c4-model "$c4" --output-dir "$news" --gpu 0

--experiment accepts several groups and deduplicates their shared runs.

After news fine-tuning

Use modernbert_c4_1024_ts as the MIND teacher and write its silver labels to a new directory:

python scripts/prepare_data.py mind --teacher-model "$news/modernbert_c4_1024_ts" \
    --output-dir "$mind/data" --gpu 0

--output-dir redirects all MIND preparation outputs; the raw articles still come from the configured dataset.

MIND adaptation

Train adaptations from the new C4 model and silver labels, then fine-tune them on news:

python scripts/run_mind_adaptation.py --experiment scale final objective_matrix matched_steps clinical_initialization \
    --c4-model "$c4" --prepared-root "$mind/data/prepared" --output-dir "$mind" --gpu 0

Each adaptation runs before the news models that depend on it. To run only news fine-tuning, use --from-adaptation DIR to reuse adaptations from that directory, or --recorded-adaptation to reuse the saved thesis adaptations.

News inference

Use the reproduced directories for all model references in an inference experiment:

python scripts/run_long_document_inference.py --experiment stage_matrix \
    --news-models-dir "$news" --mind-models-dir "$mind" --gpu 0

Clinical transfer

Use the same new checkpoints for zero-shot inference and clinical fine-tuning:

python scripts/run_clinical_transfer.py zero-shot --gpus 0 \
    --news-ft "$news/modernbert_c4_1024_full_sentences" \
    --news-pretrain "$mind/mind_25k_cssl_news_ts" --output-dir "$clinical"
python scripts/run_clinical_transfer.py scaling --gpus 0 1 \
    --news-ft "$news/modernbert_c4_1024_full_sentences" \
    --news-pretrain "$mind/mind_25k_cssl_news_ts" --output-dir "$clinical"

Use a new clinical output directory when changing checkpoints: scaling skips completed runs in an existing directory.

Direct inference

Pass a checkpoint directly and select the device:

python -m topic_segmentation.inference --model "$mind/mind_25k_full_news_ts" --input data/sample_input.jsonl --device cuda:0