`_extract_representative_docs`: duplicate sampling (`replace=True`) and text-based index mapping select wrong documents

[pidefrem]

Describe the bug

Two related bugs in BERTopic._extract_representative_docs() (bertopic/_bertopic.py) affect the correctness of representative_docs_. They are independent but live in the same method, so I'm reporting them together and will fix both in a single PR.

Bug 1 — sampling with replace=True produces duplicate representative documents

documents_per_topic = (
    documents.drop("Image", axis=1, errors="ignore")
    .groupby("Topic")
    .sample(n=nr_samples, replace=True, random_state=42)
    .drop_duplicates()
)

When a topic has fewer than nr_samples unique documents (common for small topics), replace=True can draw the same document multiple times. Those duplicates are fed into the c-TF-IDF similarity calculation, inflating scores and producing duplicate entries in representative_docs_. The trailing .drop_duplicates() only removes exact duplicate rows across the whole frame after sampling — it does not stop the same document being drawn repeatedly within a single topic.

Bug 2 — text-based in matching maps documents to the wrong topic

doc_ids = [selected_docs_ids[index] for index, doc in enumerate(selected_docs) if doc in docs]

The selected documents are mapped back to their original indices via text membership testing. When the same document text appears in more than one topic (short texts, boilerplate, near-duplicates), doc in docs matches the first occurrence regardless of which topic the document actually belongs to. This causes:

• documents skipped entirely (their index never matches for the right topic),
• misaligned doc_ids ↔ selected_docs pairs (wrong similarity scores assigned to wrong documents),
• representative_docs_ containing documents that don't belong to their assigned topic.

Reproduction

Bug 1 — duplicate representative docs:

from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups

docs = fetch_20newsgroups(subset="all", remove=("headers", "footers", "quotes"))["data"][:500]
topic_model = BERTopic(min_topic_size=5)
topics, _ = topic_model.fit_transform(docs)

for topic_id, docs_list in topic_model.representative_docs_.items():
    if len(docs_list) != len(set(docs_list)):
        print(f"Topic {topic_id}: {len(docs_list)} docs, {len(set(docs_list))} unique")

Bug 2 — documents mapped to the wrong topic:

from bertopic import BERTopic

docs = [
    "machine learning is great",   # shared text
    "deep learning neural networks",
    "machine learning is great",   # same text, different topic
    "topic modeling with BERTopic",
]
topic_model = BERTopic(min_topic_size=2)
topics, _ = topic_model.fit_transform(docs)
# representative_docs_ may contain docs mapped to the wrong topic

Proposed fix

• Sample without replacement, capped at the topic's unique-document count (n=min(nr_samples, len(x)), replace=False), and de-duplicate per (Topic, Document) before sampling.
• Track the positional indices returned by the similarity/MMR selection and use those to look up doc_ids, instead of text membership testing.

Your contribution

I have both fixes ready in my fork with tests, and will open a PR that closes this issue.

BERTopic Version

0.17.4

---

Supersedes #2491 and #2492, which are being closed in favour of this consolidated issue + PR.

[pidefrem]

PR opened: #2496 (branch pidefrem:fix/repr-docs-sampling-and-indexing). It fixes both bugs described here and adds 11 tests.