Skip to content

Make page classification less expensive during crawls - #401

Merged
KingAkeem merged 2 commits into
devfrom
improve/nlp-classifier-runtime
Jul 28, 2026
Merged

Make page classification less expensive during crawls#401
KingAkeem merged 2 commits into
devfrom
improve/nlp-classifier-runtime

Conversation

@KingAkeem

Copy link
Copy Markdown
Member

What changed

This cleans up the NLP classifier so crawling more than one page does not retrain the model over and over. The classifier now trains lazily from the packaged CSV, stays cached for the process, and returns a confidence score for the selected category instead of the old misleading accuracy value.

I also removed the runtime training_data/ generation from classification and stopped the NLP helpers from changing the process working directory. The optional data export helper is still there for experiments, but normal crawls no longer write generated files into the package directory.

Version is bumped to 4.3.0 and the changelog has the release notes.

Checks

  • .venv/bin/python -m pytest tests/test_nlp.py
  • .venv/bin/python -m pytest
  • .venv/bin/python -m flake8 src/torbot/modules/nlp/main.py src/torbot/modules/nlp/gather_data.py tests/test_nlp.py
  • Real classifier smoke test against the packaged CSV

@KingAkeem
KingAkeem merged commit 4863c44 into dev Jul 28, 2026
12 checks passed
@KingAkeem
KingAkeem deleted the improve/nlp-classifier-runtime branch July 28, 2026 22:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant