Ask HN: How important is document category recognition in data classification?

A Hacker News user asks whether document category recognition (e.g., technical manual, customer list, employee record) is important for data classification, noting that regex-based classification of IDs and numbers leads to false positives and negatives. The user is building a non-AI, light model and embedding-based classifier to semantically understand documents and seeks opinions on the usefulness of adding document category recognition.

Classifying files based solely on presence of IDs, numbers and other regex patterns, from experience, leads to much false positives and negatives. A technical manual document with a 9 digit number could trigger a SSN pattern check and get wrongly classified as having personal data /Confidential. Personal data on a HR document would be treated differently from personal data on a customer list. Would identifying the document category tech manual, customer list, employee record be a good additional layer to classifier logic? I'm trying to build a non-AI, light model and embedding based classifier that can semantically understand documents, and like opinions on how useful such a thing would be. Comments URL: https://news.ycombinator.com/item?id=49280386 https://news.ycombinator.com/item?id=49280386 Points: 1 Comments: 0