Class SimpleNaiveBayesDocumentClassifier
java.lang.Object
org.apache.lucene.classification.SimpleNaiveBayesClassifier
org.apache.lucene.classification.document.SimpleNaiveBayesDocumentClassifier
- All Implemented Interfaces:
Classifier<BytesRef>
,DocumentClassifier<BytesRef>
public class SimpleNaiveBayesDocumentClassifier
extends SimpleNaiveBayesClassifier
implements DocumentClassifier<BytesRef>
A simplistic Lucene based NaiveBayes classifier, see
http://en.wikipedia.org/wiki/Naive_Bayes_classifier
-
Field Summary
FieldsFields inherited from class org.apache.lucene.classification.SimpleNaiveBayesClassifier
analyzer, classFieldName, indexReader, indexSearcher, query, textFieldNames
-
Constructor Summary
ConstructorsConstructorDescriptionSimpleNaiveBayesDocumentClassifier
(IndexReader indexReader, Query query, String classFieldName, Map<String, Analyzer> field2analyzer, String... textFieldNames) Creates a new NaiveBayes classifier. -
Method Summary
Modifier and TypeMethodDescriptionprivate void
analyzeSeedDocument
(Document inputDocument, Map<String, List<String[]>> fieldName2tokensArray, Map<String, Float> fieldName2boost) This methods performs the analysis for the seed document and extract the boosts if present.assignClass
(Document document) Assign a class (with score) to the givenDocument
private List
<ClassificationResult<BytesRef>> assignNormClasses
(Document inputDocument) private double
calculateLogLikelihood
(String[] tokenizedText, String fieldName, Term term, int docsWithClass) private double
calculateLogPrior
(Term term, int docsWithClassSize) private int
getClasses
(Document document) Get all the classes (sorted by score, descending) assigned to the givenDocument
.getClasses
(Document document, int max) Get the firstmax
classes (sorted by score, descending) assigned to the given text String.private double
getTextTermFreqForClass
(Term term, String fieldName) Returns the average number of unique terms times the number of docs belonging to the input classprotected String[]
getTokenArray
(TokenStream tokenizedText) Returns a token array from theTokenStream
in inputprivate int
getWordFreqForClass
(String word, String fieldName, Term term) Returns the number of documents of the input class ( from the whole index or from a subset) that contains the word ( in a specific field or in all the fields if no one selected)Methods inherited from class org.apache.lucene.classification.SimpleNaiveBayesClassifier
assignClass, assignClassNormalizedList, countDocsWithClass, getClasses, getClasses, normClassificationResults, tokenize
-
Field Details
-
field2analyzer
Analyzer
to be used for tokenizing document fields
-
-
Constructor Details
-
SimpleNaiveBayesDocumentClassifier
public SimpleNaiveBayesDocumentClassifier(IndexReader indexReader, Query query, String classFieldName, Map<String, Analyzer> field2analyzer, String... textFieldNames) Creates a new NaiveBayes classifier.- Parameters:
indexReader
- the reader on the index to be used for classificationquery
- aQuery
to eventually filter the docs used for training the classifier, ornull
if all the indexed docs should be usedclassFieldName
- the name of the field used as the output for the classifier NOTE: must not be heavely analyzed as the returned class will be a token indexed for this fieldtextFieldNames
- the name of the fields used as the inputs for the classifier, they can contain boosting indication e.g. title^10
-
-
Method Details
-
assignClass
Description copied from interface:DocumentClassifier
Assign a class (with score) to the givenDocument
- Specified by:
assignClass
in interfaceDocumentClassifier<BytesRef>
- Parameters:
document
- aDocument
to be classified. Fields are considered features for the classification.- Returns:
- a
ClassificationResult
holding assigned class of typeT
and score - Throws:
IOException
- If there is a low-level I/O error.
-
getClasses
Description copied from interface:DocumentClassifier
Get all the classes (sorted by score, descending) assigned to the givenDocument
.- Specified by:
getClasses
in interfaceDocumentClassifier<BytesRef>
- Parameters:
document
- aDocument
to be classified. Fields are considered features for the classification.- Returns:
- the whole list of
ClassificationResult
, the classes and scores. Returnsnull
if the classifier can't make lists. - Throws:
IOException
- If there is a low-level I/O error.
-
getClasses
public List<ClassificationResult<BytesRef>> getClasses(Document document, int max) throws IOException Description copied from interface:DocumentClassifier
Get the firstmax
classes (sorted by score, descending) assigned to the given text String.- Specified by:
getClasses
in interfaceDocumentClassifier<BytesRef>
- Parameters:
document
- aDocument
to be classified. Fields are considered features for the classification.max
- the number of return list elements- Returns:
- the whole list of
ClassificationResult
, the classes and scores. Cut for "max" number of elements. Returnsnull
if the classifier can't make lists. - Throws:
IOException
- If there is a low-level I/O error.
-
assignNormClasses
private List<ClassificationResult<BytesRef>> assignNormClasses(Document inputDocument) throws IOException - Throws:
IOException
-
analyzeSeedDocument
private void analyzeSeedDocument(Document inputDocument, Map<String, List<String[]>> fieldName2tokensArray, Map<String, throws IOExceptionFloat> fieldName2boost) This methods performs the analysis for the seed document and extract the boosts if present. This is done only one time for the Seed Document.- Parameters:
inputDocument
- the seed unseen documentfieldName2tokensArray
- a map that associated to a field name the list of token arrays for all its valuesfieldName2boost
- a map that associates the boost to the field- Throws:
IOException
- If there is a low-level I/O error
-
getTokenArray
Returns a token array from theTokenStream
in input- Parameters:
tokenizedText
- the tokenized content of a field- Returns:
- a
String
array of the resulting tokens - Throws:
IOException
- If tokenization fails because there is a low-level I/O error
-
calculateLogLikelihood
private double calculateLogLikelihood(String[] tokenizedText, String fieldName, Term term, int docsWithClass) throws IOException - Parameters:
tokenizedText
- the tokenized content of a fieldfieldName
- the input field nameterm
- theTerm
referring to the class to calculate the score ofdocsWithClass
- the total number of docs that have a class- Returns:
- a normalized score for the class
- Throws:
IOException
- If there is a low-level I/O error
-
getTextTermFreqForClass
Returns the average number of unique terms times the number of docs belonging to the input class- Parameters:
term
- the class term- Returns:
- the average number of unique terms
- Throws:
IOException
- If there is a low-level I/O error
-
getWordFreqForClass
Returns the number of documents of the input class ( from the whole index or from a subset) that contains the word ( in a specific field or in all the fields if no one selected)- Parameters:
word
- the token produced by the analyzerfieldName
- the field the word is coming fromterm
- the class term- Returns:
- number of documents of the input class
- Throws:
IOException
- If there is a low-level I/O error
-
calculateLogPrior
- Throws:
IOException
-
docCount
- Throws:
IOException
-