Showing posts with label Solr. Show all posts
Showing posts with label Solr. Show all posts

Tuesday, January 20, 2015

Solr Custom Score



Using CustomScoreQuery For Custom Solr/Lucene Scoring
When you implement your own Lucene query, you’re taking control of two things:

Matching – what documents should be included in the search results
Scoring – what score should be assigned to a document (and therefore what order should they appear in)



  1. You’ve utilized Solr’s extensive set of query parsers & features including function queries, joins, etc. None of this solved your problem
  2. You’ve exhausted the ecosystem of plugins that extend on the capabilities in (1). That didn’t work.
  3. You’ve implemented your own query parser plugin that takes user input and generates existing Lucene queries to do this work. This still didn’t solve your problem.
  4. You’ve thought carefully about your analyzers – massaging your data so that at index time and query time, text lines up exactly as it should to optimize the behavior of existing search scoring. This still didn’t get what you wanted.
  5. You’ve implemented your own custom Similarity that modifies how Lucene calculates the traditional relevancy statistics – query norms, term frequency, etc.
  6. You’ve tried to use Lucene’s CustomScoreQuery to wrap an existing Query and alter each documents score via a callback. This still wasn’t low-level enough for you, you needed even more control.
http://dev.fernandobrito.com/2012/10/building-your-own-lucene-scorer/
Query query = parser.parse("searching something");
 
CustomScoreQuery customQuery = new MyOwnScoreQuery(query);
 
ScoreDoc[] hits = searcher.search(customQuery.createWeight(searcher), null, numHits).scoreDocs;



Thursday, July 31, 2014

Learning Lucene

Indexing:
Directory dir = FSDirectory.open(new File(indexPath));
Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_4_9);
IndexWriterConfig iwc = new IndexWriterConfig(Version.LUCENE_4_9,
analyzer);
if (create) {
// Create a new index in the directory, removing any
// previously indexed documents:
iwc.setOpenMode(OpenMode.CREATE);
} else {
// Add new documents to an existing index:
iwc.setOpenMode(OpenMode.CREATE_OR_APPEND);
}
// iwc.setRAMBufferSizeMB(256.0);
IndexWriter writer = new IndexWriter(dir, iwc);
Document doc = new Document();
doc.add(new LongField("modified", file.lastModified(),Field.Store.NO));
writer.addDocument(doc);
writer.updateDocument(new Term("path", file.getPath()),doc);

Searching
IndexReader reader = DirectoryReader.open(FSDirectory.open(new File(index)));
IndexSearcher searcher = new IndexSearcher(reader);
Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_4_9);
QueryParser parser = new QueryParser(Version.LUCENE_4_9, field,analyzer);
Query query = parser.parse(line);
searcher.search(query, null, 100);

TopDocs results = searcher.search(query, 5 * hitsPerPage);
ScoreDoc[] hits = results.scoreDocs;
int numTotalHits = results.totalHits;
Syste
QParser parser = QParser.getParser(rb.getQueryString(), defType, req);
Query q = parser.getQuery();

SolrIndexSearcher.QueryCommand cmd = rb.getQueryCommand();
cmd.setTimeAllowed(timeAllowed);
searcher.search(result,cmd);

SolrPluginUtils

How solr shrds query works:
org.apache.solr.handler.component.QueryComponent.handleRegularResponses(ResponseBuilder, ShardRequest)

org.apache.solr.handler.component.QueryComponent.mergeIds(ResponseBuilder, ShardRequest)
// id to shard mapping, to eliminate any accidental dups
HashMap<Object,String> uniqueDoc = new HashMap<Object,String>();  

// Merge the docs via a priority queue so we don't have to sort *all* of the
// documents... we only need to order the top (rows+start)
ShardFieldSortedHitQueue queue= new ShardFieldSortedHitQueue(sortFields, ss.getOffset() + ss.getCount());
// this is a min heap

ShardDoc shardDoc = new ShardDoc();
shardDoc.id = id;
shardDoc.shard = srsp.getShard();
shardDoc.orderInShard = i;

String path = doc.get("path");