Solr 5 is on it’s way and along with it comes a shiny new implementation of Collapse & Expand. The general functionality remains the same but the performance characteristics have changed. This blog provides a guide to getting the best … Continued
Noggit is the world’s fastest streaming JSON parser for Java. Noggit is the streaming JSON parser used in Solr. It lives here on github. JSON features and extensions Noggit supports a number of extensions to the JSON grammar. All of … Continued
Although query parameters are often an easy method to create a Heliosearch/Solr requests by hand, they have a number of drawbacks: Inherently un-structured, requiring unsightly parameters like f.facet_name.facet.range.start=5 Inherently un-typed… everything is a string. More difficult to decipher large requests. … Continued
Lucene/Solr trunk (the future 6.0 release) is now on Java8, while version 5.x is still on Java7. Linux and Windows allows one to install a JDK any place in the filesystem, and I use the convention of installing in /opt/jdk7 … Continued
Several weeks ago Heliosearch v.08 was released, which included an early release of the Streaming Aggregation API. One of the main design goals for the Streaming API was to provide the foundation for the Heliosearch SQL Sub-Project. The Streaming API … Continued
Introduction Facet Types Testing Using Curl Terms Facet Query Facet Range Facet Related page: Facet Functions Related page: Sub-Facets Introduction Heliosearch has a completely re-written faceted search module with a structured JSON API to control the faceting commands. The structured … Continued
Traditional faceted search (also called guided navigation) involves counting search results that belong to categories (also called facet constraints). The new facet functions in Heliosearch/Solr extends normal faceting by allowing additional aggregations on document fields themselves. Combined with the new … Continued
Subfacets (also called Nested Facets) is a more generalized form of Solr’s current pivot faceting that allows adding additional facets for every bucket produced by a parent facet. Subfacet advantages over pivot faceting: Subfacets work with facet functions (statistics), enabling … Continued
Introduction Tuples, TupleStreams and Sorted Streams SolrStream Stream Decorators Streaming Set Operations Streaming Aggregation Distributed Searching, Sorting and Streaming: CloudSolrStream A Bottleneck Rears It’s Ugly Head! Distributed Aggregation: ParallelStream Scaling-up With SolrCloud Replicas Introduction This blog describes the new … Continued
Solr 4.10 and Heliosearch .07 have added a terms query (or terms filter) to more efficiently match many terms in a single field. A large number of terms are often useful for things like access control lists or security filters. … Continued
Native code faceting for Solr has just been added to Heliosearch, and benchmarks show an impressive 2x performance increase! This is faceting code written in C++ and statically compiled for maximum performance, and loaded into the JVM via JNI (Java … Continued
Solr needs a flexible cross-datacenter architecture that can handle both a variety of application needs as well as a variety of infrastructure resources. Design Goals Accommodate 2 or more data centers Accommodate active/active uses Accommodate limited band-with cross-datacenter connections Minimize … Continued
Starting with Solr 4.10, Solr has the ability to sort and export entire result sets. This new feature allows Solr to be used as a general purpose distributed sorting engine. Use Cases This new capability opens up Solr for a … Continued
The filter caching features in Solr allow for precise control over how filter queries are handled in order to maximize performance. Solr has the ability to specify if a filter is cached, specify the order filters are evaluated, and specify … Continued
Lucene/Solr background Lucene has a segmented architecture – when a small amount of documents are added to an existing index, this will often just add an additional small segment to the index. Caching data structures at the segment level (e.g. … Continued
Solr 4.9 unleashes Solr’s uniquely powerful Custom Analytics Engine. In recent blogs I wrote about the nuts and bolts of the new AnalyticsQuery API and MergeStrategy API. This blog is a high level look at that power that these API’s … Continued
In the last couple of posts, I introduced the RankQuery API and the AnalyticsQuery API. The RankQuery API allows developers to plug-in custom ranking algorithms to Solr and the AnalyticsQuery API allows developers to plug-in custom analytic logic to Solr. … Continued
I’ve often seen mistaken descriptions of Solr as just “a http wrapper around Lucene”. Unfortunately that mischaracterization was never nipped in the bud early enough and has continued to be repeated in many places such as press articles (where it … Continued
In Solr 4.9 there is a new AnalyticsQuery API that allows developers to plug custom analytic logic into Solr. The AnalyticsQuery class provides a clean and simple API that gives developers access to all the rich functionality in Lucene and … Continued
In my last blog, I discussed Solr’s new RankQuery capability, which allows developers to take full control of the ranking process. The new ReRankingQParserPlugin leverages the RankQuery framework to hook in Lucene’s new QueryRescorer. Query Re-Ranking Explained: Solr offers a … Continued
Coming in Solr 4.9 is a new RankQuery API. Before diving into how the RankQuery API works, I’ll give a little background into how ranking works in Lucene/Solr. A Lucene search can have three parts to it: 1) A Query: … Continued
Macro Expansion is a new Heliosearch feature that does parameter substitution across all request parameters. The macro expansion is done at the same point in time that default parameters are applied (i.e. when the request reaches the correct solr request … Continued
Solr 4.8 has been released. Here’s an overview of how to use some of the new features. Complex Phrase Queries The complexphrase query parser can produce phrase queries with embedded wildcards and boolean queries. It works via multiple passes, parsing … Continued
Step 1: Download HDS Apache Solr by default ships with a Jetty based Solr server in the “example” directory, but many people prefer Tomcat. Configuring your own Tomcat server can be daunting, as can be seen by the large list … Continued
In my last blog, I explained how to use the ExpandComponent to expand the groups that were collapsed with the CollapsingQParserPlugin. In this blog I’ll explain how to expand the parent/child relationship of a Solr block join. First the data: … Continued
Coming in Solr 4.8 is a new search component called the ExpandComponent. The ExpandComponent can be used to expand parent/child relationships in Solr. This blog describes how to use the ExpandComponent to expand the groups that were collapsed by the … Continued
This post will outline a method for assigning specific shard replicas to specific nodes using the Solr Collections API and, as a bonus using the Solr Core Admin API. I’m assuming basic familiarity with Solr and SolrCloud. How it works … Continued
Heliosearch’s off-heap FieldCache was previously introduced and benchmarked for integer fields. Support for all numeric field types as well as string fields has now been completed, and this post will focus on the performance of string fields. A review of … Continued
Solr 4.7 has been released! Here’s a slightly more in-depth overview of some selected features. Deep Paging Both single node, and distributed deep paging have been added to Solr! I previously created an example of how to use Solr’s deep … Continued
nCache Heliosearch has a new replacement for the Lucene FieldCache currently used by Solr for sorting, faceting, and function queries. Introducing nCache (n is for “native”): nCache has Off-Heap Data-structures, just like the Off-Heap Filters to lower garbage collection … Continued
Off-Heap Native Filters is the first feature we added to Heliosearch, a new open source project designed to bring Solr performance to the next level. Big JVM heaps can be Big Trouble JVMs have never been good at dealing with … Continued
Heliosearch – The Next Solr Project Heliosearch is about creating the next major evolution of Solr. To kick things off, I’ve started a project on github and implemented “off-heap” filters, as a first step toward taking performance to the next … Continued
Solr has had full featured Field Collapsing and Grouping since 2010. The CollapsingQParserPlugin, which was recently committed, provides an alternative approach with more specific design goals. Design Goals The initial CollapsingQParserPlugin requirements were driven by a Solr ECommerce user. The … Continued
Sometimes a small programming change can have a big impact on functionality. This is the case with SOLR-5541, which allows the QueryElevationComponent to accept elevate and exclude Id’s as http parameters. Now you have complete control over the most important … Continued
In Solr 4.5, client side document routing was added to CloudSolrServer. This feature routes document updates to the correct shard leader. This is the default behavior for CloudSolrServer so you don’t have to do anything to turn it on. Just … Continued
Background I needed a really good hash function for the distributed indexing we’re implementing for Solr. Since it will be used for partitioning documents, it needed to be really high quality (well distributed) since we don’t want uneven shards. It … Continued
Solr took another step toward increasing it’s NoSQL datastore capabilities, with the addition of realtime get. Background As readers probably know, Lucene/Solr search works off of point-in-time snapshots of the index. After changes have been made to the index, a … Continued
Lucene’s default ranking function uses factors such as tf, idf, and norm to help calculate relevancy scores. Solr has now exposed these factors as function queries. docfreq(field,term) returns the number of documents that contain the term in the field. termfreq(field,term) … Continued
I previously introduced Solr’s Result Grouping, also called Field Collapsing, that limits the number of documents shown for each “group”, normally defined as the unique values in a field or function query. Since then, there have been a number of … Continued
Solr has been able to produce JSON results for a long time, by adding wt=json to any query. A new capability has recently been added to allow indexing in JSON, as well as issuing other update commands such as deletes … Continued
Result Grouping, also called Field Collapsing, has been committed to Solr! This functionality limits the number of documents for each “group”, usually defined by the unique values in a field (just like field faceting). You can think of it like … Continued
Solr 1.4 contains a new feature that allows range queries or range filters over arbitrary functions. It’s implemented as a standard Solr QParser plugin, and thus easily available for use any place that accepts the standard Solr Query Syntax by … Continued
One of the many performance improvements in the upcoming Solr 1.4 release involves improved filtering performance. Solr 1.4 filters are both faster (anywhere from 30% to 80% faster to calculate intersections, depending on configuration), take less memory (40% smaller), and … Continued
With CPU cores constantly increasing, there has been some major work done in Lucene/Solr to increase the scalability under multi-threaded load. Read-only IndexReaders One bottleneck was synchronization around the checking of deleted docs in a Lucene IndexReader. Since another thread … Continued
Having performance issues with Solr’s faceted search and certain types of fields? Help has arrived in the form of a new Solr faceting algorithm! This new faceting implementation dramatically improves the performance of faceted search, making it suitable for a … Continued
I was surprised to discovered that there isn’t a good cross-platform hash function defined for strings. MD5, SHA, FVN, etc, all define hash functions over bytes, meaning that it’s under-specified for strings. So I set out to create a standard … Continued
A new chapter in Solr scalability has been opened with the addition of distributed search! http://wiki.apache.org/solr/DistributedSearch Distributed Search splits an index into multiple shards, and queries across all the shards, combining the results and presenting a single merged response that … Continued
I’ll be giving a Solr presentation Nov 8th in Berlin, titled “Add Powerful Full Text Search to Your Web App with Solr“. Should be fun, just wish I had more free time while in Berlin…
