Neo4j-based recommendation engine module with real-time and pre-computed recommendations.
Neo4j-based recommendation engine module with real-time and pre-computed recommendations.
As of May 2021, this repository has been retired.
GraphAware Neo4j Recommendation Engine is a library for building high-performance complex recommendation engines atop Neo4j. It is in production at a number of GraphAware's clients producing real-time recommendations on graphs with hundreds of millions of nodes.
Key Features:
The library imposes a specific recommendation engine architecture, which has emerged from our experience building recommendation engines on top of Neo4j. In return, it offers high performance and handles most of the plumbing so that you only write the recommendation business logic specific to your use case.
Besides computing recommendations in real-time, it also allows for pre-computing recommendations that are perhaps too complex to compute in real-time. The pre-computing happens on best-effort basis during quiet periods, so that it does not interfere with regular transaction processing that your Neo4j database is performing.
This open-source (GPL) version of the module is compatible with GraphAware Framework Community (GPL), which in turn is compatible with Neo4j Community Edition (GPL) only. It will not work with Neo4j Enterprise Edition, which is a proprietary and commercial software product of Neo4j, Inc..
GraphAware offers an Enterprise version of the GraphAware Framework to licensed users of Neo4j Enterprise Edition. Please get in touch to receive access.
When using Neo4j in the standalone server mode,
you will need the GraphAware Neo4j Framework and GraphAware Neo4j Recommendation Engine .jar files (both of which you can download here) dropped
into the plugins directory of your Neo4j installation.
Unlike with other GraphAware Framework Modules, you will need to write at least a few lines of your own Java code (read on).
Java developers that use Neo4j in embedded mode and those developing Neo4j server plugins, unmanaged extensions, GraphAware Runtime Modules, or Spring MVC Controllers can include the module as a dependency for their Java project.
Releases are synced to Maven Central repository. When using Maven for dependency management, include the following dependency in your pom.xml and edit the version number.
...
com.graphaware.neo4j
recommendation-engine
A.B.C.D.E
...To use the latest development version, just clone this repository, run mvn clean install and change the version in the
dependency above to A.B.C.D.E-SNAPSHOT.
The version number has two parts. The first four numbers indicate compatibility with Neo4j GraphAware Framework. The last number is the version of the Recommendation Engine library. For example, version 2.1.6.26.1 is version 1 of the Recommendation Engine compatible with GraphAware Neo4j Framework 2.1.6.26.
The purpose of a recommendation engine is (unsurprisingly) to recommend something to users. This could be products they should buy, users they should connect with, artists they should follow, etc. It turns out graph is a really good data structure for representing users' interests, behaviours, and other characteristics that might be useful for finding recommendations. More importantly, graph databases, and Neo4j especially, provide a natural way of expressing queries on this data in order to find relevant recommendations, and executing these queries very fast.
There are three main challenges when building a recommendation engine. The first is to discover the items to recommend. The second is to choose the most relevant ones to present to the user. Finally, the third challenge is to find relevant recommendations as quickly as possible. Preferably, this should happen in real-time, i.e. using the most up to date information we have. The last thing we want to do is to recommend something the user has already purchased, or a person we know she isn't interested in.
The first two points above are business rather than technical challenges. Typically, when you start building a recommendation engine, you have some idea about how the recommended items will be discovered. For instance, you might want to recommend items that other people with similar interests have bought. You also know, which items you absolutely do not want to recommend, for example, items the user has already purchased, or people that we know are married as a potential match for a date.
The issue with recommendation relevance is usually something that needs to be experimented with. When building the first recommendation engine, or perhaps even a proof of concept, one feature that shouldn't be missing is the ability to configure how the recommendation relevance is computed and, perhaps more importantly, measure how users react to recommendations produced by different relevance-computing configurations.
Finally, let's address the issue of speed, which is of a technical nature. When serving real-time recommendations, users shouldn't need to wait for more than, let's say, a couple hundred milliseconds. With Neo4j, we will be able to build many different recommendation queries that take milliseconds to execute. However, there are situations (large graphs with some very dense nodes) where we will need to take extra care in order not to slow the recommendation process down. Finally, in situations where the recommendation logic and the size of the graph simply don't allow real-time computation, we will need to look at pre-computing some recommendations, whilst avoiding the dangers of serving out of date recommendations.
The architecture of GraphAware Neo4j Recommendation Engine has been designed to address, or easily allow you to address, all of the above challenges. The library works with the following concepts:
A Recommendation Engine is a component that produces Recommendations, given an Input. Whilst the architecture is
generic enough to support other persistence mechanisms, we focus on Neo4j and so the input will typically be a Neo4j Node
representing a user for whom we want to find recommendations, a product for which we want to find buyers, etc.
A RecommendationEngine, as in the case of DelegatingRecommendationEngine can be composed of other RecommendationEngines
that it delegates to. Usually, however, a RecommendationEngine will encapsulate the querying and relevance-computing
logic for discovering recommendations based on a single logical criterion. Such engine typically extends
SingleScoreRecommendationEngine. For example, we could have one engine that discovers items a user may want to buy based
on what other users with similar tastes have bought. Another engine would discover items based on user's expressed
preferences. Yet another one could discover items to be recommended based on what is currently trending.
For performance reasons as well as to achieve good encapsulation, RecommendationEngines are only concerned with discovering
and scoring all potential recommendations, without caring about the fact that some recommendations discovered this way may
not be suitable, perhaps because the user has already purchased the discovered item. Removing irrelevant recommendations
will be discussed shortly.
Recommendations are a collection of tuples/pairs, where each pair is composed of a recommended item (again, typically a Node)
and associated relevance Score. The Score is composed of named Partial Scores. Each Partial Score has a float
value and optionally some extra details about how and why it has been computed that we with to expose to the users. Typically,
a single SingleScoreRecommendationEngine, as the name suggest, is responsible for a single Partial Score.
When an item has been discovered as a potential recommendation by multiple SingleScoreRecommendationEngines, its Parial Scores
will be tallied by the Score object. For example, an item that is currently trending and matches the user's preferred
tastes will have a total relevance Score composed of two Partial Scores, one due to the fact that it is trending, and
another one because it is a preferred item.
In some cases, an item might be discovered multiple times by the same SingleScoreRecommendationEngine. For example,
we may have an engine that suggests people a user should be friends with based on the fact that they have some friends in
common. Assuming an easy-to-imagine graph traversal that discovers these recommendations, a potential friend will be discovered
three times if he has three friends in common with the user we're computing recommendations for. However, each additional
friend in common might not bear the same relevance for the recommendation. Thus, each Partial Score can have a
Score Transformer applied to it. A ScoreTransformer can apply an arbitrary mathematical function to the Partial Score
computed by a SingleScoreRecommendationEngine.
Recommendations are always computed within a Context. Whilst each recommendation-computing process for a single input
might involve multiple RecommendationEngines and other components, there is usually a single Context per computation
that encapsulates information relevant to the process. For example, the Context knows whether a potential
recommendation discovered by a RecommendationEngine is allowed
to be served to the user. For each computation, a new Context is produced by TopLevelRecommendationEngine.
The Context also encapsulates a Config for each recommendation-computing process. This is a set of user-defined values.
By default, a Config knows how many recommendations should be produce and what is the maximum time the
recommendation-computing process should take. Optionally, arbitrary key-value pairs can be passed in, which is useful
for scenarios when score values, rewards, penalties, and other variables should not be hard-coded and differ per computation.
Rather than requiring all RecommendationEngines to know how to detect irrelevant recommendations (thus slowing the computation
down and scattering a single concern), the logic is centralised into Blacklist Builders and Filters. BlacklistBuilders,
as the name suggests, are responsible for building "blacklists" of items that must not be recommended for a given input.
Assuming that the input is a Node representing a person, an example of a BlacklistBuilder could be AlreadyPurchasedItems
which builds a blacklist of items that the person has already purchased. BlacklistBuilders are most efficient in situations
where a small number of irrelevant recommendations (let's say up to 100) can be discovered with a single query before the
recommendation process begins.
Filters, on the other hand, can tell whether a recommendation is relevant or not by looking at the recommendation
itself once it has been discovered. An obvious example of a Filter could be a class called ExcludeSelf, which would
make sure that (for example)
No open issues yet, or sync has not completed.