Distributed Recommender System Using Case Based Reasoning

Abstract

Recommender systems apply knowledge discovery techniques to the problem of making personalized recommendations for information, products or services during a live inter- action. These systems, especially the collaborative filtering based ones, are achieving widespread success on the Web. The tremendous growth in the amount of available infor- mation and the number of visitors to Web sites in recent years poses some key challenges for recommender systems.

These are: producing high quality recommendations, per- forming many recommendations per second for millions of users and items and achieving high coverage in the face of data sparsity. In traditional collaborative filtering systems the amount of work increases with the number of participants in the system. New rec- ommender system technologies are needed that can quickly produce high quality recom- mendations, even for very large-scale problems and sparse data. To address these issues we have explored item-based collaborative filtering and case based reasoning techniques. Item-based techniques first analyze the user-item matrix to identify relationships between different items, and then use these relationships to indirectly compute recommendations for users. The user-item matrix is formed after the data sparsity is addressed by using case based reasoning technique.

In this paper we analyze the method of item-based recommendation generation and applying case based reasoning on it. We manually evaluate our results and compare them to the basic approach without using case based reasoning. Our experiments suggest that case based reasoning algorithm applied recommendations provide better user satisfying recommendations than the basic approach.

Introduction

The amount of information in the world is increasing far more quickly than our ability to process it. All of us have known the feeling of being overwhelmed by the number of new books, journal articles, and conference proceedings coming out each year. Technology has dramatically reduced the barriers to publishing and distributing information. Now it is time to create the technologies that can help us shift through all the available information to and that which is most valuable to us.

One of the most promising such technologies is a recommendation system. A product recommendation system is basically a filtering system that seeks to predict and shows the items that a user would like to purchase. Recommender systems represent services that aim at predicting a users interest on information items available in the application domain, using users ratings on items. It may not be entirely accurate, but if it shows you what you like then it is doing its job right. It has become increasing popular in recent years, and are utilized in a variety of areas including movies, health care social platform etc., It is due to the benefits it offers like:-

• Driving traffic
• Target Marketing
• Offer Advice and Direction
• Provide Reports and Boost Metrics etc.,

However, there remain important research questions in overcoming two fundamental chal- lenges for collaborative filtering recommender systems. The first challenge is to improve the quality of the recommendations for the users. Users need recommendations they can trust to help them find items they will like. Users will “vote with their feet” by refusing to use recommender systems that are not consis- tently accurate for them.

The second challenge is to improve the scalability of the collaborative filtering algo- rithms. These algorithms are able to search tens of thousands of potential neighbors in real-time, but the demands of modern systems are to search tens of millions of potential neighbors. Further, existing algorithms have performance problems with individual users for whom the site has large amounts of information. For instance, if a site is using brows- ing patterns as indications of content preference, it may have thousands of data points for its most frequent visitors. These user rows slow down the number of neighbors that can be searched per second, further reducing scalability. In some ways these two challenges are in conflict, since the less time an algorithm spends searching for neighbors, the more scalable it will be, and the worse its quality. For this reason, it is important to treat the two challenges simultaneously so the so- lutions discovered are both useful and practical. In this paper, we address these issues of recommender systems by applying case based reasoning and scaling the system using distributed algorithms.

Literature Review

In practice, recommendation engines are of three kinds: Popularity-based engines: Usually the most simple to implement, also the most imper- sonal. Recommending the top rated or most popular items.

Content-based engines: Content-based engines are based on a description of the item and a profile of the users preferences. These methods are best suited to situations where there is known data on an item (name, location, description, etc.), but not on the user. Content-based recommenders treat recommendation as a user-specific classification prob- lem and learn a classifier for the user’s likes and dislikes based on product features. Collaborative filtering engines: Collaborative filtering is based on the assumption that people who agreed in the past will agree in the future, and that they will like similar kinds of items as they liked in the past. The system generates recommendations using only in- formation about rating profiles for different users or items. By locating peer users/items with a rating history similar to the current user or item, they generate recommendations using this neighborhood.

Every method has different effectiveness and accuracy regarding recommendation based on the applying areas and the activity levels. The major problem which the recommendation system is still facing is data sparsity. It usually arises from the phenomenon that users in general rate only a limited number of items. The recommendations are carried on a movie data sets. This will lead to generate unreasonable recommendation for those users who provide no rating. The major problem that is being addressed is to deal with data sparsity and making the system distributed in order to scale it to many users.

Methodology/Design

The predictions are made by recommendation engine using the principle of collaborative filtering. Collaborative filtering: This filtering method is usually based on collecting and ana- lyzing information on users behaviours, their activities or preferences and predicting what they will like based on similarity with other users. Collaborative Filtering simply put uses the “wisdom of the crowd” to recommend items. Item based collaborative filtering uses the patterns of users who liked the same movie as me to recommend me a movie (users who liked the movie that I like, also liked these other movies). Recommendation based on user’s input of any movie present in the data set is done.

In order to solve the data sparsity issue, Case Based Reasoning is applied. Case Based Reasoning in layman terms is nothing but solving a new problem by remembering a pre- vious similar situation and by reusing information and knowledge of that situation. For example: For a physician who is examining a particular patient who has similar symptoms to a previous patient. He can use the same diagnosis and treatment of the previous patient to determine the disease and treatment for the patient in front of him. At the highest level of generality, a general Case Based Reasoning cycle may be de- scribed by four process.

• Retrieve the most similar cases.

– For a given user, similar cases are retrieved from the database.

– The retrieval is done by using Euclidean distance of the movies rated by both the users to be compared.

• Reuse the information and knowledge in that case to solve the problem. – If there exists a similar case then the ratings are updated for the user with other users ratings.
• If case does not exist or partially exist then revise the proposed solution.
• Retain the parts of the experiences likely to be useful for future problem solving.

Later the system is made distributed in order to scale the system to many users and deal with the large data set.

Distributed system: A distributed system is a system whose components are located on different networked computers, which communicate and coordinate their actions by passing messages to one another. The components interact with one another in order to achieve a common goal. Some things which you can achieve by making the system distributed are:

• Achieve scalability
• Reliability as one system fault doesnt stop the system.
• Increase in performance with less cost.

Experimental Analysis/Simulation Setup

We used data from our Movie Lens recommender system. Movie Lens is a web-based research recommender system that debuted in Fall 1997. Each week hundreds of users visit Movie Lens to rate and receive recommendations. Movie Lens data sets were col- lected by the Group Lens Research Project at the University of Minnesota. This data set consists of: 100,000 ratings (1-5) from 943 users on 1682 movies. Each user has rated at least 20 movies. This data has been cleaned up - users who had less than 20 ratings or did not have complete demographic information were removed from this data set. The data undergoes preprocessing steps first and so columns which are not useful like time stamp, trailer release date, imdb URL are removed. The movies which are rated by at least 50 users are taken into consideration for better analysis.

As part of the collaborative filtering technique, user movie matrix is made. The empty columns of the user movie matrix is filled by applying Case Based Reasoning. A user with similar rated movies with similar ratings is retrieved using euclidean distance of the movies rated by both the users to be compared. If such user exists then the ratings of all the movies of that user are retrieved and the empty data is filled with those ratings. If such user doesnt exist or there is a partial match with other users in the data set, then 20 most nearly similar users are retrieved and the ratings are averaged to and ratings of the current user are filled. Doing so will eliminate bad recommendations to the current user.The movies with more cosine similarity are recommended to the user after the data sparsity of data set is solved.

Results and Discussion

The collaborative filtering is implemented successfully to provide recommendations to the user. Case Based Reasoning is also successfully implemented to deal with data sparsity issue. Unlike traditional collaborative filtering, the algorithm also performs well with limited user data, producing high-quality recommendations based on as few as two or three items.The algorithm generates recommendations based on a customers who are most similar to the user. It can measure the similarity of two customers, A and B, in various ways; a common method is to measure the cosine of the angle between the two vectors. Making the system distributed will be implemented next semester as the next part of the project as scalability is a real life issue and is very important. Currently the system takes around 20 mins to give recommendations as the data is very vast. With millions of users and items, a typical web-based recommender system running the algorithms will suffer serious scalability problems. This needs to be optimized and will be the main goal of the next part of the project.

From the experimental evaluation of the collaborative filtering scheme we make some important observations. The collaborative filtering scheme provides better quality of pre- dictions when case based reasoning is applied. The results were checked manually. The improvement in quality is consistent over different neighborhood size. However, the im- provement is not significantly large.

Summary and Conclusion

Recommender systems are a powerful new technology for extracting additional value for a business from its user databases. These systems help users find items they want from a business. Recommender systems benefit users by enabling them to find items they like. Conversely, they help the business by generating more sales. Recommender systems are rapidly becoming a crucial tool in E-commerce and other markets on the Web. Recommender systems are being stressed by the huge volume of user data in existing corporate databases, and will be stressed even more by the increasing volume of user data available on the Web. The science of recommendation is just starting despite impressive progresses, much remains to be understood. For further advances intuition alone is no longer enough and a multidisciplinary approach will surely bring powerful tools that may help innovative matchmakers to turn the immense potential of recommendations into real life applications.

New technologies are needed that can dramatically improve the scalability of recom- mender systems. In this paper we presented and experimentally evaluated a new algorithm for Collaborative Filtering-based recommender systems by applying case based reasoning on top of it. This technique show that item-based techniques hold the promise of allowing Collaborative Filtering-based algorithms to scale to large data sets and at the same time produce high-quality recommendations.