Search and recommend those things

The difference between search and recommendation

1. There are different scenarios

The search scene is named after the search scene , That is, the user provides a description of what he wants to find , The system returns the matching result to the user , Common scenarios such as text input box search , Image search , Listen to the music , Tag filtering, etc , It looks like a lot of scenes , In fact, it's just that the form of user input is different .

The recommended scenarios are common to us App Personalized recommendation on the home page （ Guess you like it / Daily song recommendation ）, Select the associated recommendation of the page （ Buy it or not , Look and see , The users who bought it also bought and so on ） etc. , The recommended scenarios are more abundant , Because there is no restriction on the content provided by users , The scenes are more diverse , There are many ways to recommend , For example, content-based recommendations , Recommendation based on user behavior , Collaborative filtering and so on .

The major Internet platforms are due to Different service contents , Different platform maturity , The emphasis on search and recommendation is not the same , But they are all indispensable .

For example, for real estate applications , User goals are clear , Search service will bring more purchasing power , But related recommendations will give users more choices , It is also indispensable .

For short video platforms , It is difficult for users to provide content description through text or pictures , Naturally, we will focus on recommendation services .

For e-commerce, it must be the search service that brings more purchase rates in the early stage , When the buying rate reaches the bottleneck , The purchase rate brought by recommendation is a necessary means to break through the bottleneck and continue to develop .

2. Input and output are different

Whether searching or recommending , In fact, for users , It is a black box that provides services , It can be based on the user / goods / Scene and other information , Select a list of items matching the user from the pool of candidate items .

The difference is for search services , In addition, it also provides users' description information about their demands （ Of course, the description may not be accurate ）.

The difference in input naturally leads to different expectations of users for the results ：

• Different degrees of personalization

The recommendation system puts more emphasis on personalization , Even more emphasis on surprise . There is often a trade-off between accuracy and diversity ; The search system emphasizes more on relevance , If the search results do not match the user's goals , The acceptance of users will be very poor , Personalization is both meaningless and risky for search systems .

• Better row and more complete search

For recommendation systems , Sorting is more important , Because only the initial recommendation results attract users , Users can only browse backwards .

For a search system , Recall is more important , Because users will take the initiative to browse backwards , Find your goal with expectation , But if you don't find it in the end , That is, the search is incomplete , There will be a poor user experience .

• Quick satisfaction or continuous service

When it comes to search systems , Matthew effect is often mentioned , Only items that match the results of the user's search will be presented to the user , Let users get quick satisfaction , There are so many items to meet the demand , The more accurate the search , The less likely users will be to browse backwards , Finally, the popularity of clicking will only focus on a small number of items . This is why advertising was first born in the search system .

Mention the recommendation system , The long tail effect is often mentioned , That is to keep users fresh and surprised , Consider the long-term interests of users , Improve user stickiness , Expect to retain users , And provide continuous service , This is why the short video can't stop .

• Real time and lag

The real-time requirement of search data is particularly high , Data often requires second level updates , For example, if a commodity is out of stock, it should not be found . Many of the recommended data can tolerate day level updates , Because the recommendation needs to consider a lot of user behavior information , It must have a certain lag .

Search for links to recommendations

1. The same essence

Search and recommendation are essentially the product of information overload in the current era , The fundamental solution is through matching （ Recall ）、 Sorting is to select the information the user wants from the overloaded information . Just according to different business scenarios , In the recall , The priorities considered in the sorting stage are different .

2. The synergy of search and recommendation

• Search in recommendations

Content-based recommendation in recommendation services is actually equivalent to a silent search , The inverted index and other technologies in the search service are often used in the implementation , For example, content-based recommendations , It is often through rules or recommendation models to get the tags of the content that users are interested in , Then we use the method of search service to search and match tags to get the final recommendation list .

• Recommendations in search

When a large amount of data matching the user is searched , It is necessary to help the search service match the needs of users according to the results such as user portraits in the recommendation service . For example, the list of results obtained by searching on Monday night and the list of results obtained by searching on Friday night will be different .

Recommendation and search often work together in one page to provide services for users , For example, the association recommendation of the search engine search results page , E-commerce software searches for relevant recommendations on the browsing page, etc .

Architecture evolution and architecture unification

Evolution of search architecture

generally speaking , An enterprise search engine , Since there are not many lines of business in the initial stage , Provide a simple search service . With the increase of business , Continuous abstraction and unification of search requirements , It can gradually develop into a platform stage , It provides the ability to write to multiple data sources and search for multiple services , Different requirements of different businesses can be flexibly configured .

Wait until the number of business lines increases , The docking business takes up most of the development time , Develop more convenient operation, maintenance and management capabilities , The help in service self-service access platform can further improve the efficiency of search function development , At this point, the search architecture has entered the stage of a more convenient cloud platform for operation and maintenance .

Recommended architecture evolution

For recommendation engines , In the initial stage, the content-based recommendation method is generally adopted , Due to insufficient data , At the initial stage, the enterprise will label items and users based on the experience rules provided by the business side , And then make recommendations by matching labels online . Continue to develop , With the continuous enrichment and iteration of the business , There will be more expectations for the recommendation system , When experience rules are constantly modified or added but cannot meet business needs , We need some model-based recommendation methods and personalized recommendation services . Further , Like search engines , The recommendation engine also needs to interface with multiple business lines , To the platform stage , Provide unified public services , Meet the needs of different business lines through configuration .

Unified architecture

From the above introduction and architecture evolution, we can find that , There are many reusable aspects of the recommended and searched architecture , Therefore, the architecture can be unified .

1. Unification of processes ：

Whether it's search or recommendation , Will experience recall - Sort - Rearrange and other processes , Finally, you get a list of items presented to the user , It's just that the goals of each stage of the process are different .

1. Reuse of data and data platform ：

The searched items and the recommended items are unified , Recall the buried point data required for sorting training model / User behavior data is also unified , So naturally get the data / The data processing platform can be reused naturally .

1. Algorithm and algorithm platform reuse ：

Search and recommendation have developed to a certain stage , When simple expert rules can no longer support complex search and recommendation requirements , Will develop to the stage of model-based recall sorting , At this time, it is necessary to use the user data / Item data / Embedding point data for model training , Just because their training objectives are different , The parameters of the trained model may be different , But algorithm platform or machine learning as we often call it /AI The platform is reusable .

1. A/B Test Reuse of experimental platform ：

Due to the changing business needs , Continuous replacement of models , adopt A/B Test The platform can get the user feedback in the real production environment through diversion , To help enterprises constantly verify and optimize search and recommendation strategies .

1. Configuration center reuse ：

You can configure different search and recommendation policies for different businesses and services through the configuration center , And provide convenient one click deployment capability .

So many companies , In the business field, search and recommendation belong to different departments , But many public parts have mature internal platforms that can be reused quickly .

summary

This article introduces the difference and connection between search and recommendation , Architecture evolution and architecture unification . We all know that architecture is constantly evolving due to the expansion of requirements , For example, from the service stage to the platform stage , The reason is to improve the docking efficiency of multiple services ; From content-based recommendation to complex personalized recommendation integrating online user portrait and offline user portrait , The reason is that simple recommendation based on rules or tags cannot meet the needs of users and business side .

So don't be tied up in the beginning by an overly complex architecture , You can search according to your own business needs / Recommended simple architectural design , Then gradually evolve and optimize the architecture .

Reference Content ：https://www.6aiq.com/article/1601333030483

---

writing /Thoughtworks Houyumei
Link to the original text ： Looking at search and recommendation from the perspective of architecture evolution and unification -Thoughtworks Insight