Search engine projects have historically been viewed as the responsibility of the IT department, covering installation, configuration, content ingestion and operations. The user interface may be developed by engineers themselves or with the participation of UX specialists. What this often leads to is a search engine that is performant and reliable but not necessarily accurate and in many cases is mis-aligned with business priorities.
At OpenSource Connections (OSC) we have long believed that without an effective search team, with members drawn from all areas of the business, you can only solve part of the problem. Your search solution is there to address a business case after all – for example to sell products, provide the correct information to users, save them time – but this business case is not always clearly presented to engineers. Search engines also depend very much on good quality content. No matter how clever the technology, this is very much a case of garbage in, garbage out. New, exciting features (nowadays often incorrectly presented as AI) will not help when metadata is inconsistent and data modelling incomplete.
So when you’re considering your next search project, what sort of skills should you look for in your search team? Let’s consider the roles in a ‘perfect’ search team. (In smaller organisations the same people may hold several of these roles, and it is rare in our experience to find an organisation that has all the roles covered.) This list is drawn from OSC’s ‘Think Like A Relevance Engineer’ training materials.
- Stakeholder – responsible for aligning search improvements with financial and corporate benefit
- Product Owner – responsible for ensuring search improvements meet the information needs of the customer
- Project Manager – responsible for planning and prioritising changes that are translated to features from the customer information needs
- Product Developer – responsible for design and UX implementation in the product
- Content Owner – responsible for defining the content set for the product, and coordinating development teams to arrange content access and transport
- Metadata Owner – responsible for defining and managing any metadata assets that are used to improve search, including synonyms, lemmatisation files, spelling dictionaries, word-wheels, etc.
- Architect – responsible for integration strategy and planning of technical changes across the system for cross cutting concerns and big-picture technology fit
- Search Relevance Strategist – responsible for solution strategy and planning of technical changes across the system related to search improvements
- Search Relevance Engineer – responsible for search engine tuning and delivering associated measurements and experiments
- Software Engineer – responsible for solution delivery and detail-oriented implementation of functionality and features related to search improvements
- Data Analyst/Scientist – responsible for analytical data access and transport, identifying customer trends and engagement signals, and coordinating judgment and rating data acquisition
You may already have some of these roles filled in your search team, and some can be generic across many projects – Product Owner for example. However, some of these roles need specific, specialist skills.
A Search Relevance Strategist is someone with long experience of information retrieval, search engine technology and implementation. They know what is cutting edge technology, but also what basic functionality should be built first. They know how to measure search quality effectively and how to design processes to make this happen. They probably don’t write production code, but they can guide and mentor others on the team who do. They have good communication skills and can inspire others to improve search quality.
A Search Relevance Engineer has practical, up-to-date knowledge of the search engine you are using, is aware of its features (and drawbacks) and how to implement them to solve relevance challenges. They are trained in information retrieval fundamentals, know how to model source data for effective search and how to format search queries correctly. They can be a highly effective member of your development team, using their experience to build features, right first time.
A Data Analyst/Scientist working on a search project needs to know what can be measured, what is important to measure and what conclusions can be drawn from the data. They can help you create meaningful metrics and visualisations so the whole search team can see the impact of a potential change and identify potential risks and benefits. Search quality is not always easy to measure, and you may not have (or be able to gather) a full picture of how your users interact with a search application, so a pragmatic approach is best. A search data analyst/scientist will have a good grasp of the various relevance measurement tools that have appeared over the last few years, many of them open source software.
Content Owners and Metadata Owners are usually subject matter experts (SMEs). SMEs know your content intimately – in e-commerce search they know what you sell (and importantly what you don’t), what your competitors sell and what is the ‘right’ answer to a search query. They’re familiar with part numbers, content areas, what is up to date and what is a little stale. They’re a helpful librarian who knows which shelf holds that obscure book you can half remember; a gardening expert who knows which fertiliser to use on your roses; a legal taxonomist or a medic aware of the difference between a cardiologist and a cardiothoracic surgeon. They are aware of the structure of your content – which fields are important to search and which are additional context. In your search team these SMEs can explain to engineers why a result is good, or bad for a particular query and they can be essential parts of any search engine test framework, giving expert ratings (but not always agreeing with their colleagues on these).
So now we know how to build the perfect search team, how do we make sure each member has the appropriate skills? As many have discovered when trying to recruit staff for their search project, these skills are relatively rare, and experts can command a high price. OSC’s advice is to focus on empowering your search team for success, supporting them to develop their skills so eventually they can fully ‘own’ the search solution. There are various ways to do this:
- Expert training. OSC and other organisations provide training in search engine basics, relevance engineering and some more advanced topics such as Learning to Rank and Natural Language Processing. The quality of this training can vary, especially at the beginner level, and we recommend you consider training delivered in a practical fashion with exercises and workshops.
- Partnering and Mentoring. If you work with external partners, think about how they can help mentor your team and teach them the skills they will need while they work on your project. It can be a bad strategy to outsource your search entirely as it can lead to over-dependence.
- Read the literature. There are many books and useful blogs on information retrieval, search engine fundamentals, relevance engineering and even advanced topics such as Deep Learning for Search. Some good starting points are available on Martin White’s website.
- Attend events. There is a range of events where search and relevance topics are discussed, ranging from large commercial conferences such as Lucidworks’ Activate and Elastic’s Elasticon, academic events such as ECIR and SIGIR, smaller and more community-driven events such as OSC’s Haystack, the British Computer Society’s Search Solutions and Berlin Buzzwords. In many larger cities there is a regular Search Meetup which is usually free to attend. Encourage and fund your team to attend these events, meet others facing the same challenges and importantly, participate by presenting or even offering to host a Meetup.
- Interact online. There are mailing lists and forums for particular search engines such as Apache Solr and Elasticsearch and more general forums such as OSC’s Relevance Slack. Since the search community is widely distributed these can be a good way to keep in touch with others outside of physical events.
In conclusion, an effective search team will be drawn from areas across the business and will require members to have a wide range of skills. The specialist skills required to improve search quality are rare and we recommend that you support your own staff in gaining these, using external partners where necessary, but being aware that you should aim at eventually owning your search. Community participation is vital and helps both with skills development and also to publicise that your own organisation has committed to building effective search – which can help when recruiting and retaining staff, as well as promoting the technical excellence of your approach.
This chapter was originally published in Search Insights 2021. Download the report for free here:
