High search quality is a necessary aspect of any successful business. What can you learn from the teams that have mastered this capability that you can replicate and use successfully in your business?
The well-functioning search team does two things: its members (1) rely on understanding customer needs based on how they express themselves through the product experience, and (2) leverage this data for improvement and decisions throughout the organisation.
In this article I will describe the need for data across the organisation, and how the various roles in a search team enable and consume that data. We will talk about roles and responsibilities across the organisation, grouped by competencies of the search team.
Decisions from data
“We need to be a data-driven organisation” may sound like a cliché but that doesn’t make it untrue. The most successful companies rely on gaining insights from expressive customer data and improving the product using those insights.
Search provides the purest form of customer expression for most products: freeform text in the form of queries, paired with the context of the person executing these queries. Most other forms of behavioural data that don’t include text, such as UI A/B tests, navigation and workflow paths, and other types of engagement metrics, can provide detailed insight into how the customer uses the product as it exists. Having a text box that anyone can type anything into and expect results, is the beautiful chaos of what your customers really want.
The only other expression tool that approaches search is customer feedback. Direct (feedback forms, surveys, and emails) or indirect (social media mentions) are included, but these types of feedback are only done by a small subset of the customer base and are usually lopsided towards negative feedback. The vocal minority should not be the only set of customers to drive changes to the product and/or its features.
At OSC we use search as the key to understanding customers, and react according to the data the search product yields.
This tree gives a visual expression to how competencies gather and leverage search data:
Business competency
How can the executive, sales, and marketing group guide search quality, without understanding how or why it works? The good news is that not much changes here aside from awareness and alignment. KPIs related to revenue, profitability and customer growth are still the north star for the business, and these lagging KPIs are influenced by search engagement. The trick is attributing these lagging metrics to search.
Depending on the domain, it may be difficult to arrive at these numbers. For eCommerce and asset search, it is straightforward. But how does one understand whether search is not only successful, but successful to the point of raising the business?
For eCommerce, I wrote extensively on site-search KPIs in a series of blog posts: (https://opensourceconnections.com/blog/2020/08/28/e-commerce-site-search-kpis/).
For other types of search, I recommend focusing on Net Promoter Score (NPS) while directly asking about search during the survey. Correlate this with data based on engagement.
Roles and their impact on search
Follow these team responsibility guidelines to enable a well-performing organisation that leverages search data.
- Executive Stakeholder – this is usually an SVP or VP of technology for medium/large businesses, and the CTO for small business. Sets revenue/growth targets and coordinates the executive team to rally around improvement. Drives the main strategic decision for search quality with the product owner. The executive stakeholder must also understand the relationship between leading product metrics of search quality and their impact on the lagging metrics of revenue and growth.
- Marketing Officer – ties site-search and content improvement with organic search (SEO) and campaigns. Works with the Engineering team to evangelise search technology internally and with the wider community to grow the technical brand of the search team.
- Account Manager – works with client/customer representatives and agencies to ensure high priority search quality feedback gets to the Product Owner.
- Product Owner – straddles the Business and Product teams to ensure unified improvement for the search product. Expresses business goals as OKRs to the product team. Ultimately accountable for all search quality initiatives.
Product and customer experience competency
Moving to the finer detail of what makes up the high-level business goals of revenue, we look to identify content/item and feature level impact on the customer, and how it shapes the product. The goal for the product team is to increase customer satisfaction. Higher satisfaction will correlate with a higher revenue/growth metric for the business. Since revenue and growth are lagging metrics, we focus on leading metrics for the product as an immediate marker for predicting this business improvement. For this, we use the KPIs of search conversion rate, time to conversion, and search abandonment/exit rate. These metrics can be set as KPIs in the team (with reasonable targets for improvement over time). Zero/low result searches are also an important metric to use as they identify gaps in the vocabulary and/or content.
Roles and their impact on search
Follow these team responsibility guidelines to enable a well-performing product that continually evolves to meet customers’ ever-changing needs.
- Product/Project Manager – decides goals and roadmap for the product with the Product Owner. Keeps the product and engineering teams on schedule and unified towards key search measurement, experimentation, testing, and deployment. Reacts to trends and changes that customers expect, facilitated through hypothesis-driven-development as an Agile methodology.
- Business Analyst – expresses goals as search requirements and stories. Uses dashboards and reports to inform and influence the product and content direction. Responsible for capturing explicit feedback (judgement lists) for search quality measurement and understands impact of changes made by the engineering team.
- Content Owner – responsible for creation/curation and quality of content or product metadata for search. Includes merchandising, landing pages, content style, and vocabularies.
- Customer Experience – defines personas and customer information needs. Accountable for wireframes and behavioural flows for search, results, facets, autocomplete, spelling, highlighting, and all other areas of the search experience.
- Support – aids customers having trouble with search and informs the Product and Engineering competencies of areas of improvement to increase satisfaction and reduce support/helpdesk overhead.
- UI/Design – expresses the search experience wireframes/workflows as implementable designs. Responsible for brand and style conformity, accessibility, and attractiveness of the product. Some teams may include front-end development in this role.
Engineering competency
Moving into the technical aspect of search, the Engineering competency supports the product with efficiency and relevance, expressed by formulaic measurements. Efficiency is set by performance KPIs and the SLA, relevance metrics may include Normalised Discounted Cumulative Gain (nDCG), Mean Reciprocal Rank (MRR), Expected Reciprocal Rank (ERR), and others.
Engineering must also support development of the search product as decided with the product competency, based on strategic direction from the business team. Metrics come from implicit or explicit judgements, with the latter usually provided by the product team in a tool such as Quepid (https://quepid.com). Implicit judgements are gathered from usage logs and verified by the data team.
Roles and their impact on search
Follow these team responsibility guidelines to build a snappy and relevant search system by leveraging the data feedback loop.
- Architect/Technical Leads – responsible for overall services, content and data flows, and system design in a multi-cloud environment. Ultimately accountable for design, improvements, and quality of the search platform. Designs search service, API contracts, and analytics capture touchpoints. Coordinates search efforts across the entire group.
- Developers – implement the services as agreed with the architect and DevOps roles. Ultimately accountable for shipping working code of a high standard.
- DevOps – responsible for CI/CD of the search platform, and sizing of the search engine and its infrastructure. Ultimately accountable for uptime, deployments, scalability, redundancy, backups, and performance.
- Quality Assurance Engineers – responsible for testing all technical aspects of the search platform and whether it meets product team requirements. Implements test tooling, and automation for relevance and code quality. Works closely with others to implement A/B or multi-armed-bandit testing frameworks. Identifies gates and safeguards for the CI/CD of the platform.
- Relevance Engineers – responsible for the relevance of search results given the customer context and query. Develops and tunes all search features such as query parsing, result scoring, autocomplete, spellcheck, highlighting, and others. Ultimately accountable for increasing nDCG or other search metrics. Informs Architect and DevOps to ensure performance of the search engine remains high.
Data competency
Can you correlate the lagging revenue and growth metrics with or search conversions? All of the above must be supported by data, and here is where the foundation for understanding is created and grown.
- Data Quality Engineers – responsible for automation and tooling of content and metadata quality. Works closely with the content owner.
- Analytics Engineers – responsible for implementation of data capture and reporting services. Works closely with the Architect, Developer, and DevOps roles to integrate a stack-wide data platform that captures customer behaviour and search platform performance. Identifies gaps in metrics and logs and works to fill them over time based on priority.
- Data Scientists – responsible for converting raw analytical data into actionable reports and insights. Develops and tunes models for Learning-to-Rank and Natural Language Search.
Roles and responsibilities of a search team
We surveyed several companies to understand what roles make up their successful teams and what their responsibilities are. This chart gives an overview of the involvement of each competency.
In conclusion
Let’s put it all together! Here is the cycle of understanding:
The product competency expresses known customer information needs and personas that are a priority for the business. The product competency also curates explicit feedback to be used by engineering in baseline setup.
The engineering competency implements baseline relevance and systems, and integrates analytics and data capture points based on direction from the data team.
Once live, search data is captured and is used by the data competency for analysis, and to create reports and dashboards showing the important metrics that all competencies require.
The product, engineering, and data competencies iterate to optimise their KPIs over time.
As search conversion rates increase (and abandonment decreases), customer satisfaction increases. Customer satisfaction is then seen as increasing the lagging KPIs of revenue, customer growth, and NPS.
Having these activities in place is a necessary condition for the improvement of search quality. A highly capable data driven team will accelerate the business to success.
