Leading Search Engine Expands Internationally with Vendor-Neutral Quality Analysts

The search engine firm turned to us for its proven expertise in quality management to provide a team of quality analysts.

The Company

A leading multilingual search engine provider ready to invest in building out its search engine across 25 markets teamed up with us, thanks to proven track record as a reliable source for quality management and high-quality data.


The Challenge

A leading multilingual search engine provider had worked with us for a few years on search relevance evaluation. In that time, we had proven to be a reliable source for high-quality data. The firm partnered with us to develop quality metrics, improve rating guidelines, and increase the success of evaluators in production.

When the provider decided to invest in building out its search engine across 25 markets, it determined it did not have the in-house expertise to provide quality feedback to all the vendors it was working with. It needed a vendor neutral team of quality analysts who would keep the evaluators across all vendors in sync.


The Solution

The search engine firm turned to us for its proven expertise in quality management to provide a team of quality analysts. We identified its strongest, most experienced evaluators and auditors as candidates for inclusion in the new team. Each candidate had to prove not only their ability to apply the rating guidelines consistently and accurately, but also had to demonstrate strong communication skills, objectivity, and critical thinking. The initial team consisted of two and later expanded to seven.

To ensure the neutrality of the quality analyst team, we kept the management of the QA team completely separate from the management of the associated programs. A vendor manual was also developed outlining the expectations for all vendors so that preference or favoritism was not given to one vendor over another.


The Result

The search engine provider significantly improved the reliability of its search results and the quality of its data collection due to the vendor-neutral quality analyst team we provided, even though data came from multiple vendors. The team ensured that all vendors received targeted feedback and that they were held to the same standards, raising the level of quality across the board. The program we developed allows the client to get consistent performance despite using multiple vendors.

Our team of seasoned judges and auditors provided the foundation for this program and is the reason for its continued success. The team provides objective feedback across multiple vendors and manages multiple communication streams to provide a single voice back to the client. Guidelines are also updated regularly to reduce confusion among the judges and to ensure that all judges are rating in the same fashion. With this clear and objective approach, we reduce the noise in the data. This has led to a strong program with stable metrics that continues to thrive today.

Website for deploying AI with world class training data


Andrew Ettinger | Chief Revenue Officer

Andrew Ettinger joined Appen as Chief Revenue Officer in May 2023 overseeing the company's revenue strategies and driving growth in the field of AI. He joined Appen with more than 25 years of sales experience in sales and services in the technology industry. 

Andrew's expertise extends to harnessing the power of data to drive insights and optimize processes. As the Chief Revenue Officer at Astronomer, he successfully grew the adoption of their open-source data solution, leading to a remarkable increase in monthly downloads and revenue. His strategic initiatives resulted in a 600% growth in customer count and a 75% win rate. 

Prior to joining Astronomer, he served as the VP of Sales at Pivotal Software, where he helped grow the business from zero to $100 million in annual recurring revenue in a single year leading up to Pivotal’s initial public offering, and up to $500 million thereafter. Under his leadership, the company achieved three consecutive years of 50% revenue growth, fueling digital transformations for Fortune 500 companies in various sectors. 

Andrew holds a Bachelor of Science in Business Marketing from The Ohio State University.