If you purchase an independently reviewed product or service through a link on our website, SPY.com may receive an affiliate commission.
SPY.com is a discovery and reviews platform, connecting you with the latest tech, style, smart gadgets, and health and grooming staples you can’t find elsewhere. We bring you under-the-radar releases and top-rated picks from across the globe. Over the past few years, the SPY team has tested thousands of products, and we have a simple goal: to help you shop smart. We will never recommend a product that we wouldn’t buy ourselves, and we love finding awesome new gear to share with SPY’s readers. So how do we do that?
The SPY editorial team includes product testers, commerce journalists, and experienced product reviewers with decades of combined experience testing products. Unlike some of our competitors, which have a no negative review policy to protect their relationships with brands, we will always be honest about the pros and cons of the products we test. If we find that a product doesn’t meet our standards, we won’t recommend it just to make a buck. In addition, brands cannot pay SPY for positive reviews, and every review and shopping guide on our site is based purely on our editorial recommendations.
To find the best products, we scour the web, carefully read customer reviews, compare product specifications, and talk to experts. When writing about the best face wash for men, we’ll consult dermatologists in addition to relying on our grooming writers’ expertise in this subject area. Most importantly, we have implemented a rigorous product testing system to ensure SPY readers are getting the best possible product recommendations.
To be as transparent as possible, we wanted to outline our product testing process in detail. You can also refer to our About Us page for more information on our editorial process.
On any given day, you’ll find SPY’s Tech Editor putting the latest security cameras to the test, or evaluating how effectively a new robot vacuum navigates obstacles. Our grooming writers are constantly testing and comparing products such as face scrubs, moisturizers, men’s razors and beard trimmers. You’ll also find our Senior Commerce Editor Taylor Galla pushing fitness mirrors, spin bikes and massage guns to their limits. These tests might involve using a decibel meter to rate noise levels or an illumination meter to determine if a projector actually offers the lumens promised by the manufacturer.
The SPY Testing Process, Explained
Whether it’s a shopping guide for Christmas Gifts or a round-up of the best earbuds for swimming, our editors and writers hand select every product we feature on SPY. There’s no algorithm picking products for you. Increasingly, we’re making as many recommendations as we can based not just on research and expert recommendations, but also in-depth product testing.
The SPY team has rapidly ramped up our product testing efforts in 2021 and 2022, and we’ve fine-tuned our process. Frankly, there’s a lot of crap you can buy online, and we’re here to do the heavy lifting for you so you don’t have to stress about whether the package arriving at your doorstep actually does what it promises.
If you’ve ever been curious to take a peek behind the shopping guide and see how we chose a specific product, we’re going to explain our process for testing step-by-step — from smartphones to robot vacuums and bald head shavers.
We’re going to cover the key categories we test: Gadgets, Health & Fitness and Grooming.
Step 1: Research
The first step in any product review is simple: research, and lots of it. We start by conducting market research to suss out the overall product category, what the top performing brands are, which products people have questions about and which are the biggest players in the space. If a brand is definitely worth testing, chances are we’ve already heard of it, but sometimes a new one emerges onto the market or there are smaller budget brands worth trying as dupes for a more expensive product.
Depending on the product category, the number of products we test can vary wildly. For a product with thousands of choices, like men’s underwear, we narrowed down our list of test products to about 100. For adjustable dumbbells, that number was closer to 15. After we have our list of products we need to test to determine rankings, we start calling in samples.
Part 2: Calling in Sample Units
In order to conduct reviews, brands will sometimes send out what are called “sample units.” These products are typically free for publishers like SPY. In return, brands typically request that we publish a review, but we make no guarantees about coverage. We have tested many products that ultimately we decided not to feature on the site. Again, we always call in sample units without a guarantee of coverage, so brands are fully aware they might not be featured on SPY if their product doesn’t meet our criteria.
In some cases, sample units are not available, and so we sometimes purchase products for our reviewers to evaluate. In any given review, there might be a mix of products provided to SPY for free and purchased by our team. This does not impact our overall rating system, and we do our best to disclose when we have received a sample unit.
Part 3: Creating Judgement Criteria
As the sample units are arriving, we create a list of review criteria. This is a detailed list of specifications and features we think are relevant to overall product quality. The criteria varies wildly depending on the specific product category. For instance, when testing robot vacuums, we test products based on the following criteria:
- Suction Power
- Battery Life
- Dustbin Size
- Ease of Use
- Loudness
- Cleaning Performance
- Navigation
- Obstacle Avoidance
- Software/App Experience
Here is a screenshot of our testing spreadsheet for the Best Weighted Jump Ropes.
And here is a screenshot of our testing spreadsheet for The Best Smartphones of 2022:
Part IV: Testing
Now we get to the fun part: testing. SPY has about a dozen experienced product testers producing reviews at present. We spend as much time as possible with each product, using it in a variety of contexts and taking note of everything. Here are a few questions we ask ourselves, regardless of what the product is:
- -How easy is the setup? Does the product come with intuitive instructions, or was the setup laborious?
- -Is the product easy to use? Does it function as intended?
- -Is the product durable? Did it maintain its form after multiple uses and tests?
- -How does it compare to other products in terms of quality? Is it worth the hefty price tag? Is it a worthwhile budget option for a more expensive product?
When testing a product, our reviewers evaluate and rate how it performs according to the criteria we’ve set. This can be a time-consuming process, and sometimes it can take three months or longer to fully produce a review like this. Reviewing products in this way allows us to objectively compare how similar products perform. We log detailed notes about how well the product performs in the given criteria.
Finally, we use a formula to calculate an objective score for each product based on our testing. Currently, the number score given to each product is not stated in the review, although we may introduce a more formal scoring system in the future.
Part IV: Re-Testing and Updating
The last step in our testing process happens weeks, or even months, after our initial testing round. We regularly update our reviews and shopping guides with the latest products and new thoughts on products we’ve already tested. Sometimes our experience of a product changes after spending more time using it, and so we need to adjust our rankings accordingly. Likewise, if there’s a new product that’s garnered a lot of attention, we’ll test it to see how it fits into our existing ranking.
Right now, we are not sharing the formulas we use to rate and score products, although we’re always happy to answer reader questions. Visit our Contact Us page to learn more.
How We Test Gadgets
Our tech editor, John Velasco, and his team of freelancers test all of the gadgets we review here on SPY. John is an experienced tech editor with close to two decades of experience working in the tech field. He’s reviewed a variety of smart home products, the latest tablets, smartphones, wireless earbuds, waterproof speakers and much more. He’s been bringing his expertise to the Gadgets vertical at SPY since December of 2021.
Gadgets come in all shapes, sizes, functionality, and experiences. Not only are we continuously stoked about testing the latest and coolest tech gadgets around, but we strive to put them through an assortment of tests when we review them. Some devices require a lot more time than others to gauge their worth, such as robot vacuums that require many cleanings over and over again to see how well they vacuum. Meanwhile, other smaller things such as phone cases and laptop chargers can be assessed in a shorter period. Here’s a rundown of how we test gadgets.
- -At the minimum, we spend five days with a gadget before reviewing, but sometimes it could be two weeks or two months
- -Objective tests are performed on some products, such as running benchmark tests on phones and using a lux meter to measure the peak brightness output of their screens.
- -Setup and installation are equally important, so we watch out for how intuitive they are to set up without any help.
- -Comparisons between other devices we’ve used help us to also determine the superior performance of one device over another. For example, we take the same photo under each condition to test the image quality of a security camera.
- -Most importantly, we test them in real-life situations, so we can get a better grasp at how they function to improve our lives.
How We Test Health and Fitness Gear
Health and fitness is another product category in which we’ve done extensive testing. SPY’s Senior E-Commerce Editor Taylor Galla has been testing health and fitness equipment for more than two years. She’s reviewed close to a hundred different massage guns, fitness mirrors, pairs of adjustable dumbbells, spin bikes, indoor rowers and now weighted jump ropes.
For each of these categories, she put each product to the test using her own body and workout routine. For the spin bikes, fitness mirrors and rowers she worked out with the machine for at least two weeks, taking a variety of classes on its platform and observing the quality of the machine. Here are a few other things she takes note of when testing smart fitness equipment like the Peloton, Hydrow rower and Tempo Studio fitness mirror:
- -Is the machine easy to set up in your home?
- -Are the instructors engaging? Are the classes effective and efficient workouts?
- -Is there equipment included with the machine? Is it high-quality and easy to use?
For tools designed for workouts, like weighted jump ropes and adjustable dumbbells, Taylor spent at least a week with each product using it during her regular morning workouts. For adjustable dumbbells, for example, she was observing how smooth the weight changes are, how comfortable/easy the dumbbells are to hold, how many pairs of dumbbells each pair replaces and how worthwhile they are for the price.
It’s a lot of movement, and a lot of working out, but it’s worth it. The overall question Taylor asks herself with every piece of fitness equipment she reviews: will the reader actually want to use this multiple days a week?
It’s a cliché with at-home fitness equipment for a reason: you spend thousands of dollars on something so you can workout at home, and after a few months it’s a glorified coat rack. We’re trying to avoid that fate as much as we can. While we can’t walk into your house and coax you onto the bike, tread or rower, we can do everything in our power to make sure you’re spending thousands on a machine you’ll (maybe) want to use every day.
How We Test Grooming Products
Many of our editors have tested grooming products in the past, but SPY’s E-Commerce and Special Projects Editor, Tyler Schoeber, has done the majority of the tests for devices like bald head shavers, as well as beard oils, face washes and serums.
When covering grooming products, our testers ensure nothing is tested in a one-and-done format. We want to be sure these products work way more than once – especially when they’re being used on your body. From razors to beard trimmers to bald head shavers, our testers test each product at least three times each before writing a proper review. This is so we can see if the device being tested works the same way it did the first time. We will continue to test thereafter if we find necessary to make sure the devices hold up. This is important specifically in relation to battery life (because so many companies tend to fib about longevity) and product design (because a product that breaks mid-shave isn’t exactly the “best”).
When testing electric and battery-operated grooming products such as bald head shavers and beard trimmers, it’s important to test more than one time to ensure the battery-life is as accurate as the brand says it is. We want to make sure you’ll go through an effortless time using and that the battery won’t die on you mid-trim. In addition to that, grip is important because nobody wants a trimming to fall out of their hand and cutting hair you didn’t want to cut. Devices should feel comfortable in your hand from start to finish. How long the trim takes, how the trim feels and how the trim looks are also incredibly important, because why would you buy something that doesn’t work for your needs?
When testing products such as beard oils, face washes and more topical oils, serums and washes, the after effect is the most important part. One use will not tell you how a product like these work. We use them a few times to check for gradual differences over time. You know what they say, Rome wasn’t built overnight!
Meet Some of Our Product Testers
John Velasco, SPY Tech Editor
John Velasco has decades of experience in the tech world, and over a decade of experience covering the mobile tech world for publications like Digital Trends, Android Authority and PhoneArena. Before coming to SPY, he was the Smart Home editor for DigitalTrends, and is now an expert in numerous gadget categories including mobile phones, smart home devices, laptops, tablets, flat screen TVs and more. He’s also passionate about virtual reality, drones and electric bikes, and has a YouTube channel where he’s shared his reviews with thousands of people.
Taylor Galla, Senior E-Commerce Editor
Taylor Galla is a certified yoga teacher and also SPY’s Senior E-Commerce Editor who focuses many of her hands-on reviews in the health & fitness space, due to her personal interests and growing expertise. She has a rigorous training regimen in her off hours and has tested close to a hundred different health, fitness and wellness-related products for SPY. She’s written reviews and review round-ups for products across the space, and before SPY she consulted with professional athletes on their media strategy and digital presences.
Tyler Schoeber, E-Commerce and Special Projects Editor
Tyler Schoeber has tested hundreds of products for SPY, from men’s underwear to bald head shavers. Tyler tried on over 100 pairs of men’s underwear to help narrow down just that category, and he’s approached every review round-up of his with the same dedication. Tyler also has an advanced degree in photography and has helped the team test the top camera brands for our best instant cameras round-up.
