How We Test Products at SPY.com

If you purchase an independently reviewed product or service through a link on our website, SPY.com may receive an affiliate commission.

Best Outdoor Home Security Cameras Review Roundup

 

SPY.com is a discovery and reviews platform, connecting you with the latest tech, style, smart gadgets, and health and grooming staples you can’t find elsewhere. We bring you under-the-radar releases and top-rated picks from across the globe. Over the past few years, the SPY team has tested thousands of products, and we have a simple goal: to help you shop smart. We will never recommend a product that we wouldn’t buy ourselves, and we love finding awesome new gear to share with SPY’s readers. So how do we do that?

The SPY editorial team includes product testers, commerce journalists, and experienced product reviewers with decades of combined experience testing products. Unlike some of our competitors, which have a no negative review policy to protect their relationships with brands, we will always be honest about the pros and cons of the products we test. If we find that a product doesn’t meet our standards, we won’t recommend it just to make a buck. In addition, brands cannot pay SPY for positive reviews, and every review and shopping guide on our site is based purely on our editorial recommendations.

To find the best products, we scour the web, carefully read customer reviews, compare product specifications, and talk to experts. When writing about the best face wash for men, we’ll consult dermatologists in addition to relying on our grooming writers’ expertise in this subject area. Most importantly, we have implemented a rigorous product testing system to ensure SPY readers are getting the best possible product recommendations.

To be as transparent as possible, we wanted to outline our product testing process in detail. You can also refer to our About Us page for more information on our editorial process.

On any given day, you’ll find SPY’s Tech Editor putting the latest security cameras to the test, or evaluating how effectively a new robot vacuum navigates obstacles. Our grooming writers are constantly testing and comparing products such as face scrubs, moisturizers, men’s razors and beard trimmers. You’ll also find our Senior Commerce Editor Taylor Galla pushing fitness mirrors, spin bikes and massage guns to their limits. These tests might involve using a decibel meter to rate noise levels or an illumination meter to determine if a projector actually offers the lumens promised by the manufacturer.

spy skincare products

 

best massage guns

 

The SPY Testing Process, Explained

Whether it’s a shopping guide for Christmas Gifts or a round-up of the best earbuds for swimming, our editors and writers hand select every product we feature on SPY. There’s no algorithm picking products for you. Increasingly, we’re making as many recommendations as we can based not just on research and expert recommendations, but also in-depth product testing.

The SPY team has rapidly ramped up our product testing efforts in 2021 and 2022, and we’ve fine-tuned our process. Frankly, there’s a lot of crap you can buy online, and we’re here to do the heavy lifting for you so you don’t have to stress about whether the package arriving at your doorstep actually does what it promises.

If you’ve ever been curious to take a peek behind the shopping guide and see how we chose a specific product, we’re going to explain our process for testing step-by-step — from smartphones to robot vacuums and bald head shavers.

We’re going to cover the key categories we test: Gadgets, Health & Fitness and Grooming.

Testing theragun mini

 

Step 1: Research

The first step in any product review is simple: research, and lots of it. We start by conducting market research to suss out the overall product category, what the top performing brands are, which products people have questions about and which are the biggest players in the space. If a brand is definitely worth testing, chances are we’ve already heard of it, but sometimes a new one emerges onto the market or there are smaller budget brands worth trying as dupes for a more expensive product.

Depending on the product category, the number of products we test can vary wildly. For a product with thousands of choices, like men’s underwear, we narrowed down our list of test products to about 100. For adjustable dumbbells, that number was closer to 15. After we have our list of products we need to test to determine rankings, we start calling in samples.

Part 2: Calling in Sample Units

In order to conduct reviews, brands will sometimes send out what are called “sample units.” These products are typically free for publishers like SPY. In return, brands typically request that we publish a review, but we make no guarantees about coverage. We have tested many products that ultimately we decided not to feature on the site. Again, we always call in sample units without a guarantee of coverage, so brands are fully aware they might not be featured on SPY if their product doesn’t meet our criteria.

In some cases, sample units are not available, and so we sometimes purchase products for our reviewers to evaluate. In any given review, there might be a mix of products provided to SPY for free and purchased by our team. This does not impact our overall rating system, and we do our best to disclose when we have received a sample unit.

Part 3: Creating Judgement Criteria

As the sample units are arriving, we create a list of review criteria. This is a detailed list of specifications and features we think are relevant to overall product quality. The criteria varies wildly depending on the specific product category. For instance, when testing robot vacuums, we test products based on the following criteria:

  • Suction Power
  • Battery Life
  • Dustbin Size
  • Ease of Use
  • Loudness
  • Cleaning Performance
  • Navigation
  • Obstacle Avoidance
  • Software/App Experience

 

Here is a screenshot of our testing spreadsheet for the Best Weighted Jump Ropes.

SPY how we test products

 

And here is a screenshot of our testing spreadsheet for The Best Smartphones of 2022:

how we test products at SPY.com

 

Part IV: Testing

Now we get to the fun part: testing. SPY has about a dozen experienced product testers producing reviews at present. We spend as much time as possible with each product, using it in a variety of contexts and taking note of everything. Here are a few questions we ask ourselves, regardless of what the product is:

  • -How easy is the setup? Does the product come with intuitive instructions, or was the setup laborious? 
  • -Is the product easy to use? Does it function as intended? 
  • -Is the product durable? Did it maintain its form after multiple uses and tests?
  • -How does it compare to other products in terms of quality? Is it worth the hefty price tag? Is it a worthwhile budget option for a more expensive product? 

 

When testing a product, our reviewers evaluate and rate how it performs according to the criteria we’ve set. This can be a time-consuming process, and sometimes it can take three months or longer to fully produce a review like this. Reviewing products in this way allows us to objectively compare how similar products perform. We log detailed notes about how well the product performs in the given criteria.

Finally, we use a formula to calculate an objective score for each product based on our testing. Currently, the number score given to each product is not stated in the review, although we may introduce a more formal scoring system in the future.

Part IV: Re-Testing and Updating

The last step in our testing process happens weeks, or even months, after our initial testing round. We regularly update our reviews and shopping guides with the latest products and new thoughts on products we’ve already tested. Sometimes our experience of a product changes after spending more time using it, and so we need to adjust our rankings accordingly. Likewise, if there’s a new product that’s garnered a lot of attention, we’ll test it to see how it fits into our existing ranking.

Right now, we are not sharing the formulas we use to rate and score products, although we’re always happy to answer reader questions. Visit our Contact Us page to learn more.

Best Smartphones Review Roundup

How We Test Gadgets

Our tech editor, John Velasco, and his team of freelancers test all of the gadgets we review here on SPY. John is an experienced tech editor with close to two decades of experience working in the tech field. He’s reviewed a variety of smart home products, the latest tablets, smartphones, wireless earbuds, waterproof speakers and much more. He’s been bringing his expertise to the Gadgets vertical at SPY since December of 2021.

Gadgets come in all shapes, sizes, functionality, and experiences. Not only are we continuously stoked about testing the latest and coolest tech gadgets around, but we strive to put them through an assortment of tests when we review them. Some devices require a lot more time than others to gauge their worth, such as robot vacuums that require many cleanings over and over again to see how well they vacuum. Meanwhile, other smaller things such as phone cases and laptop chargers can be assessed in a shorter period. Here’s a rundown of how we test gadgets.

  • -At the minimum, we spend five days with a gadget before reviewing, but sometimes it could be two weeks or two months
  • -Objective tests are performed on some products, such as running benchmark tests on phones and using a lux meter to measure the peak brightness output of their screens.
  • -Setup and installation are equally important, so we watch out for how intuitive they are to set up without any help.
  • -Comparisons between other devices we’ve used help us to also determine the superior performance of one device over another. For example, we take the same photo under each condition to test the image quality of a security camera.
  • -Most importantly, we test them in real-life situations, so we can get a better grasp at how they function to improve our lives.

best adjustable dumbbells

How We Test Health and Fitness Gear

Health and fitness is another product category in which we’ve done extensive testing. SPY’s Senior E-Commerce Editor Taylor Galla has been testing health and fitness equipment for more than two years. She’s reviewed close to a hundred different massage guns, fitness mirrors, pairs of adjustable dumbbells, spin bikes, indoor rowers and now weighted jump ropes.

For each of these categories, she put each product to the test using her own body and workout routine. For the spin bikes, fitness mirrors and rowers she worked out with the machine for at least two weeks, taking a variety of classes on its platform and observing the quality of the machine. Here are a few other things she takes note of when testing smart fitness equipment like the Peloton, Hydrow rower and Tempo Studio fitness mirror:

  • -Is the machine easy to set up in your home?
  • -Are the instructors engaging? Are the classes effective and efficient workouts?
  • -Is there equipment included with the machine? Is it high-quality and easy to use?

 

For tools designed for workouts, like weighted jump ropes and adjustable dumbbells, Taylor spent at least a week with each product using it during her regular morning workouts. For adjustable dumbbells, for example, she was observing how smooth the weight changes are, how comfortable/easy the dumbbells are to hold, how many pairs of dumbbells each pair replaces and how worthwhile they are for the price.

It’s a lot of movement, and a lot of working out, but it’s worth it. The overall question Taylor asks herself with every piece of fitness equipment she reviews: will the reader actually want to use this multiple days a week? 

It’s a cliché with at-home fitness equipment for a reason: you spend thousands of dollars on something so you can workout at home, and after a few months it’s a glorified coat rack. We’re trying to avoid that fate as much as we can. While we can’t walk into your house and coax you onto the bike, tread or rower, we can do everything in our power to make sure you’re spending thousands on a machine you’ll (maybe) want to use every day.

the best bald head shavers

How We Test Grooming Products

Many of our editors have tested grooming products in the past, but SPY’s E-Commerce and Special Projects Editor, Tyler Schoeber, has done the majority of the tests for devices like bald head shavers, as well as beard oils, face washes and serums.

When covering grooming products, our testers ensure nothing is tested in a one-and-done format. We want to be sure these products work way more than once – especially when they’re being used on your body. From razors to beard trimmers to bald head shavers, our testers test each product at least three times each before writing a proper review. This is so we can see if the device being tested works the same way it did the first time. We will continue to test thereafter if we find necessary to make sure the devices hold up. This is important specifically in relation to battery life (because so many companies tend to fib about longevity) and product design (because a product that breaks mid-shave isn’t exactly the “best”).

When testing electric and battery-operated grooming products such as bald head shavers and beard trimmers, it’s important to test more than one time to ensure the battery-life is as accurate as the brand says it is. We want to make sure you’ll go through an effortless time using and that the battery won’t die on you mid-trim. In addition to that, grip is important because nobody wants a trimming to fall out of their hand and cutting hair you didn’t want to cut. Devices should feel comfortable in your hand from start to finish. How long the trim takes, how the trim feels and how the trim looks are also incredibly important, because why would you buy something that doesn’t work for your needs?

When testing products such as beard oils, face washes and more topical oils, serums and washes, the after effect is the most important part. One use will not tell you how a product like these work. We use them a few times to check for gradual differences over time. You know what they say, Rome wasn’t built overnight!

Meet Some of Our Product Testers

John Velasco, SPY Tech Editor 

John Velasco has decades of experience in the tech world, and over a decade of experience covering the mobile tech world for publications like Digital Trends, Android Authority and PhoneArena. Before coming to SPY, he was the Smart Home editor for DigitalTrends, and is now an expert in numerous gadget categories including mobile phones, smart home devices, laptops, tablets, flat screen TVs and more. He’s also passionate about virtual reality, drones and electric bikes, and has a YouTube channel where he’s shared his reviews with thousands of people.

Ray-Ban Stories Smart Sunglasses Review

 

Taylor Galla, Senior E-Commerce Editor 

Taylor Galla is a certified yoga teacher and also SPY’s Senior E-Commerce Editor who focuses many of her hands-on reviews in the health & fitness space, due to her personal interests and growing expertise. She has a rigorous training regimen in her off hours and has tested close to a hundred different health, fitness and wellness-related products for SPY. She’s written reviews and review round-ups for products across the space, and before SPY she consulted with professional athletes on their media strategy and digital presences.

SPY editor testing Tempo Studio, best fitness mirrors

 

Tyler Schoeber, E-Commerce and Special Projects Editor

Tyler Schoeber has tested hundreds of products for SPY, from men’s underwear to bald head shavers. Tyler tried on over 100 pairs of men’s underwear to help narrow down just that category, and he’s approached every review round-up of his with the same dedication. Tyler also has an advanced degree in photography and has helped the team test the top camera brands for our best instant cameras round-up.

Wahl Bump-Free Rechargeable Foil Shaver

How We Test Products FAQ

1. How Do We Get Products For Testing? 

We use a number of different channels to acquire products for testing. The most common is requesting sample units from brands, who, more often than not, are more than willing to send them. If a product isn’t available as a sample unit, we’ll purchase it ourselves using a designated section of our site’s budget.

2. How Long Do We Test Products? 

The amount of time we test a product largely depends on the nature of the product itself. More expensive, high-ticket items require longer testing because we want to make sure they’re worth the investment. We test large pieces of fitness equipment, like spin bikes and fitness mirrors, for a month before we write our reviews.

Sometimes, we only have a limited amount of time with a product, if we’re at a launch party or press event. This means we occasionally have to form our opinions in just a few minutes or hours, but this is rarer.

Most of the time, we spend 1-3 weeks with a product, testing all of the features and compiling our data into a spreadsheet.

3. What Do We Do With Products After We’re Done Testing? 

When we coordinate with a brand about sending the sample unit, we also ask them if they’d include a return label with the product, so we can return it promptly after we’re done testing. Some brands request the unit be returned within a certain window, or after we’re done testing, while others encourage us to keep or donate the products after we’re done.

Some of our editors have kept sample units, and continue to test them. We frequently upgrade reviews after 3, 6, 9 and even 12 months of use, to give our readers an even more in-depth view of a product.

4. Do You Use a Number Scale?

Yes, we have an internal scoring system that enables us to compare products to one another as objectively as possible and informs our rankings.

5. What Do You Do If You Don’t Like a Product, Or Have a Bad Experience?

If we have a truly terrible experience with a product, we either don’t feature it on the side, or we include a watered down version of our thoughts. We try to be as honest with our readers as possible, but we also don’t want to trash a certain product or brand if it’s unnecessary.

Instead, we’ll find alternatives to the product, or move on and review something else. We’re more about promoting products we fully believe in and love, than picking apart everything we try.

6. How do you make your money?

SPY earns money when our readers trust our recommendations enough to purchase a product. When readers click a link on SPY and go on to purchase a product at sites like Amazon or Target, we may earn a commission on that purchase. This is standard practice for commerce publishers. Our full affiliate disclaimer appears on all of our content. It reads:

If you purchase an independently reviewed product or service through a link on our website, SPY.com may receive an affiliate commission.

In addition, we run advertising and sponsored content campaigns. We always label sponsored content using a “SPONSORED” label, and ads are always clearly marked. We do not accept money for reviews or guest posts under any circumstances.

 

Why Trust SPY?

If it isn’t already clear, we care about product testing and conducting honest reviews of the products we recommend. Our editors have fine-tuned a review process wherein we get the most accurate view of a certain product, and are able to pass those findings on to you, the reader.

We’ve been leaders in the e-commerce space for over five years, and our editors have developed deep knowledge bases of the product categories they specialize in. With each review we conduct (and we’re conducting tens of them a month), we’re gaining even more insider knowledge of the product categories we work within, the brands we work with and what it means for a product to be worth your money.

Stay tuned for more weekly updates from our editors on the products we love, the ones that’ll definitely make your life better, and the ones that won’t.

Interested in reading more? Here are a few of our editors’ favorite reviews published so far in 2022:

 

best instant film cameras Tyler Schoeber & George Chinsee | SPY.com

More About SPY

SPY’s Mission

Our mission is to be the best e-commerce destination for men on the web. This means we’re not only fully up-to-date on all the most recent, best products for men across all of our verticals, but we’re also testing those products ourselves to make sure they’re worth the hype.

Our Goal When Reviewing Products

Our #1 goal when reviewing products is to give you, the reader, the most accurate picture possible of what this product is like. At the end of our reviews, we want you to feel like you’ve held the product in your hands and have as many details as possible so you can make an informed purchase, or decide it’s not the right pick for you. We want to give you all the necessary objective information you need about the product — from its size and dimensions to its battery life, capabilities and price — as well as subjective opinions you can only form if you’ve actually used something.

What Sets SPY Reviews Apart

While other sites specialize in reviews only in the health space or only in the gadget space, we cover it all. We also don’t only leave you with two products to choose from at the end, and a lengthy, technical description of why we’ve eliminated all of the other choices. We craft our review round-up lists with plenty of users in mind, because almost every product is not one size fits all, and different brands are going to suit different sets of needs.

SPY’s History

SPY launched in the e-commerce world in 2017, and since then we’ve been modifying and improving our product review and recommendation process. We started doing in-depth product reviews in 2020, and have been putting more and more time and resources into them since then. Since that time, we’ve reviewed hundreds of products and spent thousands of hours observing the run times of gadgets, the sheen of moisturizers, the power of beard trimmers and user experience of blenders.

Here are some of our most popular recent reviews: