Each robot vacuum we consider for recommendation gets put through its paces in our test lab in Louisville, Kentucky. In addition to test floors where we run our controlled pickup tests, we monitor each robot vacuum in a special test room filled with mock furniture to gauge how well it navigates around common obstacles. Past that, we check each robot vacuum’s ability to gobble up pet hair without getting clogged or leaving loose strands behind, we consider mopping capabilities and we check to see how well it navigates fake dog messes, too.
Let’s dive a little deeper into the main considerations, starting with our performance tests.
Robot vacuum scoring metric
undefined
| Subrating category | Weight | What we looked for | |||||
|---|---|---|---|---|---|---|---|
| Performance | 30% | Performance score extrapolated from AVG(AVG_sand + AVG_blackrice) | |||||
| Value/price | 25% | Retail price rating considering all other features. i.e. Does this price seem fair for the value offered? Is it justified by performance, features, and NAV efficiency? | |||||
| Features | 15% | What features does the vacuum offer? Self-emptying station? Multiple batteries? NAV tech? Mopping? | |||||
| Runtime | 20% | Consider Navigation efficiency score (1-10), based on time taken to complete a full cleaning cycle in CNET LABS custom robot vacuum NAV testing room. | |||||
| Ease of use | 10% | UX – How easy/quikck was the setup experience? Did it come with smart home functionality? Smartphone app? Voice assist? |
Robot vacuum pickup power
When it comes to vacuuming prowess, we want to know how effective each robot is against common crumbs and other debris, and also how it fares against much smaller particles like dust, dirt and sand. To find out, we use dry, uncooked black rice as a stand-in for the crumbs and sand as an analog for finer particles.
In each case, we scatter a controlled amount across three test floors: low-pile carpet, midpile carpet and hardwood floors. Low-pile carpet is shorter, less plush carpet with shorter fibers, so typically robot vacuums have an easier time picking up from it (though not always). Midpile is softer, more plush carpet with taller fibers. It tends to be more challenging for robot vacuums (though again, not always). Then, we take the robot vacuum, thoroughly empty its dust bin, send it to clean the affected area and finally measure the weight of whatever it managed to pick up. That gives us a pickup percentage of the full amount. From there, we repeat each run two more times and average the results.
In recent months, we eliminated our test for black rice on hardwood floors since, more or less, every robot vacuum we tested was scoring near 100%. We now use the sand test as our primary benchmark in evaluating cleaning performance. We consider anything 50% and above to be a good score for sand.
Hardwood floor testing
Low-pile carpet testing
Midpile carpet testing
These overhead long exposure shots each show the trail of a Roborock S7 MaxV Ultra as it cleans our test room. We attach glow sticks to the top of the cleaner directly above the vacuum intake to get a sense of how much coverage the vacuum offers and how intelligently it navigates. In this case, the S7 MaxV Ultra is as thorough and consistent as robot vacuums get.
Robot vacuum navigation skills
Your robot vacuum will only clean your home as thoroughly as it’s capable of navigating it. The ideal cleaner will make easy work of finding its way from room to room and automatically avoiding obstacles along the way, all of which makes for proper, low-maintenance automated cleaning.
We make sure to observe each robot vacuum as it cleans to get a good sense of how well it navigates but to get the best comparison from cleaner to cleaner, we take overhead long exposure shots of each one as it cleans our darkened test room, with glow sticks attached to the top of each one directly above the vacuum intake. The images that result show us light trails that reveal the robot’s path as it navigates the room and cleans around our mock furniture.
Now, compare that to this next GIF, which shows you three runs from the iRobot Roomba Combo J7 Plus. Notice the difference? The Roomba was less effective at covering the entire room, missing the bottom-left corner in two out of three runs, and it had plenty of difficulty providing adequate coverage around the legs of that mock dining table, too.
In large part, it comes down to the tech at play. Over the years, we’ve consistently noted that robot vacuums that use laser-guided lidar navigation tend to be very good at mapping their environment and finding their way around. Meanwhile, 3D-mapping cameras with object recognition smarts can give robot vacuums the extra ability to identify and adapt to obstacles in their path. The Roborock S8 Pro Ultra uses both technologies, which helps explain why it performs so well here. Meanwhile, the Roomba relies on cameras and sensors alone, with lasers left out of the mix.
The iRobot Roomba J7 Plus delivered on its promise of steering clear of dog poop (fake or otherwise).
Still, those cameras definitely come in handy. Just watch the above GIF, which shows what happened when we put the iRobot Roomba J7 Plus to the test — specifically, its promise of identifying and avoiding pet waste. With a variety of (I assure you, fake) dog poop scattered about a small, enclosed test floor, the Roomba did its best to vacuum the area without touching any of them. It succeeded, never bumping into any of our disgusting-looking test turds at all.
The Samsung JetBot AI Plus consistently failed our solid pet waste test. In each run, it would eventually run over or push one of our dog poop models.
Now, compare that with the Samsung JetBot AI Plus, which also promises to use its cameras to spot and avoid pet droppings. The result was not great; in each test run, it would eventually bump into one of our test piles. Thank goodness they weren’t real.
