Last year it was revealed that Google and NASA were teaming up to build a quantum computing laboratory based on a system from D-Wave. The 512 qubit D-Wave 2 quantum computer has since been installed and Google was kind enough to recently share some benchmarking results from the machine.

In early testing, Google used random instances and put the machine against popular off-the-shelf solvers (Tabu Search, Akmaxsat and CPLEX). The search giant found the D-Wave system (at 509 qubits) was about 35,500 times faster than the best off-the-shelf solver which is certainly impressive. But it's worth pointing out that the competitors are simply general-purpose solvers and things certainly look different when an optimized system is used.

That is exactly what Google did next with the help of two world-class teams and the results are a mixed bag. The optimized machines were able to keep pace with the D-Wave and sometimes even outperform it but the inverse is also true: the quantum system sometimes beats the optimized machines.

The team believes that classical solvers are still competitive right now because the qubits in the D-Wave are still only sparsely connected. As the connectivity in future versions of quantum annealing processors gets denser, traditional systems will be much less effective.

In the interim, the quantum team is building up its datasets and note that it is easy to make premature conclusions on such small sets. With more than 400,000 problem instances now on hand, they are trying to identify a class of problems for which the current quantum hardware might outperform all known classical solvers.