Quantcast
Channel: Intel® oneAPI Math Kernel Library & Intel® Math Kernel Library
Viewing all articles
Browse latest Browse all 2652

New Floating Point Math Error in Subtraction for Intel Core Processors?

$
0
0

To duplicate type the following in to an Excel spreadsheet.

=(6.377-6.376)*1000

I get: 0.999999999999446 if I allow the decimal places to be shown. That is three significant digits that are in error, way too many for rounding error.

 

I get a true rounding  error using Google (0.99999999999)

 

I understand rounding error, but look at the significant digits and this occurs in subtraction. Second of all there are only four significant digits to begin with there should not be any rounding error at all.

 

The error exists before multiplying by 1000, that is done, just for convenience.

 

This error occurs on the latest generation of Intel Processors (i7-8550U as well as my older i7-4770).

 

I had to track down this error from a 1 > 1 problem (when the number was put into a logical statement it failed).


Viewing all articles
Browse latest Browse all 2652

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>