Tag: precision

1225 Limiting floats to two decimal points 2009-01-18T18:16:41.153

786 decimal vs double! - Which one should I use and when? 2009-07-22T14:37:54.860

340 What is the difference between float and double? 2010-03-05T12:48:32.230

279 How do I print a double value with full precision using cout? 2009-02-16T18:15:18.240

229 JavaScript displaying a float to 2 decimal places 2010-07-02T03:21:24.700

202 How do I interpret precision and scale of a number in a database? 2010-03-04T05:54:01.350

191 Unexpected results when working with very big integers on interpreted languages 2013-08-04T18:46:25.513

183 Why is a round-trip conversion via a string not safe for a double? 2014-06-19T05:58:36.660

178 Get DateTime.Now with milliseconds precision 2013-04-16T08:41:56.690

146 Is floating-point math consistent in C#? Can it be? 2011-07-13T17:29:55.543

145 Why does adding 0.1 multiple times remain lossless? 2014-09-30T11:59:48.803

120 Retain precision with double in Java 2008-11-27T01:54:03.550

117 What's the difference between a single precision and double precision floating point operation? 2009-04-29T06:30:19.177

101 In which order should floats be added to get the most precise result? 2011-07-14T19:44:58.943

100 Difference between toFixed() and toPrecision()? 2010-07-26T18:35:41.867

88 How many double numbers are there between 0.0 and 1.0? 2010-06-05T02:51:23.187

86 C# DateTime.Now precision 2010-01-26T22:24:24.593

86 Controlling number of decimal digits in print output in R 2010-02-18T09:48:37.263

84 Why is 199.96 - 0 = 200 in SQL? 2018-07-20T12:32:34.083

82 Is it safe to check floating point values for equality to 0? 2009-01-27T20:50:14.620

71 PHP - Floating Number Precision 2010-09-16T12:39:59.903

67 In MATLAB, are variables REALLY double-precision by default? 2010-11-19T16:33:03.557

65 What range of numbers can be represented in a 16-, 32- and 64-bit IEEE-754 systems? 2009-05-16T14:37:41.893

63 Why does double in C print fewer decimal digits than C++? 2018-10-05T07:20:05.410

61 Adjusting decimal precision, .net 2009-07-15T17:24:52.313

59 Double precision floating values in Python? 2011-07-12T11:09:25.960

56 Why is 24.0000 not equal to 24.0000 in MATLAB? 2009-03-26T16:10:38.863

54 Set the display precision of a float in Ruby 2009-12-19T19:49:51.543

53 Are doubles faster than floats in C#? 2008-10-01T17:58:46.337

53 Emulate "double" using 2 "float"s 2011-07-20T23:53:55.357

53 Are all integer values perfectly represented as doubles? 2017-04-27T10:51:28.723

48 Convert Java Number to BigDecimal : best way 2013-04-25T13:36:20.183

46 Floating point equality 2018-07-02T10:26:35.347

45 Why do I see a double variable initialized to some value like 21.4 as 21.399999618530273? 2008-10-07T07:40:17.490

45 Precision of multiplication by 1.0 and int to float conversion 2012-11-15T15:31:02.173

44 Dividing two integers to produce a float result 2012-09-16T13:39:31.327

39 PHP7.1 json_encode() Float Issue 2017-03-23T16:11:14.420

38 BigDecimal, precision and scale 2016-02-16T14:53:31.167

37 sine result depends on C++ compiler used 2017-10-12T13:46:57.527

36 Changing precision of numeric column in Oracle 2012-02-10T19:49:58.317

34 Double precision - decimal places 2012-04-03T18:31:35.797

33 Find angle between hour and minute hands in an analog clock 2010-05-01T05:07:01.423

33 Is there any way to get current time in nanoseconds using JavaScript? 2011-05-14T15:29:57.770

32 How best to sum up lots of floating point numbers? 2008-12-26T19:40:09.170

32 Equals operator for zeros (BigDecimal / Double) in Java 2012-04-03T11:06:44.557

31 How to properly round up half float numbers in Python? 2015-10-08T15:11:43.340

30 Haskell: Force floats to have two decimals 2009-10-13T11:15:48.493

30 Why can't I get a p-value smaller than 2.2e-16? 2011-08-07T04:16:06.117

29 ArithmeticException thrown during BigDecimal.divide 2010-05-01T09:03:07.440

29 Half-precision floating-point in Java 2011-05-28T15:41:09.927

29 How to create a high resolution timer in Linux to measure program performance? 2011-07-19T15:20:24.977

28 How to print a double with full precision on iOS? 2011-04-20T10:43:20.123

28 C# double to decimal precision loss 2011-09-17T09:50:35.607

28 Is there a reliable way in JavaScript to obtain the number of decimal places of an arbitrary number? 2012-03-02T19:53:13.143

27 C# Converting 20 digit precision double to string and back again 2009-03-04T17:09:49.997

26 How to manually parse a floating point number from a string 2008-09-17T16:48:24.287

25 How can I make sure a float will always be rounded up with PHP? 2011-02-23T16:18:13.787

24 How to 'cout' the correct number of decimal places of a double value? 2010-11-18T17:02:37.627

24 What does "real*8" mean? 2012-05-09T17:08:45.000

24 Why does the Java increment operator allow narrowing operations without explicit cast? 2012-12-19T10:04:42.090

24 How to convert milliseconds to seconds with precision 2012-12-30T12:58:55.370

23 How to Java String.format with a variable precision? 2010-02-20T04:26:12.157

23 Loss of precision - int -> float or double 2010-05-06T12:35:00.337

23 How to convert to double with 2 precision - string after dot? 2011-02-14T20:49:30.103

23 How to display a number with always 2 decimal points using BigDecimal? 2013-01-25T05:01:19.277

23 High precision floating point numbers in Haskell? 2013-06-01T23:23:13.667

23 Changes to Math.Exp or double implementation in .net 4.5.2 2015-10-15T16:09:02.570

23 Does precision change slightly during casting int to double in Java? 2018-06-06T08:50:10.247

22 PHP and unit testing assertions with decimals 2009-12-26T10:30:39.467

21 What is the precision of long double in C++? 2009-01-24T16:08:20.817

21 fopen / fopen_s and writing to files 2010-04-04T16:48:49.073

21 Convert float to double loses precision but not via ToString 2011-05-13T03:28:39.947

21 Truncate a floating point number without rounding up 2011-11-12T18:43:07.093

21 How to calculate precision and recall in Keras 2017-03-28T17:53:43.010

20 What is the standard (or best supported) big number (arbitrary precision) library for Lua? 2008-11-13T23:10:22.700

20 Why do bean validation Min/Max constraints not support the double type? 2011-09-14T20:49:31.417

20 Different std in pandas vs numpy 2014-07-27T18:26:18.517

19 T-SQL Decimal Division Accuracy 2009-01-08T11:15:02.433

19 Representing integers in doubles 2009-04-17T06:00:44.010

18 What class to use for money representation? 2009-09-10T17:52:04.707

18 Rails: Cannot add :precision or :scale options with change_column in a migration? 2010-03-15T21:07:07.393

18 sampling multinomial from small log probability vectors in numpy/scipy 2015-11-16T15:01:51.540

18 why in matlab sin(pi) is not exact but sin(pi/2) is exact? 2016-07-22T05:42:53.110

17 Ensuring C++ doubles are 64 bits 2009-04-15T15:44:52.550

17 Why does a C# System.Decimal remember trailing zeros? 2010-06-08T11:17:02.827

17 Is there a floating point literal suffix in C++ to make a number double precision? 2012-08-30T20:41:02.770

16 "new BigDecimal(13.3D)" results in imprecise "13.3000000000000007105.."? 2009-01-20T10:36:26.307

16 Is there a way to get the "significant figures" of a decimal? 2010-09-10T10:01:21.843

16 How do I see high-precision query times in mysql command line? 2011-05-23T20:37:22.467

16 Exact decimal datatype for C++? 2013-03-10T07:30:16.573

16 Show two digits after decimal point in c++ 2013-04-29T13:37:01.870

16 Numpy: Difference between dot(a,b) and (a*b).sum() 2013-08-07T01:02:11.893

16 Golang floating point precision float32 vs float64 2014-03-11T21:50:20.980

16 Invertability of IEEE 754 floating-point division 2016-08-01T22:44:08.743

15 SQL Server decimal scale length - can be or has to be? 2013-04-05T19:53:23.687

14 Auto-Interval precision in MS Chart 2010-09-17T22:01:54.887

14 Are arithmetic operations on double variables holding integer values exact? 2011-03-11T12:21:43.233

14 gcc rounding difference between versions 2012-01-10T11:50:56.590

14 Precision of floats with printf 2012-04-01T08:51:29.197