Search

What the Quote?

"I'm laughing 'cause I licked you, not because of what you said."

Laura Hearron

"Of course they still make Tang. Whaddaya think the astronauts drink?"

Steven Rodgers

"Where's the 'existing server'? Hold on, I don't know if 'redneck' has an I.P. address."

Steven Rodgers

« Easy way to add type-ahead to text fields in the Notes client | Main| Ohhhh, so that's where the other 48 GB went »

LotusScript isn't the only language that struggles with floating point

Category javascript
Jerry Carter recently mentioned that if you're doing raw calculations using floating point (i.e. using the data types Double or Single - which are just short for, respectively, double-precision floating point and single-precision floating point) numbers, you may not get the results you're expecting. A lively conversation ensued.

This morning, Ajaxian covered an article from Douglas Crockford that explains why JavaScript suffers from the same problem (they also hinted that Java does as well). Moral of the story: if your calculations must be precise, don't rely on floating point, no matter what language you're using (unless the language includes a specific data type that does the necessary logic for you automatically, such as the Currency type in LotusScript). Rather, nudge your scale up to the point that all operands involved are integers (i.e. multiply all of them by 100, 1000, etc.), perform the operation, then divide by the same scale to get the correct result.

Comments

Gravatar Image1 - I've gotten in the habit using BigDecimal in Java for anything involving money. It's quite surprising the number of bright people working in IT that don't know the pitfalls of working with floating point.

Gravatar Image2 - I'm thinking more and more that a BigDecimal LSX is something that I want to do, once the new version of the toolkit is released.

Gravatar Image3 - Oh how my heart yearns for BCD Emoticon

Post A Comment

:-D:-o:-p:-x:-(:-):-\:angry::cool::cry::emb::grin::huh::laugh::lips::rolleyes:;-)