Are Fit Parameters single or double precision?

jameslord · January 14, 2016, 2:56pm

I’m testing a user fit function now and as part of the debugging it prints out the parameters it’s working on, when evaluated.
I note that parameters are usually double precision (good). But when starting a fit, or re-starting a fit from the Fit Function pane to do more iterations, the values requested for the initial evaluation seem to be rounded to fewer decimal places than are seen in the dialog.
At the moment the rounding error is smaller than the errors of the fit, but that may not always be the case especially with sharp peaks centred at large X values or high frequency oscillations… It may be the reason why sometimes if a fit converges nicely (even with built in simple functions) and then I press “fit” again it gives slightly different numbers each time.

mantid_roman · January 15, 2016, 9:21am

I opened an issue (#14972) to have a look at it.

github.com/mantidproject/mantid

Function parameter precision.

opened 09:17AM - 15 Jan 16 UTC

closed 08:23PM - 01 Mar 21 UTC

mantid-roman

Stale

Parameters are usually double precision (good). But when starting a fit, or re-s…tarting a fit from the Fit Function pane to do more iterations, the values requested for the initial evaluation seem to be rounded to fewer decimal places than are seen in the dialog. At the moment the rounding error is smaller than the errors of the fit, but that may not always be the case especially with sharp peaks centred at large X values or high frequency oscillations.. It may be the reason why sometimes if a fit converges nicely (even with built in simple functions) and then I press "fit" again it gives slightly different numbers each time.