Fitting
- Fitting a model
- Automatic decomposition
- How many curves
- Minimize Number of Curves
- Minimize Difference
- Stop
- Why a fit command is greyed
- The difference (R-factor)
- Max Acceptable Difference
- Curve scaling
- Minimizer
- Loss Function
- Weighting
- What the fit will actually do
- When a fit ends
- Samples no curve covers
- Is the difference only noise?
- Compute Server
- Compute backends
- Setting up the Python engine
Fitting a model
This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.
A fit adjusts the parameters of the model's curves until the calculated profile matches the measured one as closely as it can.
A fit needs a profile to fit and a curve type to fit it with. Everything else - where the curves start, which stretches of the profile are scored, the background - can be prepared by hand or left to the program, which fills in whatever is missing before it starts.
There are three ways to start a fit, all in the Fit menu:
- Fit > Automatically puts a background curve in the model where the build has one, places curves where the data need them, marks fit intervals and then removes every curve the data does not need. Use it for a first answer to "how many curves, and where?".
- Fit > Minimize Number of Curves (F4) runs the same removal on the curves and intervals you already have.
- Fit > Minimize Difference (F5) fits the model as it stands and removes nothing. The Fit button in the Fit group of the Tools tab runs the same command.
While a fit runs, the chart area shows its progress and every command that would change the model is greyed. Fit > Stop, or the Stop button of the Tools tab, ends it.
The fit itself runs in the compute server the window is connected to (Fit > Compute Server), not in the window. What it makes as small as possible is chosen under Fit > Loss Function, and the algorithm that does it under Fit > Minimizer.
Limitations
- A fit refines the curves from where they start. It finds the best match near that start, which is not always the best match there is: curves started far from the peaks can settle on a poorer answer.
See also: Automatic decomposition · Minimize Difference · Why a fit command is greyed · The difference (R-factor) · The Tools tab · Fit progress
Automatic decomposition
This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.
Fit > Automatically decides how many curves the profile needs: it starts with far more than any profile needs and removes every one the fit can do without.
Fit > Automatically - also the last button of the toolbar above the chart - runs five steps in the compute server:
- 1. Where the build has background shapes, it puts a background curve in the model - a Quadratic background, unless you chose another - to be fitted with the peaks. Nothing is subtracted from the profile. With the background variation on, the variation is the background and no curve is added; the status line says so. Background points you had picked are discarded. The background's own chapter of this guide says what each of these is.
- 2. It places a curve on every point of every peak. A peak is found by starting at the highest point not yet taken and walking down both of its flanks; the search stops at the first peak whose top is lower than a fraction of the highest point - 1/30 by default, the background fraction. The search looks at the copy with the background taken off, so a high baseline is not mistaken for peaks. If you placed curve positions yourself, those are used instead.
- 3. It marks a fit interval around each peak, replacing any intervals you had marked. When that changes them, it says so when it ends - in a dialog and on the status bar - naming your intervals and its own, and the points yours fitted that are no longer fitted.
- 4. In each fit interval separately, it fits the curves and then removes every one the fit can do without, by the rule chosen under Fit > Remove Curves By. Which curve goes, and when it stops, is the subject of "How many curves".
- 5. It fits the curves that are left to full precision, together with the background curve, over the profile as measured. Steps 2 to 4 work on a copy of the profile with the background taken off, the way Subtract > Automatically would take it; the copy is discarded and the profile you see is never changed.
Every curve is of the curve type selected on the Tools tab. When the run ends, the chart, the tables and the Model panel show the curves that remain. The removed curves are gone together with the positions they were placed from, so a later edit or fit does not bring them back.
The setting that decides the answer is that rule. By BIC, the default, the data decide; by Max Acceptable Difference, the limit you set does - ask for less accuracy and you get fewer, broader curves.
It is offered as soon as a profile is open, including before the background has been dealt with - which is exactly what its first step does.
A curve type placed from its own markup is not decomposed as peaks. With nothing marked, its pack finds and places the instances in the data as measured, one fit interval each, and they are fitted; then, as for peaks, the instances that no longer pay for themselves on the fitted model are removed by the Bayesian information criterion, smallest first, and the rest fitted again - though the pack decides which of its instances may go. With a markup already there, the run only fits it, so what you corrected or deleted by hand stays as you left it. No background curve is added to a model whose type takes none. The pack's guide says how it finds them.
Limitations
- The fit intervals and background points you had marked are discarded by steps 1 and 3; only the curve positions you picked are kept. Use Fit > Minimize Number of Curves to keep your intervals. A point your intervals fitted and the run's do not is left out of every figure, so two models compared over different points can differ by far more than their fit.
- The copy steps 2 to 4 work on has the background taken off the way Subtract > Automatically would, which assumes a bowl-shaped background. On a background of another shape a curve may be placed on the baseline's bumps, or a small peak missed; the background curve is fitted in step 5 either way.
See also: How many curves · Max Acceptable Difference · Minimize Number of Curves · Fit progress
How many curves
Convention — Common practice or a guideline in the field, not a rule its sources require.
Fit > Remove Curves By chooses how the program decides how many curves a profile needs: by BIC, the default, or within a maximum difference you set.
With enough curves any profile can be fitted perfectly, noise included, so a least-squares fit alone cannot say how many curves there are. Fit starts with more curves than any profile needs and removes them one at a time; the rule decides which removals are kept.
- Fit > Remove Curves By > BIC, the default for new work. A removal is kept when the Bayesian information criterion drops: chi-squared + k ln n, k the parameters varied and n the points of the fit interval. Each difference is divided by the noise measured from the data itself, which grows with the measured value as counts do. A curve stays exactly when it improves the fit by more than its parameters cost, so no limit is needed and Fit > Set Max Acceptable Difference is greyed.
- Fit > Remove Curves By > Max Acceptable Difference. A removal is kept while the R-factor of the fit interval stays within the limit set with Fit > Set Max Acceptable Difference: the precision you claim for the measurement, rather than a figure derived from the data.
The choice is saved in the project file. A project saved before the choice existed reopens with Max Acceptable Difference, the rule it was made with, so its results do not change.
Each fit interval is reduced on its own. Under either rule a curve whose amplitude has fallen below a thousandth of the largest in the interval is dropped at once, and candidates are taken in two passes: first the curve standing where the measured profile is steepest - on a flank it is often doing the work of a shoulder its neighbours can take over - then the curve with the smallest amplitude. A pass also ends when one curve is left, and a final fit polishes what survived.
By Max Acceptable Difference, a removal that breaks the limit is re-fitted, and if the fit still cannot meet it the curve goes back and the pass ends. Each re-fit stops as soon as the limit is met again, which keeps the method quick.
By BIC, a pass works in sweeps. Each candidate is judged as the model stands, its amplitude handed to the nearest curve; every removal that lowers BIC is kept, and what is left is re-fitted once before the next sweep. When a sweep removes nothing, the five candidates that came closest are each tried with a full re-fit, and the pass ends when none of them lowers BIC. Re-fitting after every candidate, as the limit does, would take hours on a crowded start.
Limitations
- The noise is measured from the data as a straight line in the measured value, from how much neighbouring points scatter. A peak only a few samples wide adds its own curvature to that measure, and BIC then keeps fewer curves on it than it might.
- The fit itself minimises the plain difference, while BIC divides each difference by the noise where it is, so a re-fit can leave BIC slightly higher than before it.
- Curves are removed one at a time, so either rule ends with a small model that meets it, not necessarily the smallest one that could. A curve shape that does not match the data can make BIC keep two curves where one peak stands.
- By Max Acceptable Difference, if the first fit of an interval does not meet the limit, no curve is removed from it; the difference compared with the limit is always the R-factor of the interval, whichever loss function is selected.
- The steepest-slope pass measures the slope from a curve's starting position to the next point of the profile, so it cannot remove a curve that starts on the profile's last point.
Sources
- Schwarz, G., "Estimating the dimension of a model", The Annals of Statistics 6(2), 461-464 (1978) https://doi.org/10.1214/aos/1176344136
- Foi, A. et al., "Practical Poissonian-Gaussian noise modeling and fitting for single-image raw-data", IEEE Transactions on Image Processing 17(10), 1737-1754 (2008) https://doi.org/10.1109/TIP.2008.2001399
- Riener, M. et al., "GaussPy+: A fully automated Gaussian decomposition package for emission line spectra", Astronomy & Astrophysics 628, A78 (2019) https://doi.org/10.1051/0004-6361/201935519
See also: Automatic decomposition · Max Acceptable Difference · Weighting · The difference (R-factor)
Minimize Number of Curves
This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.
Removes every curve the fit can do without, starting from the curves and fit intervals you already have.
Fit > Minimize Number of Curves (F4) runs steps 4 and 5 of Fit > Automatically - the removal passes and the final fit - on the model as it stands. It does not touch the background, and it keeps the positions and fit intervals you placed.
What it needs is filled in only where it is missing: with fewer than two interval bounds it marks a fit interval around each peak, and with no curve positions it places a curve on every point of every peak, as the automatic run does.
Use it when you have placed more curves than you think are needed and want the data to decide which of them stay, by the rule chosen under Fit > Remove Curves By.
Curves it removes are gone together with their positions.
Limitations
- It is greyed until the background has been subtracted or you have started building the model, and for a curve type placed from its pack's own markup; see "Why a fit command is greyed".
- On a profile with many points, "a curve on every point of every peak" can be a great many curves, and the first fit then takes correspondingly long. Placing positions first avoids it.
See also: How many curves · Automatic decomposition · Why a fit command is greyed
Minimize Difference
This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.
Fits the model as it stands: every curve is kept and only their parameters change.
Fit > Minimize Difference (F5), and the Fit button in the Fit group of the Tools tab, adjust the parameters of every curve until the selected loss function can be made no smaller. No curve is added or removed, and the maximum acceptable difference plays no part.
Anything missing is filled in first, as for the other fit commands: a fit interval around each peak when fewer than two interval bounds are marked, and a curve on every point of every peak when there are no curve positions.
Each fit interval is fitted as a problem of its own. The difference reported afterwards pools all of them into one figure.
Use it after placing curves by hand, after changing a setting such as the loss function or the engine, or to fit again from where the last fit left off.
Limitations
- It is greyed until the background has been subtracted or you have started building the model; see "Why a fit command is greyed".
- Without curve positions it seeds a curve on every point of every peak, which on a finely sampled profile is a great many curves. Place positions first, or use Fit > Automatically.
See also: Fitting a model · Loss Function · Minimizer · When a fit ends
Stop
This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.
Ends the fit or other long computation that is running.
Fit > Stop, and the Stop button in the Fit group of the Tools tab, are available only while a computation runs; the rest of the time they are greyed.
The engine is told to end, and does so at its next step. The window then shows what the fit had reached, exactly as it does when a fit finishes by itself: the curves, the tables and the statistics in the status bar are refreshed.
Stopping an automatic decomposition ends its passes where they are: curves already removed stay removed.
Closing the window while a computation runs stops it the same way. The status bar says so, and the window closes once the computation has ended - after asking, as it always does, whether to save the project, which now holds what the computation reached.
Limitations
- A stopped fit has not converged. Its numbers are the best found so far, not the best the model can do; run Fit > Minimize Difference to carry on from there.
See also: When a fit ends · Fit progress
Why a fit command is greyed
This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.
The fit commands are offered only when there is something to fit, the background has been dealt with, and nothing else is running.
The window decides which commands it offers from the state of the compute server. The rules, in the order they apply:
- Nothing open: every fit command is greyed until a profile is imported or a project is opened.
- A computation running: only Fit > Stop is offered. Every command that would change the model waits until it is over, so that nothing is changed under a fit.
- Background not yet dealt with: right after a profile is imported or reloaded, or a point of it is edited, Fit > Minimize Number of Curves, Fit > Minimize Difference and the Tools Fit button are greyed and Fit > Automatically is the fit on offer. Fitting curves to a profile that still carries its background is something to ask for deliberately, not to reach by the ordinary command. They become available once the background has been subtracted or you start building the model - placing curve positions or fit intervals.
- Nothing running: Fit > Stop is greyed.
- A curve type placed from its pack's own markup: Fit > Minimize Number of Curves is greyed, and refused if asked for. Its instances are whole patterns, not peaks to remove one by one; Fit > Automatically decides what such a model holds.
Within the settings, single entries can be greyed with a reason in their hint, shown in the status bar:
- An engine under Fit > Minimizer that fits by evaluating a formula, when the selected curve type has none: "This curve type has no formula, so it is fitted by the native engine."
- An objective under Fit > Loss Function that cannot be used with the selected curve type, with the reason.
- Fit > Set Max Acceptable Difference, while Fit > Remove Curves By > BIC is chosen: nothing reads the limit then.
Fit > Enable Curve Scaling and Fit > Compute Server are never greyed: they are settings, and a changed setting applies to the next fit.
Limitations
- The compute server itself would accept a manual fit before the background is subtracted and complete what is missing; it is the window that does not offer it. A program talking to the server directly is not held to this rule.
See also: Fitting a model · Automatic decomposition · What the fit will actually do
The difference (R-factor)
This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.
The number Fit reports for how well the model matches the data is an R-factor: the squared differences divided by the square of the summed data.
Throughout the program "difference" and "R-factor" name the same number. Over the points inside the fit intervals it is
R = sum( (s * calc - obs)^2 ) / ( sum obs )^2
where obs is a measured value, calc the model's value at the same point, and s the curve-scaling factor of the interval (1 when Fit > Enable Curve Scaling is off). With several fit intervals, the sums are pooled over all of them before dividing, so the figure means the same however the profile is divided up.
Dividing by the data makes it a relative measure: the same sample measured for ten times as long gives the same R-factor. A perfect fit gives 0. It is shown as a plain number, such as 7.7E-6 - not in percent - while Fit > Set Max Acceptable Difference asks for its ceiling in percent.
It is the default loss function, so it is usually also what the fit minimises. Whatever loss function is selected, the number reported and the number compared with the maximum acceptable difference is this R-factor.
While a fit runs, it is shown in the status bar and above the progress chart. When the fit ends, the status bar shows the reduced chi-squared and R-squared of the result in its place.
In diffraction the name R-factor usually means the profile R-factors of Rietveld refinement: Rp, the summed absolute differences over the summed data, and Rwp, the square root of the weighted squared differences over the weighted squared data. Fit's number shares their intent - a difference relative to the data - but is neither: it is unweighted, squared, and has no square root, so it is not directly comparable with an Rp or Rwp quoted elsewhere.
Limitations
- Only the points inside the fit intervals count. Parts of the profile outside every interval can be fitted badly without changing the number.
- Every point inside them counts, including one no curve of the model reaches - the model is 0 there, and that one point can outweigh all the others. Fit says when that happens; see "Samples no curve covers".
Sources
- McCusker, L. B., Von Dreele, R. B., Cox, D. E., Louer, D. and Scardi, P., "Rietveld refinement guidelines", Journal of Applied Crystallography 32, 36-50 (1999) https://doi.org/10.1107/S0021889898009856
See also: Max Acceptable Difference · Loss Function · Curve scaling · Samples no curve covers
Max Acceptable Difference
This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.
The ceiling on the R-factor that decides how many curves are kept while Fit > Remove Curves By > Max Acceptable Difference is chosen.
Fit > Set Max Acceptable Difference opens the Max Acceptable Difference dialog, which has one field: "Max acceptable difference (percent):". Type the ceiling in percent, with a full stop as the decimal separator, and press OK. The default is 0.01, meaning an R-factor of 0.0001.
Anything that is not a number is refused with "Please enter a valid number (for example, 1.5)." and the dialog stays open. Cancel leaves the ceiling as it was.
The ceiling is used by Fit > Automatically and Fit > Minimize Number of Curves while curves are removed by it: a curve is removed only if the fit interval it belongs to still meets the ceiling without it. Under Fit > Remove Curves By > BIC nothing reads it, and the entry is greyed. Fit > Minimize Difference never uses it.
Choose it near the level the noise of your data allows. Too low, and noise is fitted with extra curves; too high, and real components are merged away.
It is saved in the project file, so a project reopens with the ceiling it was fitted with.
Limitations
- The ceiling is compared with the R-factor, whichever loss function is selected, and with the R-factor of each fit interval on its own rather than of the whole profile.
See also: How many curves · The difference (R-factor) · Automatic decomposition
Curve scaling
This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.
With curve scaling on, the model is multiplied by one factor per fit interval so that its area matches the data before the two are compared.
Fit > Enable Curve Scaling is a tick, and it is on by default. While it applies, the built-in engine compares s times the model with the data, where s is the sum of the measured values in the fit interval divided by the sum of the model's. The curves then only have to get the shape right, and the factor takes care of the overall height, which helps the fit converge.
It does not apply in two cases, whatever the tick says:
- With the Python engine selected: it fits the amplitudes itself, so scaling afterwards would rescale values already fitted.
- With a curve type that sets its own amplitude, free to grow. A second, overall multiplier would let the fit flatten the shape while the multiplier absorbs the difference. Hovering over the status bar explains it for this case.
The setting is saved in the project file.
Limitations
- Combined with the R-factor (legacy) loss function, scaling lets a fit lower the number by inflating the model rather than by matching the data; that combination is refused for curve types whose amplitude is free.
See also: The difference (R-factor) · Loss Function · What the fit will actually do
Minimizer
This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.
Fit > Minimizer chooses the algorithm that searches for the best parameters; the first one, the default, needs nothing extra.
The entries of Fit > Minimizer are the engines this build offers, in order. The first is the default, and is also what a saved choice falls back to when it names an engine this build does not have. The choice is remembered between sessions.
- Fit > Minimizer > Downhill Simplex (native) (the default): The original algorithm. Needs no Python and fits any curve type, including those with no formula.
- Fit > Minimizer > Levenberg-Marquardt (Python/lmfit): Trust-region least squares with uncertainties. Needs the Python sidecar, and a curve type that has a formula. It runs in a Python process the compute server starts on first use. While it is selected, Fit > Weighting is shown. Curve scaling does not apply to it.
- Fit > Minimizer > Downhill Simplex, curve by curve (native): The same simplex over one curve's parameters at a time, widest curve first, repeated until a pass gains nothing. Faster on models of many curves that each shape their own stretch; curves that trade off against each other may settle less well than under the simplex over everything.
- Fit > Minimizer > Downhill Simplex, restarted (native): The same simplex over every parameter, started again from its own best, stepping each parameter forward or backward at random, until a run improves the loss by less than one percent. Never worse than the simplex, which is its first run; slower, and often a good deal better.
Downhill Simplex (native) is the Nelder-Mead simplex method. It needs only the value of the loss function, not its derivatives, so it fits any curve type, including those that compute their points without a formula. It gives no uncertainties for the parameters.
Levenberg-Marquardt (Python/lmfit) uses lmfit's least_squares method - despite the name, scipy's Trust Region Reflective solver. It reports an uncertainty for each fitted parameter, shown after the value in the Curve Attributes table as "value ± error".
Downhill Simplex, curve by curve (native) runs the same simplex over one curve's parameters at a time: each curve that shapes only its own stretch of the data is a block of its own, and everything that shapes the whole interval - curves that are never zero, such as a Gaussian, and any parameter the curves share - is one block together. The widest block goes first, and the blocks are fitted again until a pass improves the loss by less than one percent; then one simplex over every parameter finishes from there. A model whose curves all shape the whole interval is one block, and gets exactly what Downhill Simplex (native) gives it.
Downhill Simplex, restarted (native) runs the simplex over every parameter, then builds a fresh simplex around the best point it reached and runs again, until a run improves the loss by less than one percent. Its first run is exactly Downhill Simplex (native), so it never ends above it. A simplex stops when its corners agree, which can be on a ridge rather than at the bottom; starting again looks around once more. Each later run steps each parameter forward or backward at random, so every parameter is tried from both sides - the simplex's own restarts only ever turn the first two. The random choices come from a fixed seed, so fitting the same model again gives the same result.
An engine that fits by evaluating a formula is greyed while the selected curve type has none, and if it was selected, the default engine is selected instead. An engine that can only minimise a sum of squares leaves a loss function that is not one to the built-in engine; a dialog and the status bar say so.
Without Python: an installed copy carries the Python engine with it and needs nothing more. A copy built from source needs Python and three libraries set up on the machine that runs the compute server (Help > Compute Backends says how). If they are missing, a fit with that engine does not start: the server answers that the Python backend is not available. Nothing switches engines behind your back; choose Downhill Simplex (native), which needs no Python, to fit.
Limitations
- No engine searches globally. Each refines the curves from where it starts, and can stop at a poorer answer when the start is far from a good one.
- Curve by curve is this software's own choice, offered beside the simplex to be compared on real tasks, not proven better. On wave-pattern fits it reached a lower loss than the simplex in several times the time; curves that trade off against each other are reconciled pass by pass, which is slow when they are strongly coupled.
- Restarted is this software's own choice too. On wave-pattern fits and on separate peaks it reached about half the simplex's loss in two to four times the time; how much a restart gains depends on the model, and nothing promises it gains anything.
Sources
- Nelder, J. A. and Mead, R., "A simplex method for function minimization", The Computer Journal 7(4), 308-313 (1965) https://doi.org/10.1093/comjnl/7.4.308
- Newville, M. et al., "LMFIT: Non-Linear Least-Squares Minimization and Curve-Fitting for Python" https://doi.org/10.5281/zenodo.11813
- Branch, M. A., Coleman, T. F. and Li, Y., "A subspace, interior, and conjugate gradient method for large-scale bound-constrained minimization problems", SIAM Journal on Scientific Computing 21(1), 1-23 (1999)
See also: Compute backends · Weighting · What the fit will actually do · The Curve Attributes table
Loss Function
This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.
Fit > Loss Function chooses the number a fit makes as small as possible; the default, R-factor, suits almost every fit.
Fit > Loss Function is the last entry of the Fit menu. Its entries are the objectives this build offers, the default first; the choice is remembered between sessions. Each entry's hint says what it measures:
- Fit > Loss Function > R-factor (the default): Squared deviations normalised by the data, so values are comparable between datasets.
- Fit > Loss Function > R-factor (legacy): The original objective. Normalised by the model rather than by the data, so with curve scaling on it can be lowered by inflating the model. Kept for comparison. It is not a sum of squares, so a fit with the Python engine runs on the built-in engine instead. It is greyed for a curve type whose amplitude is free to grow.
- Fit > Loss Function > Sum of squares: Plain least squares, with no normalisation at all - the value scales with the data, so compare it only within one dataset.
- Fit > Loss Function > Relative deviation: Absolute deviations as a fraction of the data - reads directly as "the model is off by this much". It is not a sum of squares, so a fit with the Python engine runs on the built-in engine instead.
R-factor and Sum of squares differ only by a constant factor, so they find the same parameters; only the number differs. Relative deviation sums absolute rather than squared differences, so a few large misfits count for less. R-factor (legacy) divides by the model instead of the data: with curve scaling on, a fit can lower it by inflating the model without matching the data any better, which is why it is greyed for a curve type whose amplitude is free, and replaced by R-factor if it was selected.
Whatever is chosen, the difference the program reports, and compares with the maximum acceptable difference, is the R-factor. The statistics shown after a fit do not depend on the choice either; only the fitted parameters do.
Limitations
- Every objective scores only the points inside the fit intervals.
See also: The difference (R-factor) · What the fit will actually do · Curve scaling
Weighting
Convention — Common practice or a guideline in the field, not a rule its sources require.
Weighting decides how much each point counts in a fit by the Python engine; counting data are conventionally weighted by one over the square root of the count.
The Fit > Weighting submenu is shown only while an engine that can be weighted is selected under Fit > Minimizer - the Python one. The built-in engine always fits unweighted, so the menu is hidden under it. It has two entries:
- Fit > Weighting > Poisson (counting statistics), the default: each difference is divided by the square root of the measured value, with values below 1 taken as 1. This is the usual choice for counts - of photons, neutrons, electrons - whose statistical error is the square root of the count, so a faint feature counts for as much as its precision allows.
- Fit > Weighting > None (unweighted): every point counts equally, the same objective the built-in engine minimises. Use it for data that are not counts, or to compare the Python result term for term with the built-in one.
The choice is remembered between sessions. The reduced chi-squared shown in the status bar after any fit uses the Poisson weights, whichever engine ran.
Limitations
- Poisson weighting assumes the values are counts. Values that have been rescaled, normalised or had a background subtracted no longer have the square root of the value as their error, and the weights are then only approximate.
Sources
- McCusker, L. B., Von Dreele, R. B., Cox, D. E., Louer, D. and Scardi, P., "Rietveld refinement guidelines", Journal of Applied Crystallography 32, 36-50 (1999) https://doi.org/10.1107/S0021889898009856
See also: Minimizer · Compute backends
What the fit will actually do
This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.
When a choice cannot be honoured, the fit changes it and says so: always in the status bar, and in a dialog when you make the choice.
The right-hand end of the status bar always says what the next fit will do - for example "Minimising R-factor." or "Fitting with the built-in engine, minimising Relative deviation." When something was changed, hovering over the status bar shows why.
Three things can be changed. The compute server decides with the same rule the status bar reports, so the two cannot disagree:
- The loss function: R-factor (legacy) is replaced by R-factor for a curve type whose amplitude is free to grow.
- The engine: with the Python engine selected, the fit runs on the built-in engine when the curve type has no formula, or when the loss function is not a sum of squares. Your objective is kept; what you lose is the per-parameter uncertainties.
- Curve scaling: switched off for a curve type that sets its own amplitude.
When you change the engine, the loss function or the curve type and the result would be changed, a dialog titled "About this fit" explains it. It is not repeated while you stay with that selection, and not shown at start-up for a setting chosen earlier. Curve scaling being switched off never opens the dialog; hovering over the status bar explains it.
See also: Loss Function · Minimizer · Curve scaling · The status bar
When a fit ends
This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.
The chart, the tables and the Model panel are refreshed from the result, and the status bar shows how long the fit took and how good it is.
While a fit runs, the R-factor tab of the chart area draws its progress - and, with View > Animation Mode ticked, the Model tab the model moving - and the tables keep the last finished result. The R-factor curve is kept with the model the run reached, saved in the project, and drawn again whenever that model is opened.
When it ends, whether it finished or was stopped, the window:
- draws the profile again with the model over it: each curve, their sum (Total Amplitude) and the Difference between data and model;
- draws the curve positions you picked and the fitted positions where the fit put the curves, and the fit interval bounds;
- refills the Curve Positions and Fit Intervals tables, the Curve Attributes table with every curve's parameters, and the Summary table with the data, the model and each curve at every point of every interval;
- rebuilds the list of curves in the Model panel;
- shows in the status bar the elapsed time and, when they can be computed, the reduced chi-squared and R-squared of the fit - and, when some points inside the fit intervals are covered by no curve, says how many, with a dialog naming them (see "Samples no curve covers").
The elapsed time counts only the time the computer was awake. A fit left running while the machine slept or hibernated does not count the sleep, and neither does the time axis of the progress chart.
The result is part of the project from then on, and is kept on disk when the project is saved. It is also recorded in the History tab, beside every model an earlier fit reached, so it can be made current again after later changes.
A model put back by value - a project reopened, or a model made current from the History tab - is measured as it comes back, so its R-factor is the one it was recorded with. A model that was never fitted reads "Not calculated", as it did while it was being edited.
Limitations
- The R-factor itself is on screen only while the fit runs; when it ends, the status bar shows the reduced chi-squared and R-squared in its place.
See also: Fit progress · Animation Mode · The status bar · The difference (R-factor) · Samples no curve covers
Samples no curve covers
This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.
A point inside a fit interval that no curve of the model reaches is modelled as 0, still counts in the R-factor, and is named where the result is shown.
Some curves exist only over a stretch of the data and are exactly zero outside it - a wave pattern, for instance, runs from its first point to its last. A Gaussian or a Lorentzian is never exactly zero, so it always reaches every point of its fit interval.
Where no curve reaches a point inside a fit interval, the model is 0 there, and the point still counts in the R-factor and in every other figure. On data that sits far from zero - a price series near 1000, say - that one point can outweigh every other difference together: one point left at the start of a series once made a model score seventy times worse over the whole profile than over two intervals that began one point later.
Fit does not drop such points from the figures. A number that changed with how the model happens to be placed would no longer compare one model with another. It says instead where they are. When a fit ends, a dialog gives how many there are, where, and how much of the squared difference between the data and the model they make; the status bar says how many, and the History entry of the model keeps the whole of it.
Before a fit starts - Fit > Minimize Difference or Fit > Minimize Number of Curves - the same is asked first, while it can still be corrected: Yes fits the model as it is, No leaves it untouched so the intervals or the curves can be changed. Fit > Automatically is not asked: it replaces the intervals with its own, and says when it ends what that left out.
The question and the dialog appear once for the same points between them. A later fit that leaves the same points uncovered says so only in the status bar and in History; points elsewhere, or the same ones coming back after they were covered, bring the dialog back.
Two ways forward. To measure only what the model describes, narrow the fit interval to the stretch the curves cover (Model > Fit Intervals > Start Manual Selection, or the Fit Intervals table). To describe the points as well, extend the model over them - move or add a curve so that one reaches them.
Limitations
- A background the fit varies is part of the model at every point, so with one on no point counts as uncovered, whatever the curves reach.
- The points are named from the model as it stands. A project saved by an earlier version has no record of them, so its History entries say nothing about them until a fit runs again.
See also: The difference (R-factor) · When a fit ends · The status bar
Is the difference only noise?
Canonical — Stated by the field's authoritative sources, which the explanation quotes and cites.
A model that explains the data leaves a difference that is noise; after every fit the Ljung-Box test asks whether it is, and the wavelet transform names where it is not.
The R-factor says how large the difference between the data and the model is, not what it is made of. A model can score well and still leave a wave or a peak it does not describe; what is left then follows itself - each point of the difference says something about the next. Noise does not.
After every fit the difference over each fit interval is tested whole. Each point is first divided by the noise expected where it is - the same estimate the automatic reduction weighs curves with - so a quiet stretch and a loud one count alike. The test is Ljung and Box's: the difference's first autocorrelations, up to ten of them or one for every five points, squared and summed. The intervals' tests are combined by Fisher's method into one p.
WHERE it is not noise is found point by point, by the wavelet transform (Torrence and Compo): a Fourier transform over a window that grows with the rhythm looked at, from four points up. At each point it asks how much rhythm of each length the difference holds there, against what noise would hold, and the chance noise holds that much is known exactly. Every point of every fit interval is judged together, held to a false discovery rate of 5 % (Benjamini and Hochberg): tested one by one at 5 %, a thousand points of pure noise would name fifty.
The status bar says "Difference: random (p = ...)" or how many stretches are not, the chart shades those stretches, and the model's History entry keeps the verdict.
Under the model's chart, the Difference structure strip draws that judgement point by point: how much rhythm the difference holds at each point of the fit intervals, against what noise would hold there - 1 for noise. Where the curve reaches the dashed line the point is named, and those are the shaded stretches above it; a spike is a place the model has something left to describe. The difference itself is drawn in the strip too, against its own axis on the right, so a spike of the curve sits over the stretch of the difference that made it. The strip moves and zooms with the chart above it.
A module included in the build may draw the same transform as a chart of its own, and its chapter says how to read it.
A shaded stretch is where the model has something left to describe: a curve to add, one to move, or a fit interval that cuts a feature in two.
Q = n(n + 2) sum_(k=1..h) r_k^2 / (n - k), which for white noise follows a chi-square of h degrees of freedom.
Limitations
- A test is not a proof. It sees what makes the difference follow itself; a difference that changes only in size from one stretch to another can pass, and a wrong estimate of the noise moves every verdict.
- The standardising, and starting the wavelet at a rhythm of four points, are this software's choices: a shorter one is not a wave a pattern can draw, and the wavelet that short passed noise more often than its test allows. An interval of fewer than ten points is not tested; one of fewer than twenty is not searched for where.
- A false discovery rate of 5 % allows some of what is named to be noise: with a large stretch named, a few short ones elsewhere may be chance. And a shaded stretch reaches beyond the feature that made it - up to two or three of its rhythms, as far as the windows of the longer rhythms it registers in - while near the ends of a fit interval only short rhythms can be judged at all.
- A cell of the map is tested on its own, at 5 %: in a large map of noise, one cell in twenty passes by chance, scattered. A patch spanning several places and rhythms is what to read, and the shaded stretches - judged together - are the places named.
Sources
- G. M. Ljung and G. E. P. Box, On a measure of lack of fit in time series models, Biometrika 65 (1978), pp. 297-303
- Y. Benjamini and Y. Hochberg, Controlling the false discovery rate, J. R. Statist. Soc. B 57 (1995), pp. 289-300
- C. Torrence and G. P. Compo, A practical guide to wavelet analysis, Bull. Amer. Meteor. Soc. 79 (1998), pp. 61-78
- J. P. Imhof, Computing the distribution of quadratic forms in normal variables, Biometrika 48 (1961), pp. 419-426
- R. A. Fisher, Statistical Methods for Research Workers (1925), section 21.1
See also: The difference (R-factor) · When a fit ends · The status bar
Compute Server
This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.
The window does no fitting itself: every computation runs in a compute server, fit_server, and Fit > Compute Server says where it is.
Fit > Compute Server opens a box titled Compute Server that asks for the "Server URL (fit_server must be running there):". It offers the address in use, http://127.0.0.1:8787 - a server on this computer - unless another has been set. Leaving it empty also means that default.
On Windows, the launcher the installed shortcut starts makes sure a server is answering at the default address before it opens the window.
Enter the address of a fit_server running elsewhere - a machine with more cores, say - to fit there. After OK the window checks at once whether anything answers. If nothing does, it says "No server answered at" the address, and "Start fit_server there - the application cannot fit without it." If a server answers and a profile is open, the profile is sent to it.
The address is remembered between sessions. A running fit_server describes its own interface: open the address followed by /docs in a web browser.
Limitations
- Only the profile is sent to the new server. The curves, positions, intervals and background points held by the old one are not; save the project before switching and open it again afterwards to take the whole model across.
See also: Compute backends · Minimizer
Compute backends
This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.
Help > Compute Backends summarises the engines a fit can run on and how to set up the optional Python one.
Fitting runs in the compute server the window is connected to (Fit > Compute Server). Behind it there are two engines, chosen under Fit > Minimizer:
- The built-in engine, Downhill Simplex (native), is part of fit_server. It needs nothing extra and fits every curve type.
- The Python engine, Levenberg-Marquardt (Python/lmfit), adds an uncertainty for each fitted parameter. It needs Python 3 with numpy, scipy and lmfit at pinned versions, installed in a separate environment so that your own Python is left alone.
fit_server looks for that environment in a folder named sidecar under %LOCALAPPDATA%\Fit\py on Windows and ~/.local/share/fit/py elsewhere ($XDG_DATA_HOME replaces ~/.local/share when it is set), or under the folder named by the environment variable FIT_PY_HOME when that is set. The libraries and their versions are listed in Worker/py/requirements.txt.
You never start the Python engine yourself: fit_server starts it the first time a fit asks for it and stops it when it exits. The window only ever talks to fit_server.
A third arrangement needs no engine choice at all: the same fit_server on another machine, reached through Fit > Compute Server.
Limitations
- The box Help > Compute Backends opens is a short summary; it names the setup steps rather than performing them.
See also: Setting up the Python engine · Minimizer · Compute Server · Weighting · Where Fit keeps its files
Setting up the Python engine
This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.
An installed copy needs no setup: the installer puts the Python engine beside the compute server, with its own Python. A copy built from source needs a one-time setup: a private Python environment with numpy, scipy and lmfit at the tested versions.
WHAT AN INSTALLER CARRIES: the engine frozen into one folder, sidecar, beside fit_server - Python, numpy, scipy, lmfit and the parts any module of that product adds - with the licence texts of each in sidecar/licenses. The compute server looks there first, so nothing on the machine is used or changed, and nothing below is needed.
THE FIRST START IS SLOW. The engine starts when you choose it, so it is usually ready before the first fit. The first time it starts after installing, the operating system checks each of its libraries once, and that can take a minute or two; every later start takes seconds. A fit that has to wait for it says so above the chart - "Starting the Python engine" - with the time elapsed. Stop ends the wait the way it ends a fit: the model is kept as it was, nothing is fitted, and the engine goes on starting, so the next fit finds it further along.
From source, you need Python 3.10 to 3.13 and the list of pinned libraries, Worker/py/requirements.txt, from the source repository (https://github.com/dvmorozov/fit). The libraries go into a separate virtual environment at exactly those versions, so your own Python is not touched and fitted numbers reproduce.
On Linux and macOS, from the folder holding the repository:
- python3 -m venv ~/.local/share/fit/py/sidecar
- ~/.local/share/fit/py/sidecar/bin/python -m pip install --only-binary=:all: -r Worker/py/requirements.txt
On Windows, in PowerShell:
- py -3.12 -m venv "$env:LOCALAPPDATA\Fit\py\sidecar"
- & "$env:LOCALAPPDATA\Fit\py\sidecar\Scripts\python.exe" -m pip install --only-binary=:all: -r Worker/py/requirements.txt
--only-binary=:all: stops pip from compiling numpy or scipy from source when your Python is newer than the pinned versions have ready-made packages for; it fails instead, and a Python in the range above is the fix.
The environment belongs to the machine, not to a copy of the program: every installation and every checkout on the machine shares it. Set the environment variable FIT_PY_HOME to keep it elsewhere; it then goes in the sidecar folder under that path.
To check it, start the Python engine by hand with sidecar/bin/python Worker/py/fit_backend.py --port 8788 (on Windows, sidecar\Scripts\python.exe) and open http://127.0.0.1:8788/health in a browser: it answers {"ok": true, ...}. Stop it with Ctrl+C - fit_server starts its own when the engine is chosen or a fit needs it.
Limitations
- On Debian and Ubuntu, creating the environment needs the separate python3-venv package; without it venv fails on ensurepip. Delete the half-made folder before trying again.
- It is deliberately not installed system-wide: Debian, Fedora and Homebrew refuse such an install (PEP 668), and the next unrelated pip install into a shared Python would change the versions the results depend on.
See also: Compute backends · Minimizer