Fitting

Fitting a model
Automatic decomposition
How many curves
Minimize Number of Curves
Minimize Difference
Stop
Why a fit command is greyed
The difference (R-factor)
Max Acceptable Difference
Curve scaling
Minimizer
Loss Function
Weighting
How the fit's figures are weighted
What the fit will actually do
When a fit ends
Samples no curve covers
The background, fitted in turns
The stages of a run
Is the difference only noise?
Compute Server
Compute backends
Setting up the Python engine

Fitting a model

This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.

A fit adjusts the parameters of the model's curves until the calculated profile matches the measured one as closely as it can.

A fit needs a profile to fit and a curve type to fit it with. Everything else - where the curves start, which stretches of the profile are scored, the background - can be prepared by hand or left to the program, which fills in whatever is missing before it starts.

There are three ways to start a fit, all in the Fit menu:

Fit > Automatically puts a background curve in the model where the build has one, places curves where the data need them, marks fit intervals and then removes every curve the data does not need. Use it for a first answer to "how many curves, and where?".
Fit > Minimize Number of Curves (F4) runs the same removal on the curves and intervals you already have.
Fit > Minimize Difference (F5) fits the model as it stands and removes nothing. The Fit button in the Fit group of the Tools tab runs the same command.

While a fit runs, the chart area shows its progress and every command that would change the model is greyed. Fit > Stop, or the Stop button of the Tools tab, ends it.

The fit itself runs in the compute server the window is connected to (Fit > Compute Server), not in the window. What it makes as small as possible is chosen under Fit > Loss Function, and the algorithm that does it under Fit > Minimizer.

Limitations

A fit refines the curves from where they start. It finds the best match near that start, which is not always the best match there is: curves started far from the peaks can settle on a poorer answer.

Automatic decomposition

This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.

Fit > Automatically decides how many curves the profile needs: it starts with far more than any profile needs and removes every one the fit can do without.

Fit > Automatically - also the last button of the toolbar above the chart - runs five steps in the compute server:

1. Where the build has background shapes, the background curve the model has - a Chebyshev one for a new model, unless you chose another shape - is fitted with the peaks, and nothing is subtracted from the profile. A model you left without a background (None, among the background's curve choices) is fitted without one: the run adds no curve behind your choice. Background points you had selected are discarded. The background's own chapter of this guide says what each of these is.
2. It places a curve on every point of every peak. A peak is found by starting at the highest point not yet taken and walking down both of its flanks; the search stops at the first peak whose top is lower than a fraction of the highest point - 1/30 by default, the background fraction. The search looks at the copy with the background taken off, so a high baseline is not mistaken for peaks. If you placed curve positions yourself, those are used instead.
3. It marks a fit interval around each peak, replacing any intervals you had marked. When that changes them, it says so when it ends - in a dialog and on the status bar - naming your intervals and its own, and the points yours fitted that are no longer fitted.
4. In each fit interval separately, it fits the curves and then removes every one the fit can do without, by the rule chosen under Fit > Remove Curves By. Which curve goes, and when it stops, is the subject of "How many curves".
5. It fits the curves that are left to full precision, in turns with the background curve, over the profile as measured. Steps 2 to 4 work on a copy of the profile with the background taken off, the way Subtract > Automatically would take it; the copy is discarded and the profile you see is never changed.

Every curve is of the curve type selected on the Tools tab. When the run ends, the chart, the tables and the Model panel show the curves that remain. The removed curves are gone together with the positions they were placed from, so a later edit or fit does not bring them back.

The setting that decides the answer is that rule. By BIC, the default, the data decide; by Max Acceptable Difference, the limit you set does - ask for less accuracy and you get fewer, broader curves.

It is offered as soon as a profile is open, including before the background has been dealt with - which is exactly what its first step does.

A curve type placed from its own markup is not decomposed as peaks. With nothing marked, its pack finds and places the instances in the data as measured, one fit interval each, and they are fitted; then, as for peaks, the instances that no longer pay for themselves on the fitted model are removed by the Bayesian information criterion, smallest first, and the rest fitted again - though the pack decides which of its instances may go. With a markup already there, the run only fits it, so what you corrected or deleted by hand stays as you left it. No background curve is added to a model whose type takes none. The pack's guide says how it finds them.

Limitations

The fit intervals and background points you had marked are discarded by steps 1 and 3; only the curve positions you picked are kept. Use Fit > Minimize Number of Curves to keep your intervals. A point your intervals fitted and the run's do not is left out of every figure, so two models compared over different points can differ by far more than their fit.
The copy steps 2 to 4 work on has the background taken off the way Subtract > Automatically would, which assumes a bowl-shaped background. On a background of another shape a curve may be placed on the baseline's bumps, or a small peak missed; the background curve is fitted in step 5 either way.

How many curves

Convention — Common practice or a guideline in the field, not a rule its sources require.

Fit > Remove Curves By chooses how the program decides how many curves a profile needs: by BIC, the default, or within a maximum difference you set.

With enough curves any profile can be fitted perfectly, noise included, so a least-squares fit alone cannot say how many curves there are. Fit starts with more curves than any profile needs and removes them one at a time; the rule decides which removals are kept.

Fit > Remove Curves By > BIC, the default for new work. A removal is kept when the Bayesian information criterion drops: chi-squared + k ln n, k the parameters varied and n the points of the fit interval. Each difference is divided by the noise measured from the data itself by the noise rule the curve type names - for counts a noise that grows with the measured value; each curve type's explanation links its rule. When that rule says the data are not counts, the fit's reduced chi-squared, AIC and BIC are weighted by the same noise, so the BIC shown is the one the curves were chosen by. A curve stays exactly when it improves the fit by more than its parameters cost, so no limit is needed and Fit > Set Max Acceptable Difference is greyed.
Fit > Remove Curves By > Max Acceptable Difference. A removal is kept while the R-factor of the fit interval stays within the limit set with Fit > Set Max Acceptable Difference: the precision you claim for the measurement, rather than a figure derived from the data.

The choice is saved in the project file. A project saved before the choice existed reopens with Max Acceptable Difference, the rule it was made with, so its results do not change.

Each fit interval is reduced on its own. Under either rule a curve whose amplitude has fallen below a thousandth of the largest in the interval is dropped at once, and candidates are taken in two passes: first the curve standing where the measured profile is steepest - on a flank it is often doing the work of a shoulder its neighbours can take over - then the curve with the smallest amplitude. A pass also ends when one curve is left, and a final fit polishes what survived.

By Max Acceptable Difference, a removal that breaks the limit is re-fitted, and if the fit still cannot meet it the curve goes back and the pass ends. Each re-fit stops as soon as the limit is met again, which keeps the method quick.

By BIC, a pass works in sweeps. Each candidate is judged as the model stands, its amplitude handed to the nearest curve - only when that curve is within reach of it, twice its width or three samples: given to one further away the amplitude is intensity added somewhere else, not what the removed curve held; every removal that lowers BIC is kept, and what is left is re-fitted once before the next sweep. When a sweep removes nothing, the five candidates that came closest are each tried with a full re-fit, and the pass ends when none of them lowers BIC. Re-fitting after every candidate, as the limit does, would take hours on a crowded start.

The Run Log gives, for every curve removed or kept against BIC, the figures it was judged by: the chi-square with the curve and without it, what its parameters are worth, the noise in that interval, and where its amplitude went or why it did not.

Limitations

The noise is only as good as the rule that measures it: for counts, a peak only a few samples wide adds its own curvature to the scatter, and BIC then keeps fewer curves on it than it might.
The fit itself minimises the plain difference, while BIC divides each difference by the noise where it is, so a re-fit can leave BIC slightly higher than before it.
Curves are removed one at a time, so either rule ends with a small model that meets it, not necessarily the smallest one that could. A curve shape that does not match the data can make BIC keep two curves where one peak stands.
By Max Acceptable Difference, if the first fit of an interval does not meet the limit, no curve is removed from it; the difference compared with the limit is always the R-factor of the interval, whichever loss function is selected.
The steepest-slope pass measures the slope from a curve's starting position to the next point of the profile, so it cannot remove a curve that starts on the profile's last point.

Sources

Schwarz, G., "Estimating the dimension of a model", The Annals of Statistics 6(2), 461-464 (1978) https://doi.org/10.1214/aos/1176344136
Foi, A. et al., "Practical Poissonian-Gaussian noise modeling and fitting for single-image raw-data", IEEE Transactions on Image Processing 17(10), 1737-1754 (2008) https://doi.org/10.1109/TIP.2008.2001399
Riener, M. et al., "GaussPy+: A fully automated Gaussian decomposition package for emission line spectra", Astronomy & Astrophysics 628, A78 (2019) https://doi.org/10.1051/0004-6361/201935519

Minimize Number of Curves

This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.

Removes every curve the fit can do without, starting from the curves and fit intervals you already have.

Fit > Minimize Number of Curves (F4) runs steps 4 and 5 of Fit > Automatically - the removal passes and the final fit - on the model as it stands. It does not touch the background, and it keeps the positions and fit intervals you placed.

What it needs is filled in only where it is missing: with fewer than two interval bounds it marks a fit interval around each peak, and with no curve positions it places a curve on every point of every peak, as the automatic run does.

Use it when you have placed more curves than you think are needed and want the data to decide which of them stay, by the rule chosen under Fit > Remove Curves By.

Curves it removes are gone together with their positions.

Limitations

It is greyed until the background has been subtracted or you have started building the model, and for a curve type placed from its pack's own markup; see "Why a fit command is greyed".
On a profile with many points, "a curve on every point of every peak" can be a great many curves, and the first fit then takes correspondingly long. Placing positions first avoids it.

Minimize Difference

This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.

Fits the model as it stands: every curve is kept and only their parameters change.

Fit > Minimize Difference (F5), and the Fit button in the Fit group of the Tools tab, adjust the parameters of every curve until the selected loss function can be made no smaller. No curve is added or removed, and the maximum acceptable difference plays no part.

Anything missing is filled in first, as for the other fit commands: a fit interval around each peak when fewer than two interval bounds are marked, and a curve on every point of every peak when there are no curve positions.

Each fit interval is fitted as a problem of its own. A background curve is one curve under all of them, fitted in turns with them - see "The background, fitted in turns". The difference reported afterwards pools all of them into one figure.

Use it after placing curves by hand, after changing a setting such as the loss function or the engine, or to fit again from where the last fit left off.

Limitations

It is greyed until the background has been subtracted or you have started building the model; see "Why a fit command is greyed".
Without curve positions it seeds a curve on every point of every peak, which on a finely sampled profile is a great many curves. Place positions first, or use Fit > Automatically.

Stop

This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.

Ends the fit or other long computation that is running.

Fit > Stop, and the Stop button in the Fit group of the Tools tab, are available only while a computation runs; the rest of the time they are greyed.

The engine is told to end, and does so at its next step. The window then shows what the fit had reached, exactly as it does when a fit finishes by itself: the curves, the tables and the statistics in the status bar are refreshed.

Stopping an automatic decomposition ends its passes where they are: curves already removed stay removed.

Closing the window while a computation runs stops it the same way. The status bar says so, and the window closes once the computation has ended - after asking, as it always does, whether to save the project, which now holds what the computation reached.

Limitations

A stopped fit has not converged. Its numbers are the best found so far, not the best the model can do; run Fit > Minimize Difference to carry on from there.

See also: When a fit ends · Fit progress

Why a fit command is greyed

This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.

The fit commands are offered only when there is something to fit, the background has been dealt with, and nothing else is running.

The window decides which commands it offers from the state of the compute server. The rules, in the order they apply:

Nothing open: every fit command is greyed until a profile is imported or a project is opened.
A computation running: only Fit > Stop is offered. Every command that would change the model waits until it is over, so that nothing is changed under a fit.
Background not yet dealt with: right after a profile is imported or reloaded, or a point of it is edited, Fit > Minimize Number of Curves, Fit > Minimize Difference and the Tools Fit button are greyed and Fit > Automatically is the fit on offer. Fitting curves to a profile that still carries its background is something to ask for deliberately, not to reach by the ordinary command. They become available once the background has been subtracted or you start building the model - placing curve positions or fit intervals.
Nothing running: Fit > Stop is greyed.
A curve type placed from its pack's own markup: Fit > Minimize Number of Curves is greyed, and refused if asked for. Its instances are whole patterns, not peaks to remove one by one; Fit > Automatically decides what such a model holds.

Within the settings, single entries can be greyed with a reason in their hint, shown in the status bar:

An engine under Fit > Minimizer that fits by evaluating a formula, when the selected curve type has none: "This curve type has no formula, so it is fitted by the native engine."
An objective under Fit > Loss Function that cannot be used with the selected curve type, with the reason.
Fit > Set Max Acceptable Difference, while Fit > Remove Curves By > BIC is chosen: nothing reads the limit then.

Fit > Enable Curve Scaling and Fit > Compute Server are never greyed: they are settings, and a changed setting applies to the next fit.

Limitations

The compute server itself would accept a manual fit before the background is subtracted and complete what is missing; it is the window that does not offer it. A program talking to the server directly is not held to this rule.

The difference (R-factor)

This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.

The number Fit reports for how well the model matches the data is an R-factor: the squared differences divided by the square of the summed data.

Throughout the program "difference" and "R-factor" name the same number. Over the points inside the fit intervals it is

R = sum( (s * calc - obs)^2 ) / ( sum obs )^2

where obs is a measured value, calc the model's value at the same point, and s the curve-scaling factor of the interval (1 when Fit > Enable Curve Scaling is off). With several fit intervals, the sums are pooled over all of them before dividing, so the figure means the same however the profile is divided up.

Dividing by the data makes it a relative measure: the same sample measured for ten times as long gives the same R-factor. A perfect fit gives 0. It is shown as a plain number, such as 7.7E-6 - not in percent - while Fit > Set Max Acceptable Difference asks for its ceiling in percent.

It is the default loss function, so it is usually also what the fit minimises. Whatever loss function is selected, the number reported and the number compared with the maximum acceptable difference is this R-factor.

While a fit runs, it is shown in the status bar and above the progress chart. When the fit ends, the status bar shows the reduced chi-squared and R-squared of the result in its place.

In diffraction the name R-factor usually means the profile R-factors of Rietveld refinement: Rp, the summed absolute differences over the summed data, and Rwp, the square root of the weighted squared differences over the weighted squared data. Fit's number shares their intent - a difference relative to the data - but is neither: it is unweighted, squared, and has no square root, so it is not directly comparable with an Rp or Rwp quoted elsewhere.

Limitations

Only the points inside the fit intervals count. Parts of the profile outside every interval can be fitted badly without changing the number.
Every point inside them counts, including one no curve of the model reaches - the model is 0 there, and that one point can outweigh all the others. Fit says when that happens; see "Samples no curve covers".

Sources

McCusker, L. B., Von Dreele, R. B., Cox, D. E., Louer, D. and Scardi, P., "Rietveld refinement guidelines", Journal of Applied Crystallography 32, 36-50 (1999) https://doi.org/10.1107/S0021889898009856

Max Acceptable Difference

This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.

The ceiling on the R-factor that decides how many curves are kept while Fit > Remove Curves By > Max Acceptable Difference is chosen.

Fit > Set Max Acceptable Difference opens the Max Acceptable Difference dialog, which has one field: "Max acceptable difference (percent):". Type the ceiling in percent, with a full stop as the decimal separator, and press OK. The default is 0.01, meaning an R-factor of 0.0001.

Anything that is not a number is refused with "Please enter a valid number (for example, 1.5)." and the dialog stays open. Cancel leaves the ceiling as it was.

The ceiling is used by Fit > Automatically and Fit > Minimize Number of Curves while curves are removed by it: a curve is removed only if the fit interval it belongs to still meets the ceiling without it. Under Fit > Remove Curves By > BIC nothing reads it, and the entry is greyed. Fit > Minimize Difference never uses it.

Choose it near the level the noise of your data allows. Too low, and noise is fitted with extra curves; too high, and real components are merged away.

It is saved in the project file, so a project reopens with the ceiling it was fitted with.

Limitations

The ceiling is compared with the R-factor, whichever loss function is selected, and with the R-factor of each fit interval on its own rather than of the whole profile.

Curve scaling

This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.

With curve scaling on, the model is multiplied by one factor per fit interval so that its area matches the data before the two are compared.

Fit > Enable Curve Scaling is a tick, and it is on by default. While it applies, the built-in engine compares s times the model with the data, where s is the sum of the measured values in the fit interval divided by the sum of the model's. The curves then only have to get the shape right, and the factor takes care of the overall height, which helps the fit converge.

It does not apply in two cases, whatever the tick says:

With the Python engine selected: it fits the amplitudes itself, so scaling afterwards would rescale values already fitted.
With a curve type that sets its own amplitude, free to grow. A second, overall multiplier would let the fit flatten the shape while the multiplier absorbs the difference. Hovering over the status bar explains it for this case.

The setting is saved in the project file.

Limitations

Combined with the R-factor (legacy) loss function, scaling lets a fit lower the number by inflating the model rather than by matching the data; that combination is refused for curve types whose amplitude is free.

Minimizer

This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.

Fit > Minimizer chooses the algorithm that searches for the best parameters; the first one, the default, needs nothing extra.

The entries of Fit > Minimizer are the engines this build offers, in order. The first is the default, and is also what a saved choice falls back to when it names an engine this build does not have. The choice is remembered between sessions.

Fit > Minimizer > Downhill Simplex (native) (the default): The original algorithm. Needs no Python and fits any curve type, including those with no formula.
Fit > Minimizer > Levenberg-Marquardt (Python/lmfit): Trust-region least squares with uncertainties. Needs the Python sidecar, and a curve type that has a formula. It runs on the Python sidecar; while that is not available, the default fits instead and says why. While it is selected, Fit > Weighting is shown. Curve scaling does not apply to it.
Fit > Minimizer > Downhill Simplex, curve by curve (native): The same simplex over one curve's parameters at a time, widest curve first, repeated until a pass gains nothing. Faster on models of many curves that each shape their own stretch; curves that trade off against each other may settle less well than under the simplex over everything.
Fit > Minimizer > Downhill Simplex, restarted (native): The same simplex over every parameter, started again from its own best, stepping each parameter forward or backward at random, until a run improves the loss by less than one percent. Never worse than the simplex, which is its first run; slower, and often a good deal better.

Downhill Simplex (native) is the Nelder-Mead simplex method. It needs only the value of the loss function, not its derivatives, so it fits any curve type, including those that compute their points without a formula. It gives no uncertainties for the parameters.

Levenberg-Marquardt (Python/lmfit) uses lmfit's least_squares method - despite the name, scipy's Trust Region Reflective solver. It reports an uncertainty for each fitted parameter, shown after the value in the Curve Attributes table as "value ± error".

Downhill Simplex, curve by curve (native) runs the same simplex over one curve's parameters at a time: each curve that shapes only its own stretch of the data is a block of its own, and everything that shapes the whole interval - curves that are never zero, such as a Gaussian, and any parameter the curves share - is one block together. The widest block goes first, and the blocks are fitted again until a pass improves the loss by less than one percent; then one simplex over every parameter finishes from there. A model whose curves all shape the whole interval is one block, and gets exactly what Downhill Simplex (native) gives it.

Downhill Simplex, restarted (native) runs the simplex over every parameter, then builds a fresh simplex around the best point it reached and runs again, until a run improves the loss by less than one percent. Its first run is exactly Downhill Simplex (native), so it never ends above it. A simplex stops when its corners agree, which can be on a ridge rather than at the bottom; starting again looks around once more. Each later run steps each parameter forward or backward at random, so every parameter is tried from both sides - the simplex's own restarts only ever turn the first two. The random choices come from a fixed seed, so fitting the same model again gives the same result.

An engine that fits by evaluating a formula is greyed while the selected curve type has none, and if it was selected, the default engine is selected instead. An engine that can only minimise a sum of squares leaves a loss function that is not one to the built-in engine; a dialog and the status bar say so.

Without Python: an installed copy carries the Python engine with it and needs nothing more. A copy built from source needs Python and three libraries set up on the machine that runs the compute server (Help > Compute Backends says how). If they are missing, the engine you chose is kept but Downhill Simplex (native) fits instead, and both the choice and the fit say so - an optional provider only ever adds alternatives. If Python is set up but fails to start, the fit does not start: the server answers that the Python backend is not available.

Limitations

No engine searches globally. Each refines the curves from where it starts, and can stop at a poorer answer when the start is far from a good one.
Curve by curve is this software's own choice, offered beside the simplex to be compared on real tasks, not proven better. On wave-pattern fits it reached a lower loss than the simplex in several times the time; curves that trade off against each other are reconciled pass by pass, which is slow when they are strongly coupled.
Restarted is this software's own choice too. On wave-pattern fits and on separate peaks it reached about half the simplex's loss in two to four times the time; how much a restart gains depends on the model, and nothing promises it gains anything.

Sources

Nelder, J. A. and Mead, R., "A simplex method for function minimization", The Computer Journal 7(4), 308-313 (1965) https://doi.org/10.1093/comjnl/7.4.308
Newville, M. et al., "LMFIT: Non-Linear Least-Squares Minimization and Curve-Fitting for Python" https://doi.org/10.5281/zenodo.11813
Branch, M. A., Coleman, T. F. and Li, Y., "A subspace, interior, and conjugate gradient method for large-scale bound-constrained minimization problems", SIAM Journal on Scientific Computing 21(1), 1-23 (1999)

Loss Function

This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.

Fit > Loss Function chooses the number a fit makes as small as possible; the default, R-factor, suits almost every fit.

Fit > Loss Function is the last entry of the Fit menu. Its entries are the objectives this build offers, the default first; the choice is remembered between sessions. Each entry's hint says what it measures:

Fit > Loss Function > R-factor (the default): Squared deviations normalised by the data, so values are comparable between datasets.
Fit > Loss Function > R-factor (legacy): The original objective. Normalised by the model rather than by the data, so with curve scaling on it can be lowered by inflating the model. Kept for comparison. It is not a sum of squares, so a fit with the Python engine runs on the built-in engine instead. It is greyed for a curve type whose amplitude is free to grow.
Fit > Loss Function > Sum of squares: Plain least squares, with no normalisation at all - the value scales with the data, so compare it only within one dataset.
Fit > Loss Function > Relative deviation: Absolute deviations as a fraction of the data - reads directly as "the model is off by this much". It is not a sum of squares, so a fit with the Python engine runs on the built-in engine instead.

Each objective also has a page of its own under Losses in Help > Explain Everything: its formula, why it exists, whether it is the field's convention or this program's own choice, what it does not cover and where to read more.

R-factor and Sum of squares differ only by a constant factor, so they find the same parameters; only the number differs. Relative deviation sums absolute rather than squared differences, so a few large misfits count for less. R-factor (legacy) divides by the model instead of the data: with curve scaling on, a fit can lower it by inflating the model without matching the data any better, which is why it is greyed for a curve type whose amplitude is free, and replaced by R-factor if it was selected.

Whatever is chosen, the difference the program reports, and compares with the maximum acceptable difference, is the R-factor. The statistics shown after a fit do not depend on the choice either; only the fitted parameters do.

Limitations

Every objective scores only the points inside the fit intervals.

Weighting

Convention — Common practice or a guideline in the field, not a rule its sources require.

Weighting decides how much each point counts in a fit by the Python engine; counting data are conventionally weighted by one over the square root of the count.

The Fit > Weighting submenu is shown only while an engine that can be weighted is selected under Fit > Minimizer - the Python one. The built-in engine always fits unweighted, so the menu is hidden under it. It has two entries:

Fit > Weighting > Poisson (counting statistics), the default: each difference is divided by the square root of the measured value, with values below 1 taken as 1. This is the usual choice for counts - of photons, neutrons, electrons - whose statistical error is the square root of the count, so a faint feature counts for as much as its precision allows.
Fit > Weighting > None (unweighted): every point counts equally, the same objective the built-in engine minimises. Use it for data that are not counts, or to compare the Python result term for term with the built-in one.

The choice is remembered between sessions. The reduced chi-squared shown in the status bar after any fit uses the Poisson weights, whichever engine ran - unless the model's curves describe data that are not counts, which is said beside the figure (see How the fit's figures are weighted).

Limitations

Poisson weighting assumes the values are counts. Values that have been rescaled, normalised or had a background subtracted no longer have the square root of the value as their error, and the weights are then only approximate.

Sources

McCusker, L. B., Von Dreele, R. B., Cox, D. E., Louer, D. and Scardi, P., "Rietveld refinement guidelines", Journal of Applied Crystallography 32, 36-50 (1999) https://doi.org/10.1107/S0021889898009856

How the fit's figures are weighted

Convention — Common practice or a guideline in the field, not a rule its sources require.

The reduced chi-squared, AIC and BIC after a fit weigh each difference as counting statistics, unless the model's curves describe data that are not counts - then by the noise measured from the data, and the status bar says "Reduced Chi2 (measured noise)".

A chi-squared is a sum of differences between the data and the model, each divided by how far that point could stray by chance alone. What it is divided by decides what the figure means, so the figures are weighted in one of two ways, chosen by the model and never by a menu:

As counts, the default: each difference over the square root of the measured value (values below 1 taken as 1), the counting statistics of photons, neutrons or electrons. AIC and BIC are then n ln(chi2/n) + 2k and n ln(chi2/n) + k ln n, as lmfit reports them, for n points and k parameters. Every curve type of a diffraction or spectroscopy profile is weighted so.
By the measured noise, when the noise rule a curve type in the model names says its data are not counts - a price, whose error is not the square root of its value. Each squared difference is divided by the variance of the noise at its point, which the rule measures from the data themselves - every field has its own, and each curve type's explanation links the one it names. The variance is then known rather than estimated from the fit, so AIC and BIC are chi2 + 2k and chi2 + k ln n - the same score Fit > Remove Curves By > BIC compares, so the BIC kept with the project is the one the model's curves were chosen by. The status bar then reads "Reduced Chi2 (measured noise):".

One such curve is enough: saying the data are not counts is a statement about the data the whole model is fitted to. R-squared is never weighted, either way. The noise is measured over the whole profile and read over the fit intervals, the samples every figure is taken over.

A reduced chi-squared near 1 says the model leaves a difference the size of the noise; well above 1, it leaves something it does not describe. Under the counting weights that reading holds only for data that really are counts.

Limitations

The measured noise is an estimate from the data, from the scatter of neighbouring points: it assumes the noise of neighbouring points is independent, and a signal that changes sharply from one point to the next is partly counted as noise.
Figures weighted in the two ways cannot be compared with each other; compare models of the same data, weighted the same way.

Sources

Schwarz, G., "Estimating the dimension of a model", Annals of Statistics 6 (2), 461-464 (1978) https://doi.org/10.1214/aos/1176344136
Foi, A., Trimeche, M., Katkovnik, V. and Egiazarian, K., "Practical Poissonian-Gaussian noise modeling and fitting for single-image raw-data", IEEE Transactions on Image Processing 17 (10), 1737-1754 (2008) https://doi.org/10.1109/TIP.2008.2001399
Newville, M. et al., "LMFIT: Non-Linear Least-Squares Minimization and Curve-Fitting for Python", fit statistics https://lmfit.github.io/lmfit-py/fitting.html

What the fit will actually do

This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.

When a choice cannot be honoured, the fit changes it and says so: always in the status bar, and in a dialog when you make the choice.

The right-hand end of the status bar always says what the next fit will do - for example "Minimising R-factor." or "Fitting with the built-in engine, minimising Relative deviation." When something was changed, hovering over the status bar shows why.

Three things can be changed. The compute server decides with the same rule the status bar reports, so the two cannot disagree:

The loss function: R-factor (legacy) is replaced by R-factor for a curve type whose amplitude is free to grow.
The engine: with the Python engine selected, the fit runs on the built-in engine when the curve type has no formula, or when the loss function is not a sum of squares. Your objective is kept; what you lose is the per-parameter uncertainties.
Curve scaling: switched off for a curve type that sets its own amplitude.

When you change the engine, the loss function or the curve type and the result would be changed, a dialog titled "About this fit" explains it. It is not repeated while you stay with that selection, and not shown at start-up for a setting chosen earlier. Curve scaling being switched off never opens the dialog; hovering over the status bar explains it.

See also: Loss Function · Minimizer · Curve scaling · The status bar

When a fit ends

This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.

The chart, the tables and the Model panel are refreshed from the result, and the status bar shows how long the fit took and how good it is.

While a fit runs, the R-factor tab of the chart area draws its progress - and, with View > Animation Mode ticked, the Model tab the model moving - and the tables keep the last finished result. The R-factor curve is kept with the model the run reached, saved in the project, and drawn again whenever that model is opened. A long run reports thousands of improvements; at most 4000 of them are kept - the first, the last, and for each equal stretch of the run's time its lowest and its highest, the stretches widening as the run goes on, as a chart of a squeezed range keeps them - so every part of the run is kept alike, a spike or a dip one sample wide stays on the curve, and the project stays small. A project saved before this limit is read back the same way.

When it ends, whether it finished or was stopped, the window:

draws the profile again with the model over it: each curve, their sum (Total Amplitude) and the Difference between data and model;
draws the curve positions you picked and the fitted positions where the fit put the curves, and the fit interval bounds;
refills the Curve Positions and Fit Intervals tables, the Curve Attributes table with every curve's parameters, and the Summary table with the data, the model and each curve at every point of every interval;
rebuilds the list of curves in the Model panel;
shows in the status bar the elapsed time and, when they can be computed, the reduced chi-squared and R-squared of the fit - and, when some points inside the fit intervals are covered by no curve, says how many, with a dialog naming them (see "Samples no curve covers").

The elapsed time counts only the time the computer was awake. A fit left running while the machine slept or hibernated does not count the sleep, and neither does the time axis of the progress chart.

The result is part of the project from then on, and is kept on disk when the project is saved. It is also recorded in the History tab, beside every model an earlier fit reached, so it can be made current again after later changes.

A model put back by value - a project reopened, or a model made current from the History tab - is measured as it comes back, so its R-factor is the one it was recorded with. A model that was never fitted reads "Not calculated", as it did while it was being edited.

Limitations

The R-factor itself is on screen only while the fit runs; when it ends, the status bar shows the reduced chi-squared and R-squared in its place.

Samples no curve covers

This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.

A point inside a fit interval that no curve of the model reaches is modelled as 0, still counts in the R-factor, and is named where the result is shown.

Some curves exist only over a stretch of the data and are exactly zero outside it - a wave pattern, for instance, runs from its first point to its last. A Gaussian or a Lorentzian is never exactly zero, so it always reaches every point of its fit interval.

Where no curve reaches a point inside a fit interval, the model is 0 there, and the point still counts in the R-factor and in every other figure. On data that sits far from zero - a price series near 1000, say - that one point can outweigh every other difference together: one point left at the start of a series once made a model score seventy times worse over the whole profile than over two intervals that began one point later.

Fit does not drop such points from the figures. A number that changed with how the model happens to be placed would no longer compare one model with another. It says instead where they are. When a fit ends, a dialog gives how many there are, where, and how much of the squared difference between the data and the model they make; the status bar says how many, and the History entry of the model keeps the whole of it.

Before a fit starts - Fit > Minimize Difference or Fit > Minimize Number of Curves - the same is asked first, while it can still be corrected: Yes fits the model as it is, No leaves it untouched so the intervals or the curves can be changed. Fit > Automatically is not asked: it replaces the intervals with its own, and says when it ends what that left out.

The question and the dialog appear once for the same points between them. A later fit that leaves the same points uncovered says so only in the status bar and in History; points elsewhere, or the same ones coming back after they were covered, bring the dialog back.

Two ways forward. To measure only what the model describes, narrow the fit interval to the stretch the curves cover (Model > Fit Intervals > Start Manual Selection, or the Fit Intervals table). To describe the points as well, extend the model over them - move or add a curve so that one reaches them.

Limitations

A background the fit varies, or a background curve, is part of the model at every point, so with one no point counts as uncovered, whatever the other curves reach.
The points are named from the model as it stands. A project saved by an earlier version has no record of them, so its History entries say nothing about them until a fit runs again.

The background, fitted in turns

This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.

A background curve is one curve under the whole profile, so a fit takes turns: the background with the peaks held, then the peaks of every fit interval with the background held.

Each fit interval is fitted on its own, side by side with the others. A background fitted inside each interval would be a different curve in each - lines pointing every way under one pattern. So the background is fitted apart: to the whole of the data with the peaks every interval found held, the stretches between the intervals included, where the data is nothing but background. Then every interval fits its peaks with that background held under them.

Each such pair is a turn. The turns go on while one improves the R-factor by at least a thousandth of it, twenty at most, and Stop ends them where they stand. The Run Log says how many turns there were and why they ended.

Fit > Automatically fits the background differently: to the FLOOR of the data first, before the reduction and with no peak curve subtracted. The samples standing more than a few noise units above the curve are left out and it is fitted through the rest, round after round - the rule that finds the fit intervals. Fitted to the data minus the peaks, a hump the starting curves already sat on left the background nothing to improve, and those curves were never reduced. Once they are gone the turns above take over, because through the floor's samples alone the curve is free under a peak, where the data has none. A fit you start yourself fits it with the peaks only, as above.

A series background - the Chebyshev - has a number of terms, the parameter "terms" of its curve, and a curve starts with six. Fit > Automatically starts it at twelve (fewer for data with too few samples to fix them, and never fewer than it already has), then scores 16, 24, 32, 48 and 64 terms and keeps the best, and then tries two more and two fewer for as long as the score falls: a hump is clipped as a peak until the curve has the terms to follow it, so the score falls at a cliff and not by degrees. The score is a BIC in which each sample's part is capped at a few noise units, so a peak counts as one miss however tall it is and a hump the background cannot follow costs every one of its samples. The Run Log says "Background went from n to m terms" with both figures. A fit you start yourself leaves the terms as they are.

Curve scaling multiplies each interval's peaks and never the background, which belongs to no interval. It scales them to the data above the background, so the background's step weighs each background it tries against the peaks as they would be scaled over it, not as they stood.

Limitations

Twelve to begin with, a step of two, and sixty-four as the most are this software's choices, not measured optima. A background with many terms begins to model the peaks, which is the cost BIC weighs; it cannot tell a hump from peaks nobody placed.
Fitted in turns rather than together, the model stops a little short of the best one when a turn no longer improves it. The thousandth and the twenty turns are this software's choice.
Intensity no curve describes between the intervals is taken for background there, and lifts it.

See also: Minimize Difference · Curve scaling

The stages of a run

This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.

The Run Log names every stage a fit or Fit > Automatically goes through, in order, and says of a stage it left out that it was skipped and why.

The stages are: finding the peaks, finding the fit intervals, fitting the background to the floor of the data, removing curves that pay for nothing, fitting the peaks, and fitting the background in turns with the intervals. A plain fit finds peaks or intervals only where it has none, removes no curve and fits the background with the peaks, in turns; Fit > Automatically fits the background to the floor of the data first, before the reduction, and then takes the turns like any fit.

Finding the fit intervals says how they were found: they are what is left above the background curve fitted to the data, so it gives that fit - how many terms the curve had, how many samples lie on it, the noise, and the rounds of fitting and clipping it took. Fitting the background to the floor says how many terms the curve started and ended with, the score it was judged by, and every change of terms is a line of its own with both figures.

A stage is skipped when there is nothing for it to do - the model has no background curve, the picks were already there - or when Stop ended the run before it. The line says which.

Is the difference only noise?

Canonical — Stated by the field's authoritative sources, which the explanation quotes and cites.

A model that explains the data leaves a difference that is noise; after every fit the Ljung-Box test asks whether it is, and the wavelet transform names where it is not.

The R-factor says how large the difference between the data and the model is, not what it is made of. A model can score well and still leave a wave or a peak it does not describe; what is left then follows itself - each point of the difference says something about the next. Noise does not.

After every fit the difference over each fit interval is tested whole. Each point is first divided by the noise expected where it is - the same estimate the fit's figures and the automatic reduction weigh the difference by - so a quiet stretch and a loud one count alike. That noise is measured by the noise rule the model's curve type names, which is its field's: a count's grows with the count, a price's with the square of the price. Each curve type's explanation links its rule. The test is Ljung and Box's: the difference's first autocorrelations, up to ten of them or one for every five points, squared and summed. The intervals' tests are combined by Fisher's method into one p.

WHERE it is not noise is found point by point, by the wavelet transform (Torrence and Compo): a Fourier transform over a window that grows with the rhythm looked at, from four points up. At each point it asks how much rhythm of each length the difference holds there, against what noise would hold, and the chance noise holds that much is known exactly. Every point of every fit interval is judged together, held to a false discovery rate of 5 % (Benjamini and Hochberg): tested one by one at 5 %, a thousand points of pure noise would name fifty.

The status bar says "Difference: random (p = ...)" or how many stretches are not, the chart shades those stretches, and the model's History entry keeps the verdict.

Under the model's chart, the Difference structure strip draws that judgement point by point: how much rhythm the difference holds at each point of the fit intervals, against what noise would hold there - 1 for noise. Where the curve reaches the dashed line the point is named, and those are the shaded stretches above it; a spike is a place the model has something left to describe. The difference itself is drawn in the strip too, against its own axis on the right, so a spike of the curve sits over the stretch of the difference that made it. The strip moves and zooms with the chart above it.

A module included in the build may draw the same transform as a chart of its own, and its chapter says how to read it.

A shaded stretch is where the model has something left to describe: a curve to add, one to move, or a fit interval that cuts a feature in two.

Q = n(n + 2) sum_(k=1..h) r_k^2 / (n - k), which for white noise follows a chi-square of h degrees of freedom.

Limitations

A test is not a proof. It sees what makes the difference follow itself; a difference that changes only in size from one stretch to another can pass, and a wrong estimate of the noise moves every verdict.
The standardising, and starting the wavelet at a rhythm of four points, are this software's choices: a shorter one is not a wave a pattern can draw, and the wavelet that short passed noise more often than its test allows. An interval of fewer than ten points is not tested; one of fewer than twenty is not searched for where.
A false discovery rate of 5 % allows some of what is named to be noise: with a large stretch named, a few short ones elsewhere may be chance. And a shaded stretch reaches beyond the feature that made it - up to two or three of its rhythms, as far as the windows of the longer rhythms it registers in - while near the ends of a fit interval only short rhythms can be judged at all.
A cell of the map is tested on its own, at 5 %: in a large map of noise, one cell in twenty passes by chance, scattered. A patch spanning several places and rhythms is what to read, and the shaded stretches - judged together - are the places named.

Sources

G. M. Ljung and G. E. P. Box, On a measure of lack of fit in time series models, Biometrika 65 (1978), pp. 297-303
Y. Benjamini and Y. Hochberg, Controlling the false discovery rate, J. R. Statist. Soc. B 57 (1995), pp. 289-300
C. Torrence and G. P. Compo, A practical guide to wavelet analysis, Bull. Amer. Meteor. Soc. 79 (1998), pp. 61-78
J. P. Imhof, Computing the distribution of quadratic forms in normal variables, Biometrika 48 (1961), pp. 419-426
R. A. Fisher, Statistical Methods for Research Workers (1925), section 21.1

Compute Server

This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.

The window does no fitting itself: every computation runs in a compute server, fit_server, and Fit > Compute Server says where it is.

Fit > Compute Server opens a box titled Compute Server that asks for the "Server URL (fit_server must be running there):". It offers the address in use, http://127.0.0.1:8787 - a server on this computer - unless another has been set. Leaving it empty also means that default.

On Windows, the launcher the installed shortcut starts makes sure a server is answering at the default address before it opens the window.

Enter the address of a fit_server running elsewhere - a machine with more cores, say - to fit there. After OK the window checks at once whether anything answers. If nothing does, it says "No server answered at" the address, and "Start fit_server there - the application cannot fit without it." If a server answers and a profile is open, the profile is sent to it - unless it belongs to another product, which is said instead.

The address is remembered between sessions. A running fit_server describes its own interface: open the address followed by /docs in a web browser.

Limitations

Only the profile is sent to the new server. The curves, positions, intervals and background points held by the old one are not; save the project before switching and open it again afterwards to take the whole model across.

Compute backends

This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.

Help > Compute Backends summarises the engines a fit can run on and how to set up the optional Python one.

Fitting runs in the compute server the window is connected to (Fit > Compute Server). Behind it are the engines chosen under Fit > Minimizer, of two kinds:

The built-in engines - Downhill Simplex (native), the default, and its curve-by-curve and restarted variants - are part of fit_server. They need nothing extra and fit every curve type.
The Python engine, Levenberg-Marquardt (Python/lmfit), adds an uncertainty for each fitted parameter. It needs Python 3 with numpy, scipy and lmfit at pinned versions, installed in a separate environment so that your own Python is left alone.

fit_server looks for that environment in a folder named sidecar under %LOCALAPPDATA%\Fit\py on Windows and ~/.local/share/fit/py elsewhere ($XDG_DATA_HOME replaces ~/.local/share when it is set), or under the folder named by the environment variable FIT_PY_HOME when that is set. The libraries and their versions are listed in Worker/py/requirements.txt.

You never start the Python engine yourself: fit_server starts it the first time a fit asks for it, on a port it finds free, and stops it when it exits. The window only ever talks to fit_server.

A third arrangement needs no engine choice at all: the same fit_server on another machine, reached through Fit > Compute Server.

Limitations

The box Help > Compute Backends opens is a short summary; it names the setup steps rather than performing them.

Setting up the Python engine

This software's choice — A decision made by this software rather than by the field's sources; the explanation says why it was made.

An installed copy needs no setup: the installer puts the Python engine beside the compute server, with its own Python. A copy built from source needs a one-time setup: a private Python environment with numpy, scipy and lmfit at the tested versions.

WHAT AN INSTALLER CARRIES: the engine frozen into one folder, sidecar, beside fit_server - Python, numpy, scipy, lmfit and the parts any module of that product adds - with the licence texts of each in sidecar/licenses. The compute server looks there first, so nothing on the machine is used or changed, and nothing below is needed.

THE FIRST START IS SLOW. The engine starts when you choose it, so it is usually ready before the first fit. The first time it starts after installing, the operating system checks each of its libraries once, and that can take a minute or two; every later start takes seconds. A fit that has to wait for it says so above the chart - "Starting the Python engine" - with the time elapsed. Stop ends the wait the way it ends a fit: the model is kept as it was, nothing is fitted, and the engine goes on starting, so the next fit finds it further along.

From source, you need Python 3.10 to 3.13 and the list of pinned libraries, Worker/py/requirements.txt, from the source repository (https://github.com/dvmorozov/fit). The libraries go into a separate virtual environment at exactly those versions, so your own Python is not touched and fitted numbers reproduce.

On Linux and macOS, from the folder holding the repository:

python3 -m venv ~/.local/share/fit/py/sidecar
~/.local/share/fit/py/sidecar/bin/python -m pip install --only-binary=:all: -r Worker/py/requirements.txt

On Windows, in PowerShell:

py -3.12 -m venv "$env:LOCALAPPDATA\Fit\py\sidecar"
& "$env:LOCALAPPDATA\Fit\py\sidecar\Scripts\python.exe" -m pip install --only-binary=:all: -r Worker/py/requirements.txt

--only-binary=:all: stops pip from compiling numpy or scipy from source when your Python is newer than the pinned versions have ready-made packages for; it fails instead, and a Python in the range above is the fix.

The environment belongs to the machine, not to a copy of the program: every installation and every checkout on the machine shares it. Set the environment variable FIT_PY_HOME to keep it elsewhere; it then goes in the sidecar folder under that path.

To check it, start the Python engine by hand with sidecar/bin/python Worker/py/fit_backend.py --port 8788 (on Windows, sidecar\Scripts\python.exe) and open http://127.0.0.1:8788/health in a browser: it answers {"ok": true, ...}. Stop it with Ctrl+C - fit_server starts its own, on another port, when the engine is chosen or a fit needs it.

Limitations

On Debian and Ubuntu, creating the environment needs the separate python3-venv package; without it venv fails on ensurepip. Delete the half-made folder before trying again.
It is deliberately not installed system-wide: Debian, Fedora and Homebrew refuse such an install (PEP 668), and the next unrelated pip install into a shared Python would change the versions the results depend on.