Bias correction

Bias correction#

In xDEM, bias-correction methods correspond to transformations that cannot be described as a 3-dimensional affine function (see Coregistration), and aim at correcting both systematic elevation errors and spatially-structured random errors.

Contrary to affine coregistration methods, bias corrections are not limited to the information in the elevation data. They can be passed any external variables (e.g., land cover type, processing metric) to attempt to identify and correct biases. Still, many methods rely either on coordinates (e.g., deramping, along-track corrections) or terrain (e.g., curvature- or elevation-dependant corrections), derived solely from the elevation data.

Quick use#

Bias-correction methods are used the same way as coregistrations:

import xdem
import numpy as np

# Create a bias-correction
biascorr = xdem.coreg.DirectionalBias(angle=45, fit_or_bin="bin", bin_sizes=200, bin_apply_method="per_bin", bin_statistic=np.mean)

Bias correction can estimate and correct the bias by a parametric fit using fit_or_bin="fit" linked to fit_ parameters, by applying a binned statistic using fit_or_bin="bin" linked to bin_ parameters, or by a parametric fit on the binned data using fit_or_bin="bin_and_fit" linked to all parameters.

Predefined bias corrections usually take additional arguments such as angle for DirectionalBias, poly_order for Deramp and attribute for TerrainBias.

Once defined, they can be applied the same two ways as for coregistration (using fit() and apply() separately allows to re-apply the same correction to different elevation data).

# Coregister with bias correction by calling the DEM method
corrected_dem = tba_dem.coregister_3d(ref_dem, biascorr)
# (Equivalent) Or by calling the fit and apply steps
corrected_dem = biascorr.fit_and_apply(ref_dem, tba_dem)

Additionally, bias corrections can be customized to use any number of variables to correct simultaneously, by defining bias_var_names in BiasCorr and passing a bias_vars dictionary arrays or rasters to fit() and apply(). See Using custom variables for more details.

The modular `BiasCorr` object#

Inherited from `Coreg`#

Each bias-correction method in xDEM inherits their interface from the Coreg class (see The Coreg object). This implies that bias-correction methods can be combined in a CoregPipeline with any other methods, or applied in a block-wise manner through BlockwiseCoreg.

Inheritance diagram of co-registration and bias corrections:

The main difference with Coreg is that a BiasCorr has a new bias_var_names argument which allows declaring the names of N bias-correction variables that will be passed, which corresponds to the number of simultaneous dimensions in which the bias correction is performed. This step is implicit for predefined methods such as DirectionalBias.

Modular estimation#

Bias-correction methods have three ways of estimating and correcting a bias in N-dimensions:

Performing a binning of the data along variables with a statistic (e.g., median), then applying the statistics in each bin,
Fitting a parametric function to the variables, then applying that function,
(Recommended¹) Fitting a parametric function on a data binning of the variable, then applying that function.

The parameters related to fitting or binning are the same for every BiasCorr() method:

fit_or_bin to either fit a parametric model to the bias by passing “fit”, perform an empirical binning of the bias by passing “bin”, or to fit a parametric model to the binning with “bin_and_fit” (recommended),
fit_func to pass any parametric function to fit to the bias,
fit_optimizer to pass any optimizer function to perform the fit minimization,
bin_sizes to pass the size or edges of the bins for each variable,
bin_statistic to pass the statistic to compute in each bin,
bin_apply_method to pass the method to apply the binning for correction.

For predefined methods, the default values of these parameters differ. For instance, a Deramp generally performs well with a “fit” estimation on a subsample, and thus has a fixed fit_func (2D polynomial) solved by the classic optimizer scipy.optimize.curve_fit(). In contrast, a TerrainBias is generally hard to model parametrically, and thus defaults to a “bin” estimation.

Finally, each bias-correction approach has the following methods:

fit() for estimating the bias, which expects a new bias_vars dictionary except for predefined methods such as DirectionalBias,
apply() for correcting the bias on a DEM, which also expects a bias_vars dictionary except for predefined methods.

Good practices#

Several good practices help performing a successful bias correction:

Avoid using “fit” with a subsample size larger than 1,000,000: Otherwise the optimizer will be extremely slow and might fail with a memory error; consider using “bin_and_fit” instead to reduce the data size before the optimization which still allows to utilize all the data,
Avoid using “fit” or “bin_and_fit” for more than 2 dimensions (input variables): Fitting a parametric form in more than 2 dimensions is quite delicate, consider using “bin” or sequential 1D corrections instead,
Use at least 1000 bins for all dimensions, being mindful about dimension number: Using a small bin size is generally too rough, but a large bin size will grow exponentially with the number of bias variables,
Use customized bin edges for data with large extreme values: Passing simply a bin size will set the min/max of the data as the full binning range, which can be impractical (e.g., most curvatures lie between -2/2 but can take values of 10000+).

Bias-correction methods#

Important

Below we create biased elevation data to examplify the different methods in relation to their type of correction.

See bias correction on real data in the Basic and Advanced gallery examples!

Deramping#

xdem.coreg.Deramp

Performs: Correction with a 2D polynomial of degree N.
Supports weights: Yes.
Pros: Can help correct a large category of biases (lens deformations, camera positioning), and runs fast.
Cons: Overfits with limited static surfaces.

Deramping works by estimating and correcting for an N-degree polynomial over the entire elevation difference.

# Instantiate a 2nd order 2D deramping
deramp = xdem.coreg.Deramp(poly_order=2)
# Fit and apply
corrected_dem = deramp.fit_and_apply(ref_dem, tbc_dem_ramp)

_images/a9f0c5b6777564db677ef76b801e4dbe390bccb96ab1ed86d5debf31e742fead.png

Directional biases#

xdem.coreg.DirectionalBias

Performs: Correct biases along a direction.
Supports weights: Yes.
Pros: Correcting undulations or jitter, common in both stereo and radar DEMs, or strips common in scanned imagery.
Cons: Long optimization when fitting a sum of sinusoids.

For strip-like errors, performing an empirical correction using only a binning with fit_or_bin="bin" allows more flexibility than a parametric form, but requires a large amount of static surfaces.

# Define a directional bias correction at a certain angle (degrees), for a binning of 1000 bins
dirbias = xdem.coreg.DirectionalBias(angle=60, fit_or_bin="bin", bin_sizes=1000)
# Fit and apply
corrected_dem = dirbias.fit_and_apply(ref_dem, tbc_dem_strip)

_images/da26e78f3708cc89b38397f36da915fe7f32c2609185e26e743a0502a27fcbe5.png

Terrain biases#

xdem.coreg.TerrainBias

Performs: Correct biases along a terrain attribute.
Supports weights: Yes.
Pros: Useful to correct for instance curvature-related bias due to different native resolution between elevation data.
Cons: For curvature-related biases, only works for elevation data with relatively close native resolution.

The default optimizer for terrain biases optimizes a 1D polynomial with an order from 1 to 6, and keeps the best performing fit.

# Instantiate a 1st order terrain bias correction for curvature
terbias = xdem.coreg.TerrainBias(terrain_attribute="max_curvature",
                                 bin_sizes={"max_curvature": np.linspace(-5, 5, 1000)},
                                 bin_apply_method="per_bin")

# We have to pass the original curvature here
corrected_dem = terbias.fit_and_apply(ref_dem, tbc_dem_curv, bias_vars={"max_curvature": maxc})

_images/8f388ce660d72c7a6c431ed4c6eb1f9218fea19c656b8c5d8d42dfd1cdad4592.png

Using custom variables#

All bias-corrections methods are inherited from generic classes that perform corrections in 1-, 2- or N-D. Having these separate helps the user navigating the dimensionality of the functions, optimizer, binning or variables used.

xdem.coreg.BiasCorr

Performs: Correct biases with any function and optimizer, or any binning, in 1-, 2- or N-D.
Supports weights: Yes.
Pros: Versatile.
Cons: Needs more setting up!

# Create a bias correction defining three custom variable names that will be passed later
# We force a binning method, more simple in 3D
biascorr = xdem.coreg.BiasCorr(bias_var_names=["aspect", "slope", "elevation"], fit_or_bin="bin", bin_sizes=5)

# Derive curvature and slope
aspect, slope = ref_dem.get_terrain_attribute(["aspect", "slope"])

# Pass the variables to the fit_and_apply function matching the names declared above
corrected_dem = biascorr.fit_and_apply(
    ref_dem,
    tba_dem_nk,
    inlier_mask=inlier_mask,
    bias_vars={"aspect": aspect, "slope": slope, "elevation": ref_dem}
)

Warning

Using any custom variables, and especially in many dimensions, can lead to over-correction and introduce new errors. For instance, elevation-dependent corrections (as shown below) typically introduce new errors (due to more high curvatures at high elevation such as peaks, and low curvatures at low elevation with flat terrain).

For this reason, it is important to check the sanity of elevation differences after correction!

_images/123abccf4f0f5c084d8f2c60ed5161509bbfd3263aeec35fa47b657af0a56965.png