torch.use_deterministic_algorithms

torch.use_deterministic_algorithms(d)

Sets whether PyTorch operations must use “deterministic” algorithms. That is, algorithms which, given the same input, and when run on the same software and hardware, always produce the same output. When True, operations will use deterministic algorithms when available, and if only nondeterministic algorithms are available they will throw a :class:RuntimeError when called.

Warning

This feature is in beta, and its design and implementation may change in the future.

The following normally-nondeterministic operations will act deterministically when d=True:

• torch.nn.Conv1d when called on CUDA tensor

• torch.nn.Conv2d when called on CUDA tensor

• torch.nn.Conv3d when called on CUDA tensor

• torch.nn.ConvTranspose1d when called on CUDA tensor

• torch.nn.ConvTranspose2d when called on CUDA tensor

• torch.nn.ConvTranspose3d when called on CUDA tensor

• torch.bmm() when called on sparse-dense CUDA tensors

• torch.__getitem__() backward when self is a CPU tensor and indices is a list of tensors

• torch.index_put() with accumulate=True when called on a CPU tensor

The following normally-nondeterministic operations will throw a RuntimeError when d=True:

• torch.nn.AvgPool3d when called on a CUDA tensor that requires grad

• torch.nn.AdaptiveAvgPool2d when called on a CUDA tensor that requires grad

• torch.nn.AdaptiveAvgPool3d when called on a CUDA tensor that requires grad

• torch.nn.MaxPool3d when called on a CUDA tensor that requires grad

• torch.nn.AdaptiveMaxPool2d when called on a CUDA tensor that requires grad

• torch.nn.FractionalMaxPool2d when called on a CUDA tensor that requires grad

• torch.nn.FractionalMaxPool3d when called on a CUDA tensor that requires grad

• torch.nn.functional.interpolate() when called on a CUDA tensor that requires grad and one of the following modes is used:

  • linear

  • bilinear

  • bicubic

  • trilinear

• torch.nn.ReflectionPad1d when called on a CUDA tensor that requires grad

• torch.nn.ReflectionPad2d when called on a CUDA tensor that requires grad

• torch.nn.ReplicationPad1d when called on a CUDA tensor that requires grad

• torch.nn.ReplicationPad2d when called on a CUDA tensor that requires grad

• torch.nn.ReplicationPad3d when called on a CUDA tensor that requires grad

• torch.nn.NLLLoss when called on a CUDA tensor that requires grad

• torch.nn.CTCLoss when called on a CUDA tensor that requires grad

• torch.nn.EmbeddingBag when called on a CUDA tensor that requires grad

• torch.scatter_add_() when called on a CUDA tensor

• torch.index_add_() when called on a CUDA tensor

• torch.index_copy()

• torch.index_select() when called on a CUDA tensor that requires grad

• torch.repeat_interleave() when called on a CUDA tensor that requires grad

• torch.histc() when called on a CUDA tensor

• torch.bincount() when called on a CUDA tensor

• torch.kthvalue() with called on a CUDA tensor

• torch.median() with indices output when called on a CUDA tensor

A handful of CUDA operations are nondeterministic if the CUDA version is 10.2 or greater, unless the environment variable CUBLAS_WORKSPACE_CONFIG=:4096:8 or CUBLAS_WORKSPACE_CONFIG=:16:8 is set. See the CUDA documentation for more details: https://docs.nvidia.com/cuda/cublas/index.html#cublasApi_reproducibility If one of these environment variable configurations is not set, a RuntimeError will be raised from these operations when called with CUDA tensors:

• torch.mm()

• torch.mv()

• torch.bmm()

Note that deterministic operations tend to have worse performance than non-deterministic operations.

Parameters

d (bool) – If True, force operations to be deterministic. If False, allow non-deterministic operations.