The open source machine learning library PyTorch 1.7 has been released, the current version of PyTorch supports Nvidia’s programming platform CUDA 11. The release includes a number of new APIs, supports NumPy-compatible Fast Fourier Transform (FFT) operations – this feature is still in beta – and offers new profiling tools. The library for improving the performance in the Autograd Profiler contains additions for TorchScript and Stack Traces – so that users should not only see operator names in the output table of the Profiler, but also where the operator is actually located in the code.

In terms of new front-end APIs, PyTorch 1.7 includes torch.fft , a module to implement of FFT functions, supports C ++ with the nn.transformer – module abstraction and can use torch.set_deterministic to control operators that select deterministic algorithms, if available . TorchElastic for providing strict supersets in the torch.distributed.launch CLI now applies as stable and receives further functionalities to increase the error tolerance and elasticity.

Parity between Python and C ++ APIs is getting closer The PyTorch team has been working on the frontend APIs for C ++ since PyTorch 1.5: Since then, the developers have been striving for parity between the Python and C ++ APIs. The current release apparently allows developers to use the nn.transformer module abstraction from the C ++ -Frontend to use. The intermediate step of loading via Python / JIT is no longer necessary; according to the blog announcement, the module abstraction can be implemented directly in C ++ with PyTorch 1.7. The development is still in the beta stage.

Further innovations in the area of ​​mobile devices concern Torchvision, which supports tensor inputs, and a new one Video API (still in beta). Within this feature, transforms inputs now inherit from nn.modules . The inputs should be able to be implemented in TorchScript and, according to the blog post, support tensors with batch dimensions, they should also be seamlessly executable on CPU / GPU devices, as the PyTorch team illustrates with a code example:

import torch

import torchvision.transforms as T # to fix random seed, use torch .manual_seed

# instead of random.seed

torch. manual_seed (12)

transforms = torch.nn.Sequential (

T.RandomCrop (224),

T.RandomHorizontalFlip (p = 0.3),

T.ConvertImageDtype (torch.float),

T.Normalize ( , [0.229, 0.224, 0.225])

)

scripted_transforms = torch .jit.script (transforms)

# Note: we can similarly use T.Compose to define transforms

# transforms = T.Compose ([…]) and

# scripted_transforms = torch.jit.script (torch.nn.Sequential transforms.transforms))

tensor_image = torch.randint (0, 256, size = (3, 256, 256), dtype = torch.uint8)

# works directly on tensors

out_image1 = transforms (tensor_image)

# on the GPU

out_image1_cuda = transforms (tensor_image.cuda ())

# with batches

batched_image = torch.randint (0, 229, size = ( 4, 3, 256, 256), dtype = torch.uint8)

out_image_batched = transforms (batched_image)

# and has torchscript support

out_image2 = scripted_transforms (tensor_image)

Torchaudio should support voice recordings (wav2letter) and voice output of texts (WaveRNN).

Distributed training on new feet The PyTorch team has apparently fundamentally revised the distributed training based on DDP (Distributed Data Parallel) and RPC (Remote Procedure Call) , extended training should also be possible on Windows in the future (the development is still in the prototype stage). The fact that PyTorch is working harder to support Windows was already a core topic in the last release. Some previously experimental features such as user-defined C ++ classes, extensions using tensor-like objects and the memory profiler are now considered stable. Since the last release (1.6), the PyTorch team has classified the development status of features as stable, beta or prototype.

PyTorch 1.7 supports DistributedDataParallel (DDP) and the collective communication on the Windows platform. The support in the current release concerns in particular the ProcessGroup and the FileStore, which are both based on the API gateway Gloo. In order to be able to use the feature called prototype, developers must provide a file from the common file system in the init_process_group. The PyTorch team demonstrates how to do this in their blog:

# initialize the process group

dist. init_process_group (

“gloo” ,

# multi-machine example:

# init_method = “file: ////// {machine} / {share_folder} / file”

init_method = “file: /// {your local file path}”,

rank = rank,

world_size = world_size

) model = DistributedDataParallel (local_model, device_ids = [rank])

To Interested parties can consult a design document on GitHub and the documentation on the PyTorch website.

Frontend APIs with FFT function nality The PyTorch team has recently been working on functionalities that historically have only had to a minor extent received support from the ML library: The current version of PyTorch receives a new torch.fft Module that uses the same API to implement FFT functions as the NumPy program library for numerical calculations in Python. Processes with FFT (Fast Fourier Transforms) are common in scientific fields when processing signals. According to the announcement in the PyTorch blog, importing the module is essential in order to be able to use PyTorch 1.7, otherwise there will be naming conflicts with functions that are now deprecated.

How this can look in practice, the PyTorch team provides a code example in the blog and refers to the more detailed documentation:

import torch.fft

t = torch.arange (4)

t

tensor ([0, 1, 2, 3]) torch.fft.fft (t)

tensor ([ 6.+0.j, -2.+2.j, -2.+0.j, -2.-2.j])

t = tensor ( [0.+1.j, 2.+3.j, 4.+5.j, 6.+7.j])

torch.fft.fft (t)

tensor ([12.+16.j, -8.+0.j, -4.-4.j, 0.-8.j])

Additional information Notes on all of the numerous innovations and more detailed information can be found in the announcement in the PyTorch blog. Interested parties will find further links to the documentation and the download options there. All features are marked as stable, beta or prototype according to the recently adopted PyTorch conventions.

(sih)