Questions tagged [pytorch]
PyTorch is an open-source deep learning framework and API that creates a Dynamic Computational Graph, which allows you to flexibly change the way your neural network behaves on the fly and is capable of performing automatic backward differentiation.
pytorch
23,797
questions
567
votes
20
answers
1.2m
views
How do I check if PyTorch is using the GPU?
How do I check if PyTorch is using the GPU? The nvidia-smi command can detect GPU activity, but I want to check it directly from inside a Python script.
- 28.1k
386
votes
11
answers
410k
views
How do I save a trained model in PyTorch?
How do I save a trained model in PyTorch? I have read that:
torch.save()/torch.load() is for saving/loading a serializable object.
model.state_dict()/model.load_state_dict() is for saving/loading ...
- 36.7k
347
votes
8
answers
273k
views
Why do we need to call zero_grad() in PyTorch?
Why does zero_grad() need to be called during training?
| zero_grad(self)
| Sets gradients of all model parameters to zero.
- 12.7k
341
votes
9
answers
261k
views
What does .view() do in PyTorch?
What does .view() do to a tensor x? What do negative values mean?
x = x.view(-1, 16 * 5 * 5)
- 36.7k
333
votes
11
answers
420k
views
How do I print the model summary in PyTorch?
How do I print the summary of a model in PyTorch like what model.summary() does in Keras:
Model Summary:
...
- 36.7k
272
votes
4
answers
287k
views
What does model.eval() do in pytorch?
When should I use .eval()? I understand it is supposed to allow me to "evaluate my model". How do I turn it back off for training?
Example training code using .eval().
- 25.7k
268
votes
10
answers
462k
views
How do I initialize weights in PyTorch?
How do I initialize weights and biases of a network (via e.g. He or Xavier initialization)?
- 24.9k
242
votes
5
answers
143k
views
What's the difference between reshape and view in pytorch?
In numpy, we use ndarray.reshape() for reshaping an array.
I noticed that in pytorch, people use torch.view(...) for the same purpose, but at the same time, there is also a torch.reshape(...) ...
- 12.3k
220
votes
8
answers
145k
views
220
votes
6
answers
227k
views
What does model.train() do in PyTorch?
Does it call forward() in nn.Module? I thought when we call the model, forward method is being used.
Why do we need to specify train()?
- 21.5k
186
votes
7
answers
75k
views
pytorch - connection between loss.backward() and optimizer.step()
Where is an explicit connection between the optimizer and the loss?
How does the optimizer know where to get the gradients of the loss without a call liks this optimizer.step(loss)?
-More context-
...
- 21.5k
185
votes
11
answers
150k
views
Check the total number of parameters in a PyTorch model
How do I count the total number of parameters in a PyTorch model? Something similar to model.count_params() in Keras.
- 24.9k
181
votes
5
answers
78k
views
Why do we "pack" the sequences in PyTorch?
I was trying to replicate How to use packing for variable-length sequence inputs for rnn but I guess I first need to understand why we need to "pack" the sequence.
I understand why we "...
- 21.5k
179
votes
8
answers
316k
views
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same
This:
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
for data in dataloader:
inputs, labels = data
outputs = model(inputs)
Gives ...
- 3,397
178
votes
4
answers
176k
views
PyTorch preferred way to copy a tensor
There seems to be several ways to create a copy of a tensor in PyTorch, including
y = tensor.new_tensor(x) #a
y = x.clone().detach() #b
y = torch.empty_like(x).copy_(x) #c
y = torch.tensor(x) #d
b ...
- 6,950
169
votes
6
answers
528k
views
Pytorch tensor to numpy array
I have a pytorch Tensor of shape [4, 3, 966, 1296]. I want to convert it to numpy array using the following code:
imgs = imgs.numpy()[:, ::-1, :, :]
How does that code work?
- 2,279
167
votes
23
answers
638k
views
How to avoid "CUDA out of memory" in PyTorch
I think it's a pretty common message for PyTorch users with low GPU memory:
RuntimeError: CUDA out of memory. Tried to allocate X MiB (GPU X; X GiB total capacity; X GiB already allocated; X MiB free; ...
- 2,329
141
votes
4
answers
29k
views
Pytorch, what are the gradient arguments
I am reading through the documentation of PyTorch and found an example where they write
gradients = torch.FloatTensor([0.1, 1.0, 0.0001])
y.backward(gradients)
print(x.grad)
where x was an initial ...
- 4,303
128
votes
5
answers
82k
views
What does the gather function do in pytorch in layman terms?
What does torch.gather do?
This answer is hard to understand.
- 2,734
123
votes
8
answers
217k
views
How do I split a custom dataset into training and test datasets?
import pandas as pd
import numpy as np
import cv2
from torch.utils.data.dataset import Dataset
class CustomDatasetFromCSV(Dataset):
def __init__(self, csv_path, transform=None):
self.data ...
- 4,090
123
votes
9
answers
260k
views
How to fix RuntimeError "Expected object of scalar type Float but got scalar type Double for argument"?
I'm trying to train a classifier via PyTorch. However, I am experiencing problems with training when I feed the model with training data.
I get this error on y_pred = model(X_trainTensor):
...
- 1,729
121
votes
5
answers
126k
views
What does "unsqueeze" do in Pytorch?
The PyTorch documentation says:
Returns a new tensor with a dimension of size one inserted at the specified position. [...]
>>> x = torch.tensor([1, 2, 3, 4])
>>> torch.unsqueeze(x, ...
- 1,687
120
votes
4
answers
87k
views
What's the difference between torch.stack() and torch.cat() functions?
OpenAI's REINFORCE and actor-critic example for reinforcement learning has the following code:
REINFORCE:
policy_loss = torch.cat(policy_loss).sum()
actor-critic:
loss = torch.stack(policy_losses)....
- 25.7k
120
votes
3
answers
162k
views
119
votes
8
answers
244k
views
L1/L2 regularization in PyTorch
How do I add L1/L2 regularization in PyTorch without manually computing it?
- 36.7k
113
votes
5
answers
52k
views
What's the difference between "hidden" and "output" in PyTorch LSTM?
I'm having trouble understanding the documentation for PyTorch's LSTM module (and also RNN and GRU, which are similar). Regarding the outputs, it says:
Outputs: output, (h_n, c_n)
output (...
- 8,141
112
votes
7
answers
178k
views
How do I visualize a net in Pytorch?
import torch
import torch.nn as nn
import torch.optim as optim
import torch.utils.data as data
import torchvision.models as models
import torchvision.datasets as dset
import torchvision.transforms as ...
- 2,991
109
votes
14
answers
322k
views
Why `torch.cuda.is_available()` returns False even after installing pytorch with cuda?
On a Windows 10 PC with an NVidia GeForce 820M
I installed CUDA 9.2 and cudnn 7.1 successfully,
and then installed PyTorch using the instructions at pytorch.org:
pip install torch==1.4.0+cu92 ...
- 1,780
109
votes
3
answers
204k
views
Convert PyTorch tensor to python list
How do I convert a PyTorch Tensor into a python list?
I want to convert a tensor of size [1, 2048, 1, 1] into a list of 2048 elements. My tensor has floating point values. Is there a solution which ...
- 43.8k
108
votes
4
answers
204k
views
How do I get the value of a tensor in PyTorch?
Printing a tensor x gives:
>>> x = torch.tensor([3])
>>> print(x)
tensor([3])
Indexing x.data gives:
>>> x.data[0]
tensor(3)
How do I get just a regular non-tensor value 3?...
- 3,091
107
votes
4
answers
166k
views
How to do gradient clipping in pytorch?
What is the correct way to perform gradient clipping in pytorch?
I have an exploding gradients problem.
- 25.7k
105
votes
17
answers
111k
views
ModuleNotFoundError: No module named 'tools.nnwrap'
I tried to install torch using:
pip install torch
Installation started, but after a few seconds I got the error:
from tools.nnwrap import generate_wrappers as generate_nn_wrappers
...
- 2,092
102
votes
17
answers
363k
views
How can I fix this strange error: "RuntimeError: CUDA error: out of memory"?
I successfully trained the network but got this error during validation:
RuntimeError: CUDA error: out of memory
- 1,129
101
votes
5
answers
201k
views
Calculate the output size in convolution layer [closed]
How do I calculate the output size in a convolution layer?
For example, I have a 2D convolution layer that takes a 3x128x128 input and has 40 filters of size 5x5.
- 1,240
99
votes
7
answers
194k
views
How do I convert a Pandas dataframe to a PyTorch tensor?
How do I train a simple neural network with PyTorch on a pandas dataframe df?
The column df["Target"] is the target (e.g. labels) of the network. This doesn't work:
import pandas as pd
...
- 1,121
97
votes
3
answers
120k
views
How to load a list of numpy arrays to pytorch dataset loader?
I have a huge list of numpy arrays, where each array represents an image and I want to load it using torch.utils.data.Dataloader object. But the documentation of torch.utils.data.Dataloader mentions ...
- 1,577
96
votes
6
answers
95k
views
Data Augmentation in PyTorch
I am a little bit confused about the data augmentation performed in PyTorch. Now, as far as I know, when we are performing data augmentation, we are KEEPING our original dataset, and then adding other ...
- 1,323
92
votes
13
answers
217k
views
CUDA runtime error (59) : device-side assert triggered
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/generated/../generic/THCTensorMathPointwise.cu line=265 error=59 : device-side assert triggered
Traceback (most ...
- 1,235
92
votes
7
answers
154k
views
How to tell PyTorch to not use the GPU?
I want to do some timing comparisons between CPU & GPU as well as some profiling and would like to know if there's a way to tell pytorch to not use the GPU and instead use the CPU only? I realize ...
- 8,711
92
votes
1
answer
113k
views
How does the "number of workers" parameter in PyTorch dataloader actually work?
If num_workers is 2, Does that mean that it will put 2 batches in the RAM and send 1 of them to the GPU or Does it put 3 batches in the RAM then sends 1 of them to the GPU?
What does actually happen ...
- 1,471
91
votes
4
answers
164k
views
How do I multiply matrices in PyTorch?
With numpy, I can do a simple matrix multiplication like this:
a = numpy.ones((3, 2))
b = numpy.ones((2, 1))
result = a.dot(b)
However, this does not work with PyTorch:
a = torch.ones((3, 2))
b = ...
- 3,260
91
votes
2
answers
105k
views
How to change the learning rate of an optimizer at any given moment (no LR schedule)?
Is it possible in PyTorch to change the learning rate of the optimizer in the middle of training dynamically (I don't want to define a learning rate schedule beforehand)?
So let's say I have an ...
- 18.2k
90
votes
10
answers
266k
views
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! when resuming training
I saved a checkpoint while training on gpu. After reloading the checkpoint and continue training I get the following error:
Traceback (most recent call last):
File "main.py", line 140, in &...
- 901
86
votes
4
answers
317k
views
PyTorch: How to get the shape of a Tensor as a list of int
In numpy, V.shape gives a tuple of ints of dimensions of V.
In tensorflow V.get_shape().as_list() gives a list of integers of the dimensions of V.
In pytorch, V.size() gives a size object, but how ...
- 18.2k
85
votes
6
answers
146k
views
How to use multiple GPUs in pytorch?
I use this command to use a GPU.
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
But, I want to use two GPUs in jupyter, like this:
device = torch.device(&...
- 991
82
votes
3
answers
84k
views
Difference between "detach()" and "with torch.nograd()" in PyTorch?
I know about two ways to exclude elements of a computation from the gradient calculation backward
Method 1: using with torch.no_grad()
with torch.no_grad():
y = reward + gamma * torch.max(net....
- 1,051
81
votes
8
answers
178k
views
How do I display a single image in PyTorch?
How do I display a PyTorch Tensor of shape (3, 224, 224) representing a 224x224 RGB image?
Using plt.imshow(image) gives the error:
TypeError: Invalid dimensions for image data
- 43.8k
81
votes
2
answers
77k
views
What does the parameter retain_graph mean in the Variable's backward() method?
I'm going through the neural transfer pytorch tutorial and am confused about the use of retain_variable(deprecated, now referred to as retain_graph). The code example show:
class ContentLoss(nn....
- 2,855
78
votes
24
answers
625k
views
No module named "Torch"
I successfully installed pytorch via conda:
conda install pytorch-cpu torchvision-cpu -c pytorch
I also successfully installed pytorch via pip:
pip3 install https://download.pytorch.org/whl/cpu/torch-...
- 883
77
votes
4
answers
123k
views
Embedding in pytorch
Does Embedding make similar words closer to each other? And do I just need to give to it all the sentences? Or it is just a lookup table and I need to code the model?
- 1,093