Feedback about NLP From Scratch: Generating Names with a Character-Level RNN
Author: eurushCreated Aug 18, 2026Updated Aug 19, 2026
There is the following issue on this page: https://docs.pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html.
There is no non-linearity wrapper while passing the hidden state onto the next iteration, and on output layer that predicts distribution. Results are coming relatively fine though, maybe because nn.LogSoftmax is implicitly acting as non-linearity. But for learning hidden states, its bad, $h_t$ is literally linear combination of $h_{t-1}$, all past $x$ s and $Cat$.
Simple Fix:
class RNN(nn.Module):
def __init__(self,C_in:int, C_hid:int, C_out:int):
super().__init__()
self.C_in = C_in
self.C_hid = C_hid
self.C_out = C_out
self.C_cat = data1.C_cat
self.i2h = nn.Linear(data1.C_cat + C_in + C_hid,C_hid)
self.i2o1 = nn.Linear(data1.C_cat + C_in + C_hid,C_out)
self.o12o2 = nn.Linear(C_out + C_hid,C_hid)
self.o22o3 = nn.Linear(C_hid, C_out)
self.tanh = nn.Tanh()
self.todist = nn.LogSoftmax(dim=1)
def forward(self,x_C:tensor, x_X:tensor, x_H:tensor):
# Inputs = (B=1,C_cat), (B=1, C_in), (B=1,C_out),
# Output = (B=1, C_out), (B=1,C_hid)
x = torch.cat((x_C,x_X,x_H),dim=1)
h = self.tanh(self.i2h(x))
x = self.i2o1(x)
x = torch.cat((x,h),dim=1)
x = self.tanh(self.o12o2(x))
x = self.o22o3(x)
return self.todist(x), h
Edit1: nn.tanh 's range is between $[-1,1]$. Compressing logits before softmax can really impact the result. Using one more layer before projection gave me similar error of 2.2127 , and nice samples.
Thank you.
Source: pytorch/tutorials