Pytorch-1-Introduction

Statement: This series of post records the personal notes and experiences of learning the BiliBili video tutorial "Pytorch 入门学习", most of code and pictures are from the courseware PyTorch-Course. All posted content is for personal study only, do not use for other purposes. If there is infringement, please contact e-mail:yangsuoly@qq.com to delete.

1. Introduction to deep learning models

1.1 Definition

Q: What is machine learning?
A: Study of algorithms that: - Improve their performance P - At some task T - With experience E

Conclusion: Modeling, Inference, learning

• Modeling: define score function
• Inference: solve argmax
• Learning: choose w

\text{classify} (x, w) = \mathop{\text{argmax}}\limits_{y} \ \text{score} (x, y, w)

Q: What is deep learning? A:

Q: What is neural network? A:

1.2 Activation function

1.2.1 Commonly used activation functions:

• sigmoid: \sigma(x) = \frac{1}{1+e^{-x}}

• tanh: tanh(x) = 2 \sigma(2x) - 1

• ReLU: ReLU(x) = max(0, x)

• Softmax: z_i \rightarrow \frac{e^{z_i}}{\sum_{j=1}^{k} e^{z_j}}

1.2.2 Code implementation

import torch
import torch.nn.functional as F
from torch.autograd import Variable
import matplotlib.pyplot as plt

x = torch.linspace(-5, 5, 200)
x = Variable(x)
x_np = x.data.numpy()

# four activation function
y_relu = F.relu(x).data.numpy()
y_sigmoid = torch.sigmoid(x).data.numpy()
y_tanh = F.tanh(x).data.numpy()
y_softplus = F.softplus(x).data.numpy()

plt.figure(1, figsize=(8, 6))
plt.subplot(221)
plt.plot(x_np, y_relu, c='red', label='relu')
plt.ylim((-1, 5))
plt.legend(loc='best')

plt.subplot(222)
plt.plot(x_np, y_sigmoid, c='red', label='sigmoid')
plt.ylim((-0.2, 1.2))
plt.legend(loc='best')

plt.subplot(223)
plt.plot(x_np, y_tanh, c='red', label='tanh')
plt.ylim((-1.2, 1.2))
plt.legend(loc='best')

plt.subplot(224)
plt.plot(x_np, y_softplus, c='red', label='softplus')
plt.ylim((-0.2, 6))
plt.legend(loc='best')

plt.show()

Result:

1.3 examples of NN

• Standard feedforward NN

• Convolutional NN

• Recurrent NN

• Seq2Seq with Attention Reference: Effective Approaches to Attention-based Neural Machine Translation

2. Introduction to PyTorch

2.1 Framework for deep learning

Difference between PyTorch and Tensorflow: - PyTorch: 动态计算图 Dynamic Computation Graph - Tensorflow: 静态计算图 Static Computation Graph

PyTorch 代码通俗易懂，非常接近 Python 原生代码，不会让人感觉是完全在学习一门新的语言。拥有 Facebook 支持，社区活跃。

Q: What does the PyTorch do? A:

2.2. Some interesting project with PyTorch

• ResNet Image classification: ResNet

• Object Detection Project address: Here

• Image Style Transfer Project address: Here

• CycleGAN Project address: Here

• Image Captioning Project address: Here

• Sentiment Analysis Project address: Here

• Question Answering Project address: Here

• Translation: OpenNMT-py Project address: Here

• ChatBot Project address: Here

• Deep Reinforcement Learning

  Project address: Project 1, Project 2

2.3 How to learn PyTorch

• Basics of deep learning;
• Pytorch official tutorial;
• Learn tutorials on GitHub and various blogs;
• Documentation and BBS
• Re-creat the open source PyTorch project;
• Read papers about deep learning model and implement them;
• Create your own model.

3. Note content

1. Pytorch framework with autograd introduction, simple forward neural networks;
2. Word vector;
3. Image classification, CNN, Transfer learning;
4. Language Model, Sentiment Classification, RNN, LSTM, GRU;
5. Translation Model, Seq2Seq, Attention;
6. Reading Comprehension, EIMo, BERT, GPT-2;
7. ChatBot;
8. GAN, Face generation, Style Transfer.