Initializers

Initializing neural networks properly enables faster convergence and better accuracy. A number of advance initialization schemes have been developed over the past few years. The initializers package simplifies the process of applying initialization to your models in a way that is tightly integrated with pytorch's nn.init package.

To illustrate the design of the package, we shall create a simple cnn

import torch.nn as nn
class ConvBlock(nn.Module):
def __init__(self,in_channels,out_channels):
super(ConvBlock,self).__init__()
self.net = nn.Sequential(
nn.Conv2d(in_channels,out_channels,kernel_size=3,padding=1),
nn.BatchNorm2d(out_channels),
nn.ReLU()
)
def forward(self,x):
return self.net(x)
class CifarNet(nn.Module):
def __init__(self,num_classes=10):
super(CifarNet,self).__init__()
self.net = nn.Sequential(
ConvBlock(3,64),
ConvBlock(64,64),
ConvBlock(64,64),
nn.MaxPool2d(3,2),
ConvBlock(64,128),
ConvBlock(128,128),
ConvBlock(128,128),
nn.MaxPool2d(3,2),
ConvBlock(128,128),
ConvBlock(128,128),
ConvBlock(128,128),
nn.Dropout(0.25),
nn.AdaptiveAvgPool2d((1,1)),
)
self.classifier = nn.Linear(128,num_classes)
def forward(self,x):
x = self.net(x)
x = x.view(x.size(0),-1)
return self.classifier(x)

The network above is composed of different layers including convolutions, batch normalization and a Linear layer. We shall initialize them differently.

from torchfusion_utils.initializers import *
model = CifarNet()
#initialize the convolution layers with kaiming_normal
kaiming_normal_init(model,types=[nn.Conv2d])
#initialize the linear layers with normal
normal_init(model,types=[nn.Linear])
#initialize batchnorm weights with ones
ones_init(model,types=[nn.BatchNorm2d],category="weight")
#initialize batchnorm bias with zeros
zeros_init(model,types=[nn.BatchNorm2d],category="bias")

As seen above, the types and category args enables fine grained control over how different parameters in different layers are initialized.

This functionality is not limited to the initializers provided in the package, you can easily bring in custom initializers of your own or easily take advantage of new ones added to torch.nn.init even before they are added to torchfusion_utils.

Example of custom init

from torchfusion_utils.initializers import init
from torch.nn.init import xavier_normal_
#initialize weights with the xavier_normal_
init(model,xavier_normal_,init_params={"gain":1},types=[nn.Linear],category="weight")