Understanding and simple implementation of time2vec

Time2Vec Understanding and simple implementation of

Preface

Time2Vec Its function can be seen from its name , Make time Embedding, And can be applied to different models .

2019 A paper in ：Time2Vec: Learning a Vector Representation of Time

Time2Vec

Time2Vec The design of is mainly based on the following aspects ：

1. Capture periodic and aperiodic patterns
2. Scale the time unchanged
3. Easy to integrate with other models

Time2Vec The formula of is not complicated ：

$$t2v(\tau )[i]=\{\begin{array}{ll}{\omega }_i\tau +{\phi }_i,& \text{if }i=0.\\ \mathcal{F}({\omega }_i\tau +{\phi }_i),& \text{if }1\le i\le k.\end{array}$$

among $k$ by time2vec Dimensions ,$\mathcal{F}$ Activate the function for the period ,${\omega }_i,{\phi }_i$ Is a learnable parameter . In order for the algorithm to capture periodicity , therefore $\mathcal{F}$ choose $\sin$ function （$\cos$ Function has the same effect ） Capture periodicity .

PyTorch Realization

def t2v(tau, f, out_features, w, b, w0, b0, arg=None):
    if arg:
        v1 = f(torch.matmul(tau, w) + b, arg)
    else:
        v1 = f(torch.matmul(tau, w) + b)
    v2 = torch.matmul(tau, w0) + b0
    return torch.cat([v1, v2], 1)


class SineActivation(nn.Module):
    def __init__(self, in_features, out_features):
        super(SineActivation, self).__init__()
        self.out_features = out_features
        self.w0 = nn.parameter.Parameter(torch.randn(in_features, 1))
        self.b0 = nn.parameter.Parameter(torch.randn(in_features, 1))
        self.w = nn.parameter.Parameter(torch.randn(in_features, out_features - 1))
        self.b = nn.parameter.Parameter(torch.randn(in_features, out_features - 1))
        self.f = torch.sin

    def forward(self, tau):
        return t2v(tau, self.f, self.out_features, self.w, self.b, self.w0, self.b0)


class CosineActivation(nn.Module):
    def __init__(self, in_features, out_features):
        super(CosineActivation, self).__init__()
        self.out_features = out_features
        self.w0 = nn.parameter.Parameter(torch.randn(in_features, 1))
        self.b0 = nn.parameter.Parameter(torch.randn(in_features, 1))
        self.w = nn.parameter.Parameter(torch.randn(in_features, out_features - 1))
        self.b = nn.parameter.Parameter(torch.randn(in_features, out_features - 1))
        self.f = torch.cos

    def forward(self, tau):
        return t2v(tau, self.f, self.out_features, self.w, self.b, self.w0, self.b0)


class Time2Vec(nn.Module):
    def __init__(self, activation, hiddem_dim):
        super(Time2Vec, self).__init__()
        if activation == "sin":
            self.l1 = SineActivation(1, hiddem_dim)
        elif activation == "cos":
            self.l1 = CosineActivation(1, hiddem_dim)

        self.fc1 = nn.Linear(hiddem_dim, 2)

    def forward(self, x):
        x = self.l1(x)
        x = self.fc1(x)
        return x

summary

For the of time Embedding How to put it? , Personally, I feel it is necessary and unnecessary , not essential , Of course, it's not that time information is not important . I didn't read the paper carefully , Of course, the main content is also relatively small , Feel for time 、 Position the handling of these things , It's still sin、cos The effect will be better ？ When looking at the code, I saw the author's Date2Vec, The model is not understood , It doesn't explain the principle specifically , You can have a look if you are interested .

Reference material

• [1] Time2Vec: Learning a Vector Representation of Time