TabTransformer

This is an unofficial TabTransformer Pytorch implementation created by Ignacio Oguiza (oguiza@timeseriesAI.co)

Huang, X., Khetan, A., Cvitkovic, M., & Karnin, Z. (2020). TabTransformer: Tabular Data Modeling Using Contextual Embeddings. arXiv preprint https://arxiv.org/pdf/2012.06678

Official repo: https://github.com/awslabs/autogluon/tree/master/tabular/src/autogluon/tabular/models/tab_transformer

---

source

TabTransformer

def TabTransformer(
    classes, cont_names, c_out, column_embed:bool=True, add_shared_embed:bool=False, shared_embed_div:int=8,
    embed_dropout:float=0.1, drop_whole_embed:bool=False, d_model:int=32, n_layers:int=6, n_heads:int=8,
    d_k:NoneType=None, d_v:NoneType=None, d_ff:NoneType=None, res_attention:bool=True, attention_act:str='gelu',
    res_dropout:float=0.1, norm_cont:bool=True, mlp_mults:tuple=(4, 2), mlp_dropout:float=0.0, mlp_act:NoneType=None,
    mlp_skip:bool=False, mlp_bn:bool=False, bn_final:bool=False
):

Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:to, etc.

.. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool

---

source

FullEmbeddingDropout

def FullEmbeddingDropout(
    dropout:float
):

From https://github.com/jrzaurin/pytorch-widedeep/blob/be96b57f115e4a10fde9bb82c35380a3ac523f52/pytorch_widedeep/models/tab_transformer.py#L153

---

source

SharedEmbedding

def SharedEmbedding(
    num_embeddings, embedding_dim, shared_embed:bool=True, add_shared_embed:bool=False, shared_embed_div:int=8
):

Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:to, etc.

.. note:: As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool

---

source

ifnone

def ifnone(
    a, b
):

b if a is None else a

from fastai.tabular.all import *

path = untar_data(URLs.ADULT_SAMPLE)
df = pd.read_csv(path/'adult.csv')
dls = TabularDataLoaders.from_csv(path/'adult.csv', path=path, y_names="salary",
    cat_names = ['workclass', 'education', 'marital-status', 'occupation', 'relationship', 'race'],
    cont_names = ['age', 'fnlwgt', 'education-num'],
    procs = [Categorify, FillMissing, Normalize])
x_cat, x_cont, yb = first(dls.train)
model = TabTransformer(dls.classes, dls.cont_names, dls.c)
test_eq(model(x_cat, x_cont).shape, (dls.train.bs, dls.c))