TL;DR

To come up with an model architecture with better metrics on BM3 while using fingerprints as input
Explore more ways to baseline non-weighted models for sub-structure data

Introduction

1. Background

Name of the experiment: Scaffold-based subfragment resource propagation

Experiment start date: 01/04/2021

Please look at initial observations and results for this project here: https://coda.io/d/Use-cases_dgX-TWPBu3W/210121-Promiscuity-model-using-compound-sub-structures_su3as#_luGKU

1.1 Experiment motivation

The goal of this exercise is to reduce promiscuity in prediction. i.e Machine learning models tend to predict “active” for a lot of compounds. This happens a) When models don’t see enough inactive samples while training (relevant to kinase data) b) When models fail to generalize across different chemical space
An ideal model will have to generalize over a new chemical space too . i.e not throw “active” for most compounds . Metrics on BM3 should look good with an auc_roc > 0.8 as our single target model have 0.8 auc_roc. Here we want to achieve something better than that.
Goal overview: We want to achieve two things here 1) come up with a model that produces less % of actives i.e we are fp should be low 2) Model should be able to perform well with new chemical space

1.2 Experiment approcach

To achieve our goals. we split this experiment into 2 blocks a) develop a model that learns from all targets simultaneously and produce less promiscuity. b) Use sub-structures as embedding and do the same. Why is it important to do step 1? Currently even with state of the art models such as chemprop and deepchem we are not able to get less promiscuous results / good metrics on our data.
Our hypothesis is sub-strucutre present in compounds decide how a "parent compound" interacts with targets.
We generate sub-structure score for all substrucutres present in a dataset and enrich it with a fingerprint of choice. In the beginning of this experiment we started with ecfp. Currently, we are working on MAACS and ECFP fingerpint.
Before we could start seeing if sub-structures actually add any value to the learning it is important to have a baseline model that uses just the fingerprints

1.2.1 Model Approach

We wanted to develop multitask models that can simultaneously learn from correlated tasks. So far with the state of the art MTL model (chemprop) on our BM3 data we had an average auc of only 0.5. As we have no baseline models that uses fingerprint as input: our first priority is to build a MTL model that uses compounds fingerprint as input
Evaluate our substrucuture scores( obtained from bipartite graph) using non-weighted methods such using aggregating methods
Build models that can learn from substructure scores

1.3 Results previous week

Trained data on simple MTL mlp model

import pandas as pd data = pd.read_csv('./training_dense_0925.csv') data.head()

data.shape

data_long = pd.melt(data, id_vars=["Smiles"]) data_long.rename(columns={"variable":"uniprot", "value":"label"}, inplace=True) data_long['label'].value_counts()

Data backgroud:

Data was constructed from training_data_0925.parquet
This is a dense dataset, compounds that have more than 200 interactions and targets that had more than 400 interactions were retained . The reason we need dense dataaset is we will be predicting on all interested targets and if there is a lot of nan chances are that the model won't learn anything due to the sparseness of the data
Number of unique compounds : 123483, Number of unique targets: 315
Though this is a fairly dense dataset (less number of nans), we have only about 2% positive interactions present
The starting data is very imbalanced. As you can see the starting positive% is only 2% , our assumption when starting this experiment was, as it is a multitask model correlated tasks would learn simultaneously and that would help with imbalance in the data

Basic MLP

Observation

Used the mlp structure on both ecfp and MAACs data
we did varaition of training a) vanilla training b) added penalty to loss while training
while training we calcualte loss only for data points that has values i.e we mask out nan's present in the data
This model had very bad metrics on BM3 with auc = 0.5 for both ecfp and MAACS fingerprint as input
Linear layers are weights initialized wtih kaiming_uniform

Notes

This model is way too simple to learn anything from the data
For maccs features that starts with 167 feature size, we predict 315 tasks in the end which also causes the model to learn nothing

import torch import torch.nn as nn from torchsummary import summary class Baseline_model(nn.Module): def __init__(self, ): super(Baseline_model, self).__init__() n_tasks = 315 in_feats = 167 out_feats = in_feats * 2 self.baseline = nn.Sequential(nn.Linear(in_feats, out_feats), nn.BatchNorm1d(out_feats), nn.Linear(out_feats, out_feats *2), nn.BatchNorm1d(out_feats *2), nn.Linear(out_feats *2, in_feats), nn.BatchNorm1d(in_feats), nn.Linear(in_feats, 315)) def forward(self, x): out = self.baseline(x) last_layer = nn.Sigmoid() last_out = last_layer(out) return last_out

Baseline_model()

Add results here

1.4 To do from previous week

Build deep models that could take fingerprint as input: assuming that would help with improvising metrics
First pass on MAACS substrucutre score data : Substructure scores are created using bipartite graphs. Goal here is to map training data parent compoudnds to respective substructure scores.

1.4.1 Model using Residual block and Spatial Gate

Residual convolutional block : This block mimics the residual block concept out = f(x)+x where x is the input.Here we use sequential conv_bn (Conv1d->BatchNorm1d ->Relu) as block()
Spatial gate: uses compressed version of input data and adds it back to the original data, acts as an attention layer between Residual convolution block
Main block: Acts like an auto encoder / U-net with encoder part and decoder part consitis of above mentioned blocks
Model was trained on both MAACS and ECFP features

import torch.nn as nn from collections import OrderedDict import torch def conv1x1(in_channels, out_channels, stride=1, kernel_size =3, padding =1): return nn.Conv1d(in_channels, out_channels, kernel_size=3, stride=1, padding=1, bias=True) def conv_bn(in_channels, out_channels): return nn.Sequential(OrderedDict({'conv': conv1x1(in_channels, out_channels), 'bn': nn.BatchNorm1d(out_channels) })) class ConvolutionResidualBlock(nn.Module): def __init__(self, in_channels, out_channels, stride=1, downsample=None): super(ConvolutionResidualBlock, self).__init__() self.conv1 = nn.Sequential(conv_bn(in_channels, out_channels), nn.ReLU(), conv_bn(out_channels, out_channels), nn.ReLU()) self.conv2 = conv1x1(in_channels, out_channels) self.activate = nn.ReLU() def forward(self, x): residual = x x = self.conv1(x) residual = self.conv2(residual) x += residual x = self.activate(x) return x class ChannelPool(nn.Module): def forward(self, x): return torch.cat( (torch.max(x,1)[0].unsqueeze(1), torch.mean(x,1).unsqueeze(1)), dim=1 ) class SpatialGate(nn.Module): def __init__(self, in_channel, out_channel): super(SpatialGate, self).__init__() kernel_size = 3 self.compress = ChannelPool() self.spatial = conv1x1(in_channel,out_channel, stride=1,kernel_size =3, padding=(kernel_size-1) // 2) def forward(self, x): x_compress = self.compress(x) x_out = self.spatial(x_compress) scale = nn.Sigmoid()(x_out) scale_out = x * scale return scale_out class Multitask_Attn(nn.Module): def __init__(self): super().__init__() in_channels = 1 out_channels = 6 self.encode1 = ConvolutionResidualBlock(in_channels, out_channels) self.encode2 = ConvolutionResidualBlock(out_channels , out_channels *4) self.spatial_attn1 = SpatialGate(2,out_channels *4) self.encode3 = ConvolutionResidualBlock(out_channels *4,out_channels *8) self.spatial_attn2 = SpatialGate(2,out_channels *8) self.decode_1 = ConvolutionResidualBlock(out_channels *8,out_channels *4) self.spatial_attn3 = SpatialGate(2,out_channels *4) self.decode_2 = ConvolutionResidualBlock(out_channels *4,out_channels) self.decode_3 = ConvolutionResidualBlock(out_channels, in_channels) self.spatial_attn4 = SpatialGate(2, 1) self.last_layer = nn.Linear(1 * 2048, 315)# change number here based on the feature size def forward(self, x): out = self.encode1(x) out = self.encode2(out) out = self.spatial_attn1(out) out = self.encode3(out) out = self.spatial_attn2(out) out = self.decode_1(out) out = self.spatial_attn3(out) out = self.decode_2(out) out = self.decode_3(out) out = self.spatial_attn4(out) out = out.view(-1, 1 * 2048) out = self.last_layer(out) return out

from torchsummary import summary summary(Multitask_Attn(), (1, 2048))

class Training: def __init__(self, model): self.model = model pos_weight = torch.Tensor([20]) self.train_loss = [] self.criterion = torch.nn.BCEWithLogitsLoss(pos_weight = pos_weight) self.optimizer = torch.optim.SGD(self.model.parameters(), lr=1e-3) def mask_nans(self, y_label, y_pred): mask = y_label.reshape((1, len(y_label) * y_label.shape[-1]))[0] index_to_drop = torch.isnan(mask) y_label = y_label.reshape((1, len(y_label) * y_label.shape[-1]))[0] predictions = y_pred.reshape((1, len(y_pred) * y_pred.shape[-1]))[0] masked_labels = y_label[~ index_to_drop] masked_preds = predictions[~index_to_drop] return masked_preds, masked_labels def train(self, dataloader, epochs): for i in range(epochs): batch_loss = 0 len_epoch = 0 for ind, (X, y) in enumerate(dataloader): y_predicted = self.model(X) masked_predictions, masked_true = self.mask_nans(y, y_predicted) loss = self.criterion(masked_predictions, masked_true) self.optimizer.zero_grad() masked_predictions = torch.nn.Sigmoid()(masked_predictions) masked_predictions = masked_predictions > 0.5 # penalty for false positive confusion_matrix = masked_predictions / masked_true false_negative = torch.sum(confusion_matrix == 0) false_negative_ratio = torch.Tensor([false_negative/ float(masked_true.shape[0])]).requires_grad_(True) loss = loss * (1 + false_negative_ratio) # change values if required loss.backward() self.optimizer.step() len_epoch += 1 batch_loss += loss.detach().numpy()[0] epoch_loss = batch_loss / len_epoch self.train_loss.append(epoch_loss) print(f'Epoch Loss {i + 1}: {epoch_loss}') torch.save(self.model.state_dict(), "./deep_model_ecfp_0127.pt") # Update to appropriate values E = 50 training_data = get_loader(training_data_ecfp, true_label, batch_size=1000) model = Multitask_Attn() train_loop = Training(model) train_loop.train(training_data, E)

1.4.2 Residual block with different basic blocks parameters

Same architecture as above but used different activation layer SiLU as mentioned here https://arxiv.org/abs/1710.05941v1
SELU works better than RELu dealing with vanishing gradients problem

import torch.nn as nn from collections import OrderedDict import torch def conv1x1(in_channels, out_channels, stride=1, kernel_size =3, padding =1): return nn.Conv1d(in_channels, out_channels, kernel_size=3, stride=1, padding=1, bias=True) def conv_bn(in_channels, out_channels): return nn.Sequential(OrderedDict({'conv': conv1x1(in_channels, out_channels), 'bn': nn.BatchNorm1d(out_channels) })) class ConvolutionResidualBlock(nn.Module): def __init__(self, in_channels, out_channels, stride=1, downsample=None): super(ConvolutionResidualBlock, self).__init__() self.conv1 = nn.Sequential(conv_bn(in_channels, out_channels), nn.SiLU(), conv_bn(out_channels, out_channels), nn.SiLU()) self.conv2 = conv1x1(in_channels, out_channels) self.activate = nn.SiLU() def forward(self, x): residual = x x = self.conv1(x) residual = self.conv2(residual) x += residual x = self.activate(x) return x class ChannelPool(nn.Module): def forward(self, x): return torch.cat( (torch.max(x,1)[0].unsqueeze(1), torch.mean(x,1).unsqueeze(1)), dim=1 ) class SpatialGate(nn.Module): def __init__(self, in_channel, out_channel): super(SpatialGate, self).__init__() kernel_size = 3 self.compress = ChannelPool() self.spatial = conv1x1(in_channel,out_channel, stride=1,kernel_size =3, padding=(kernel_size-1) // 2) def forward(self, x): x_compress = self.compress(x) x_out = self.spatial(x_compress) scale = nn.Sigmoid()(x_out) scale_out = x * scale return scale_out class Multitask_Attn(nn.Module): def __init__(self): super().__init__() in_channels = 1 out_channels = 6 self.encode1 = ConvolutionResidualBlock(in_channels, out_channels) self.encode2 = ConvolutionResidualBlock(out_channels , out_channels *4) self.spatial_attn1 = SpatialGate(2,out_channels *4) self.encode3 = ConvolutionResidualBlock(out_channels *4,out_channels *8) self.spatial_attn2 = SpatialGate(2,out_channels *8) self.decode_1 = ConvolutionResidualBlock(out_channels *8,out_channels *4) self.spatial_attn3 = SpatialGate(2,out_channels *4) self.decode_2 = ConvolutionResidualBlock(out_channels *4,out_channels) self.decode_3 = ConvolutionResidualBlock(out_channels, in_channels) self.spatial_attn4 = SpatialGate(2, 1) self.last_layer = nn.Linear(1 * 2048, 315)# change number here based on the feature size def forward(self, x): out = self.encode1(x) out = self.encode2(out) out = self.spatial_attn1(out) out = self.encode3(out) out = self.spatial_attn2(out) out = self.decode_1(out) out = self.spatial_attn3(out) out = self.decode_2(out) out = self.decode_3(out) out = self.spatial_attn4(out) out = out.view(-1, 1 * 2048) out = self.last_layer(out) return out