YOLOv11改进 | 融合改进篇 | Damo-YOLO融合SwinTransformer轻量化创新yolov11（可替换专栏20+主干）

一、本文介绍

本文给大家带来的最新改进机制是融合改进，利用 Damo-YOLO 配合 SwinTransformer ，其中Damo- YOLO 和 SwinTransformer 本文的SwinTransformer可以替换为本专栏的20余种主干。

欢迎大家订阅我的专栏一起学习YOLO！

训练信息：YOLO11-SwinTransformer-RepGFPN summary: 353 layers, 2,736,687 parameters, 2,736,671 gradients, 6.3 GFLOPs
基础版本：YOLO11 summary: 319 layers, 2,591,010 parameters, 2,590,994 gradients, 6.4 GFLOPs

目录

一、本文介绍

二、原理介绍

三、Damo-YOLO核心代码使用方式

3.1 修改一

3.2 修改二

3.3 修改三

3.4 修改四

四、 Swintransformer的核心代码和使用方式

4.1 修改一

4.2 修改二

4.3 修改三

4.4 修改四

4.5 修改五

4.6 修改六

4.7 修改七

4.8 修改八

4.9 修改九

五、融合使用教程

六、成功运行截图

七、本文总结

二、原理介绍

这部分就不复制过来了，大家想看的可以去对应文章找。

Damo-YOLO： Damo-YOLO的地址

Swintransformer： YOLOv11改进 | 主干/Backbone篇 | 视觉变换器SwinTransformer目标检测网络（ 适配yolov11全系列模型）_yolov11代码-CSDN博客

三、Damo-YOLO核心代码使用方式

下面的代码是GFPN的核心代码，我们将其复制导' ultralytics /nn'目录下，在其中创建一个文件，我这里起名为GFPN然后粘贴进去，其余使用方式看后面。 

  1. import torch
  2. import torch.nn as nn
  3. import numpy as np
  4.  
  5.  
  6. class swish(nn.Module):
  7.     def forward(self, x):
  8.         return x * torch.sigmoid(x)
  9.  
  10. def autopad(k, p=None, d=1):  # kernel, padding, dilation
  11.     """Pad to 'same' shape outputs."""
  12.     if d > 1:
  13.         k = d * (k - 1) + 1 if isinstance(k, int) else [d * (x - 1) + 1 for x in k]  # actual kernel-size
  14.     if p is None:
  15.         p = k // 2 if isinstance(k, int) else [x // 2 for x in k]  # auto-pad
  16.     return p
  17.  
  18. class Conv(nn.Module):
  19.     """Standard convolution with args(ch_in, ch_out, kernel, stride, padding, groups, dilation, activation)."""
  20.     default_act = swish()  # default activation
  21.  
  22.     def __init__(self, c1, c2, k=1, s=1, p=None, g=1, d=1, act=True):
  23.         """Initialize Conv layer with given arguments including activation."""
  24.         super().__init__()
  25.         self.conv = nn.Conv2d(c1, c2, k, s, autopad(k, p, d), groups=g, dilation=d, bias=False)
  26.         self.bn = nn.BatchNorm2d(c2)
  27.         self.act = self.default_act if act is True else act if isinstance(act, nn.Module) else nn.Identity()
  28.  
  29.     def forward(self, x):
  30.         """Apply convolution, batch normalization and activation to input tensor."""
  31.         return self.act(self.bn(self.conv(x)))
  32.  
  33.     def forward_fuse(self, x):
  34.         """Perform transposed convolution of 2D data."""
  35.         return self.act(self.conv(x))
  36.  
  37. class RepConv(nn.Module):
  38.     default_act = swish()  # default activation
  39.  
  40.     def __init__(self, c1, c2, k=3, s=1, p=1, g=1, d=1, act=True, bn=False, deploy=False):
  41.         """Initializes Light Convolution layer with inputs, outputs & optional activation function."""
  42.         super().__init__()
  43.         assert k == 3 and p == 1
  44.         self.g = g
  45.         self.c1 = c1
  46.         self.c2 = c2
  47.         self.act = self.default_act if act is True else act if isinstance(act, nn.Module) else nn.Identity()
  48.  
  49.         self.bn = nn.BatchNorm2d(num_features=c1) if bn and c2 == c1 and s == 1 else None
  50.         self.conv1 = Conv(c1, c2, k, s, p=p, g=g, act=False)
  51.         self.conv2 = Conv(c1, c2, 1, s, p=(p - k // 2), g=g, act=False)
  52.  
  53.     def forward_fuse(self, x):
  54.         """Forward process."""
  55.         return self.act(self.conv(x))
  56.  
  57.     def forward(self, x):
  58.         """Forward process."""
  59.         id_out = 0 if self.bn is None else self.bn(x)
  60.         return self.act(self.conv1(x) + self.conv2(x) + id_out)
  61.  
  62.     def get_equivalent_kernel_bias(self):
  63.         """Returns equivalent kernel and bias by adding 3x3 kernel, 1x1 kernel and identity kernel with their biases."""
  64.         kernel3x3, bias3x3 = self._fuse_bn_tensor(self.conv1)
  65.         kernel1x1, bias1x1 = self._fuse_bn_tensor(self.conv2)
  66.         kernelid, biasid = self._fuse_bn_tensor(self.bn)
  67.         return kernel3x3 + self._pad_1x1_to_3x3_tensor(kernel1x1) + kernelid, bias3x3 + bias1x1 + biasid
  68.  
  69.     def _pad_1x1_to_3x3_tensor(self, kernel1x1):
  70.         """Pads a 1x1 tensor to a 3x3 tensor."""
  71.         if kernel1x1 is None:
  72.             return 0
  73.         else:
  74.             return torch.nn.functional.pad(kernel1x1, [1, 1, 1, 1])
  75.  
  76.     def _fuse_bn_tensor(self, branch):
  77.         """Generates appropriate kernels and biases for convolution by fusing branches of the neural network."""
  78.         if branch is None:
  79.             return 0, 0
  80.         if isinstance(branch, Conv):
  81.             kernel = branch.conv.weight
  82.             running_mean = branch.bn.running_mean
  83.             running_var = branch.bn.running_var
  84.             gamma = branch.bn.weight
  85.             beta = branch.bn.bias
  86.             eps = branch.bn.eps
  87.         elif isinstance(branch, nn.BatchNorm2d):
  88.             if not hasattr(self, 'id_tensor'):
  89.                 input_dim = self.c1 // self.g
  90.                 kernel_value = np.zeros((self.c1, input_dim, 3, 3), dtype=np.float32)
  91.                 for i in range(self.c1):
  92.                     kernel_value[i, i % input_dim, 1, 1] = 1
  93.                 self.id_tensor = torch.from_numpy(kernel_value).to(branch.weight.device)
  94.             kernel = self.id_tensor
  95.             running_mean = branch.running_mean
  96.             running_var = branch.running_var
  97.             gamma = branch.weight
  98.             beta = branch.bias
  99.             eps = branch.eps
  100.         std = (running_var + eps).sqrt()
  101.         t = (gamma / std).reshape(-1, 1, 1, 1)
  102.         return kernel * t, beta - running_mean * gamma / std
  103.  
  104.     def fuse_convs(self):
  105.         """Combines two convolution layers into a single layer and removes unused attributes from the class."""
  106.         if hasattr(self, 'conv'):
  107.             return
  108.         kernel, bias = self.get_equivalent_kernel_bias()
  109.         self.conv = nn.Conv2d(in_channels=self.conv1.conv.in_channels,
  110.                               out_channels=self.conv1.conv.out_channels,
  111.                               kernel_size=self.conv1.conv.kernel_size,
  112.                               stride=self.conv1.conv.stride,
  113.                               padding=self.conv1.conv.padding,
  114.                               dilation=self.conv1.conv.dilation,
  115.                               groups=self.conv1.conv.groups,
  116.                               bias=True).requires_grad_(False)
  117.         self.conv.weight.data = kernel
  118.         self.conv.bias.data = bias
  119.         for para in self.parameters():
  120.             para.detach_()
  121.         self.__delattr__('conv1')
  122.         self.__delattr__('conv2')
  123.         if hasattr(self, 'nm'):
  124.             self.__delattr__('nm')
  125.         if hasattr(self, 'bn'):
  126.             self.__delattr__('bn')
  127.         if hasattr(self, 'id_tensor'):
  128.             self.__delattr__('id_tensor')
  129.  
  130. class BasicBlock_3x3_Reverse(nn.Module):
  131.     def __init__(self,
  132.                  ch_in,
  133.                  ch_hidden_ratio,
  134.                  ch_out,
  135.                  shortcut=True):
  136.         super(BasicBlock_3x3_Reverse, self).__init__()
  137.         assert ch_in == ch_out
  138.         ch_hidden = int(ch_in * ch_hidden_ratio)
  139.         self.conv1 = Conv(ch_hidden, ch_out, 3, s=1)
  140.         self.conv2 = RepConv(ch_in, ch_hidden, 3, s=1)
  141.         self.shortcut = shortcut
  142.  
  143.     def forward(self, x):
  144.         y = self.conv2(x)
  145.         y = self.conv1(y)
  146.         if self.shortcut:
  147.             return x + y
  148.         else:
  149.             return y
  150.  
  151. class SPP(nn.Module):
  152.     def __init__(
  153.         self,
  154.         ch_in,
  155.         ch_out,
  156.         k,
  157.         pool_size
  158.     ):
  159.         super(SPP, self).__init__()
  160.         self.pool = []
  161.         for i, size in enumerate(pool_size):
  162.             pool = nn.MaxPool2d(kernel_size=size,
  163.                                 stride=1,
  164.                                 padding=size // 2,
  165.                                 ceil_mode=False)
  166.             self.add_module('pool{}'.format(i), pool)
  167.             self.pool.append(pool)
  168.         self.conv = Conv(ch_in, ch_out, k)
  169.  
  170.     def forward(self, x):
  171.         outs = [x]
  172.  
  173.         for pool in self.pool:
  174.             outs.append(pool(x))
  175.         y = torch.cat(outs, axis=1)
  176.  
  177.         y = self.conv(y)
  178.         return y
  179.  
  180. class CSPStage(nn.Module):
  181.     def __init__(self,
  182.                  ch_in,
  183.                  ch_out,
  184.                  n,
  185.                  block_fn='BasicBlock_3x3_Reverse',
  186.                  ch_hidden_ratio=1.0,
  187.                  act='silu',
  188.                  spp=False):
  189.         super(CSPStage, self).__init__()
  190.  
  191.         split_ratio = 2
  192.         ch_first = int(ch_out // split_ratio)
  193.         ch_mid = int(ch_out - ch_first)
  194.         self.conv1 = Conv(ch_in, ch_first, 1)
  195.         self.conv2 = Conv(ch_in, ch_mid, 1)
  196.         self.convs = nn.Sequential()
  197.  
  198.         next_ch_in = ch_mid
  199.         for i in range(n):
  200.             if block_fn == 'BasicBlock_3x3_Reverse':
  201.                 self.convs.add_module(
  202.                     str(i),
  203.                     BasicBlock_3x3_Reverse(next_ch_in,
  204.                                            ch_hidden_ratio,
  205.                                            ch_mid,
  206.                                            shortcut=True))
  207.             else:
  208.                 raise NotImplementedError
  209.             if i == (n - 1) // 2 and spp:
  210.                 self.convs.add_module('spp', SPP(ch_mid * 4, ch_mid, 1, [5, 9, 13]))
  211.             next_ch_in = ch_mid
  212.         self.conv3 = Conv(ch_mid * n + ch_first, ch_out, 1)
  213.  
  214.     def forward(self, x):
  215.         y1 = self.conv1(x)
  216.         y2 = self.conv2(x)
  217.  
  218.         mid_out = [y1]
  219.         for conv in self.convs:
  220.             y2 = conv(y2)
  221.             mid_out.append(y2)
  222.         y = torch.cat(mid_out, axis=1)
  223.         y = self.conv3(y)
  224.         return y

3.1 修改一

第一还是建立文件，我们找到如下ultralytics/nn文件夹下建立一个目录名字呢就是'Addmodules'文件夹( ！然后在其内部建立一个新的py文件将核心代码复制粘贴进去即可。

​

3.2 修改二

第二步我们在该目录下创建一个新的py文件名字为'__init__.py'( ，然后在其内部导入我们的检测头如下图所示。

3.3 修改三

第三步我门中到如下文件'ultralytics/nn/tasks.py'进行导入和注册我们的模块( ！

​

3.4 修改四

按照我的添加在parse_model里添加即可。

到此就修改完成了，大家可以复制下面的yaml文件运行。

四、 Swintransformer的核心代码和使用方式

Swintransformer的核心代码，使用方式看后面介绍。 

  1. import torch
  2. import torch.nn as nn
  3. import torch.nn.functional as F
  4. import torch.utils.checkpoint as checkpoint
  5. import numpy as np
  6. from timm.models.layers import DropPath, to_2tuple, trunc_normal_
  7.  
  8. __all__ = ['SwinTransformer']
  9.  
  10. class Mlp(nn.Module):
  11.     """ Multilayer perceptron."""
  12.  
  13.     def __init__(self, in_features, hidden_features=None, out_features=None, act_layer=nn.GELU, drop=0.):
  14.         super().__init__()
  15.         out_features = out_features or in_features
  16.         hidden_features = hidden_features or in_features
  17.         self.fc1 = nn.Linear(in_features, hidden_features)
  18.         self.act = act_layer()
  19.         self.fc2 = nn.Linear(hidden_features, out_features)
  20.         self.drop = nn.Dropout(drop)
  21.  
  22.     def forward(self, x):
  23.         x = self.fc1(x)
  24.         x = self.act(x)
  25.         x = self.drop(x)
  26.         x = self.fc2(x)
  27.         x = self.drop(x)
  28.         return x
  29.  
  30.  
  31. def window_partition(x, window_size):
  32.     """
  33.     Args:
  34.         x: (B, H, W, C)
  35.         window_size (int): window size
  37.     Returns:
  38.         windows: (num_windows*B, window_size, window_size, C)
  39.     """
  40.     B, H, W, C = x.shape
  41.     x = x.view(B, H // window_size, window_size, W // window_size, window_size, C)
  42.     windows = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(-1, window_size, window_size, C)
  43.     return windows
  44.  
  45.  
  46. def window_reverse(windows, window_size, H, W):
  47.     """
  48.     Args:
  49.         windows: (num_windows*B, window_size, window_size, C)
  50.         window_size (int): Window size
  51.         H (int): Height of image
  52.         W (int): Width of image
  54.     Returns:
  55.         x: (B, H, W, C)
  56.     """
  57.     B = int(windows.shape[0] / (H * W / window_size / window_size))
  58.     x = windows.view(B, H // window_size, W // window_size, window_size, window_size, -1)
  59.     x = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(B, H, W, -1)
  60.     return x
  61.  
  62.  
  63. class WindowAttention(nn.Module):
  64.     """ Window based multi-head self attention (W-MSA) module with relative position bias.
  65.     It supports both of shifted and non-shifted window.
  67.     Args:
  68.         dim (int): Number of input channels.
  69.         window_size (tuple[int]): The height and width of the window.
  70.         num_heads (int): Number of attention heads.
  71.         qkv_bias (bool, optional):  If True, add a learnable bias to query, key, value. Default: True
  72.         qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set
  73.         attn_drop (float, optional): Dropout ratio of attention weight. Default: 0.0
  74.         proj_drop (float, optional): Dropout ratio of output. Default: 0.0
  75.     """
  76.  
  77.     def __init__(self, dim, window_size, num_heads, qkv_bias=True, qk_scale=None, attn_drop=0., proj_drop=0.):
  78.  
  79.         super().__init__()
  80.         self.dim = dim
  81.         self.window_size = window_size  # Wh, Ww
  82.         self.num_heads = num_heads
  83.         head_dim = dim // num_heads
  84.         self.scale = qk_scale or head_dim ** -0.5
  85.  
  86.         # define a parameter table of relative position bias
  87.         self.relative_position_bias_table = nn.Parameter(
  88.             torch.zeros((2 * window_size[0] - 1) * (2 * window_size[1] - 1), num_heads))  # 2*Wh-1 * 2*Ww-1, nH
  89.  
  90.         # get pair-wise relative position index for each token inside the window
  91.         coords_h = torch.arange(self.window_size[0])
  92.         coords_w = torch.arange(self.window_size[1])
  93.         coords = torch.stack(torch.meshgrid([coords_h, coords_w]))  # 2, Wh, Ww
  94.         coords_flatten = torch.flatten(coords, 1)  # 2, Wh*Ww
  95.         relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :]  # 2, Wh*Ww, Wh*Ww
  96.         relative_coords = relative_coords.permute(1, 2, 0).contiguous()  # Wh*Ww, Wh*Ww, 2
  97.         relative_coords[:, :, 0] += self.window_size[0] - 1  # shift to start from 0
  98.         relative_coords[:, :, 1] += self.window_size[1] - 1
  99.         relative_coords[:, :, 0] *= 2 * self.window_size[1] - 1
  100.         relative_position_index = relative_coords.sum(-1)  # Wh*Ww, Wh*Ww
  101.         self.register_buffer("relative_position_index", relative_position_index)
  102.  
  103.         self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias)
  104.         self.attn_drop = nn.Dropout(attn_drop)
  105.         self.proj = nn.Linear(dim, dim)
  106.         self.proj_drop = nn.Dropout(proj_drop)
  107.  
  108.         trunc_normal_(self.relative_position_bias_table, std=.02)
  109.         self.softmax = nn.Softmax(dim=-1)
  110.  
  111.     def forward(self, x, mask=None):
  112.         """ Forward function.
  114.         Args:
  115.             x: input features with shape of (num_windows*B, N, C)
  116.             mask: (0/-inf) mask with shape of (num_windows, Wh*Ww, Wh*Ww) or None
  117.         """
  118.         B_, N, C = x.shape
  119.         qkv = self.qkv(x).reshape(B_, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)
  120.         q, k, v = qkv[0], qkv[1], qkv[2]  # make torchscript happy (cannot use tensor as tuple)
  121.  
  122.         q = q * self.scale
  123.         attn = (q @ k.transpose(-2, -1))
  124.  
  125.         relative_position_bias = self.relative_position_bias_table[self.relative_position_index.view(-1)].view(
  126.             self.window_size[0] * self.window_size[1], self.window_size[0] * self.window_size[1], -1)  # Wh*Ww,Wh*Ww,nH
  127.         relative_position_bias = relative_position_bias.permute(2, 0, 1).contiguous()  # nH, Wh*Ww, Wh*Ww
  128.         attn = attn + relative_position_bias.unsqueeze(0)
  129.  
  130.         if mask is not None:
  131.             nW = mask.shape[0]
  132.             attn = attn.view(B_ // nW, nW, self.num_heads, N, N) + mask.unsqueeze(1).unsqueeze(0)
  133.             attn = attn.view(-1, self.num_heads, N, N)
  134.             attn = self.softmax(attn)
  135.         else:
  136.             attn = self.softmax(attn)
  137.  
  138.         attn = self.attn_drop(attn)
  139.  
  140.         x = (attn @ v).transpose(1, 2).reshape(B_, N, C)
  141.         x = self.proj(x)
  142.         x = self.proj_drop(x)
  143.         return x
  144.  
  145.  
  146. class SwinTransformerBlock(nn.Module):
  147.     """ Swin Transformer Block.
  149.     Args:
  150.         dim (int): Number of input channels.
  151.         num_heads (int): Number of attention heads.
  152.         window_size (int): Window size.
  153.         shift_size (int): Shift size for SW-MSA.
  154.         mlp_ratio (float): Ratio of mlp hidden dim to embedding dim.
  155.         qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True
  156.         qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set.
  157.         drop (float, optional): Dropout rate. Default: 0.0
  158.         attn_drop (float, optional): Attention dropout rate. Default: 0.0
  159.         drop_path (float, optional): Stochastic depth rate. Default: 0.0
  160.         act_layer (nn.Module, optional): Activation layer. Default: nn.GELU
  161.         norm_layer (nn.Module, optional): Normalization layer.  Default: nn.LayerNorm
  162.     """
  163.  
  164.     def __init__(self, dim, num_heads, window_size=7, shift_size=0,
  165.                  mlp_ratio=4., qkv_bias=True, qk_scale=None, drop=0., attn_drop=0., drop_path=0.,
  166.                  act_layer=nn.GELU, norm_layer=nn.LayerNorm):
  167.         super().__init__()
  168.         self.dim = dim
  169.         self.num_heads = num_heads
  170.         self.window_size = window_size
  171.         self.shift_size = shift_size
  172.         self.mlp_ratio = mlp_ratio
  173.         assert 0 <= self.shift_size < self.window_size, "shift_size must in 0-window_size"
  174.  
  175.         self.norm1 = norm_layer(dim)
  176.         self.attn = WindowAttention(
  177.             dim, window_size=to_2tuple(self.window_size), num_heads=num_heads,
  178.             qkv_bias=qkv_bias, qk_scale=qk_scale, attn_drop=attn_drop, proj_drop=drop)
  179.  
  180.         self.drop_path = DropPath(drop_path) if drop_path > 0. else nn.Identity()
  181.         self.norm2 = norm_layer(dim)
  182.         mlp_hidden_dim = int(dim * mlp_ratio)
  183.         self.mlp = Mlp(in_features=dim, hidden_features=mlp_hidden_dim, act_layer=act_layer, drop=drop)
  184.  
  185.         self.H = None
  186.         self.W = None
  187.  
  188.     def forward(self, x, mask_matrix):
  189.         """ Forward function.
  191.         Args:
  192.             x: Input feature, tensor size (B, H*W, C).
  193.             H, W: Spatial resolution of the input feature.
  194.             mask_matrix: Attention mask for cyclic shift.
  195.         """
  196.         B, L, C = x.shape
  197.         H, W = self.H, self.W
  198.         assert L == H * W, "input feature has wrong size"
  199.  
  200.         shortcut = x
  201.         x = self.norm1(x)
  202.         x = x.view(B, H, W, C)
  203.  
  204.         # pad feature maps to multiples of window size
  205.         pad_l = pad_t = 0
  206.         pad_r = (self.window_size - W % self.window_size) % self.window_size
  207.         pad_b = (self.window_size - H % self.window_size) % self.window_size
  208.         x = F.pad(x, (0, 0, pad_l, pad_r, pad_t, pad_b))
  209.         _, Hp, Wp, _ = x.shape
  210.  
  211.         # cyclic shift
  212.         if self.shift_size > 0:
  213.             shifted_x = torch.roll(x, shifts=(-self.shift_size, -self.shift_size), dims=(1, 2))
  214.             attn_mask = mask_matrix.type(x.dtype)
  215.         else:
  216.             shifted_x = x
  217.             attn_mask = None
  218.  
  219.         # partition windows
  220.         x_windows = window_partition(shifted_x, self.window_size)  # nW*B, window_size, window_size, C
  221.         x_windows = x_windows.view(-1, self.window_size * self.window_size, C)  # nW*B, window_size*window_size, C
  222.  
  223.         # W-MSA/SW-MSA
  224.         attn_windows = self.attn(x_windows, mask=attn_mask)  # nW*B, window_size*window_size, C
  225.  
  226.         # merge windows
  227.         attn_windows = attn_windows.view(-1, self.window_size, self.window_size, C)
  228.         shifted_x = window_reverse(attn_windows, self.window_size, Hp, Wp)  # B H' W' C
  229.  
  230.         # reverse cyclic shift
  231.         if self.shift_size > 0:
  232.             x = torch.roll(shifted_x, shifts=(self.shift_size, self.shift_size), dims=(1, 2))
  233.         else:
  234.             x = shifted_x
  235.  
  236.         if pad_r > 0 or pad_b > 0:
  237.             x = x[:, :H, :W, :].contiguous()
  238.  
  239.         x = x.view(B, H * W, C)
  240.  
  241.         # FFN
  242.         x = shortcut + self.drop_path(x)
  243.         x = x + self.drop_path(self.mlp(self.norm2(x)))
  244.  
  245.         return x
  246.  
  247.  
  248. class PatchMerging(nn.Module):
  249.     """ Patch Merging Layer
  251.     Args:
  252.         dim (int): Number of input channels.
  253.         norm_layer (nn.Module, optional): Normalization layer.  Default: nn.LayerNorm
  254.     """
  255.     def __init__(self, dim, norm_layer=nn.LayerNorm):
  256.         super().__init__()
  257.         self.dim = dim
  258.         self.reduction = nn.Linear(4 * dim, 2 * dim, bias=False)
  259.         self.norm = norm_layer(4 * dim)
  260.  
  261.     def forward(self, x, H, W):
  262.         """ Forward function.
  264.         Args:
  265.             x: Input feature, tensor size (B, H*W, C).
  266.             H, W: Spatial resolution of the input feature.
  267.         """
  268.         B, L, C = x.shape
  269.         assert L == H * W, "input feature has wrong size"
  270.  
  271.         x = x.view(B, H, W, C)
  272.  
  273.         # padding
  274.         pad_input = (H % 2 == 1) or (W % 2 == 1)
  275.         if pad_input:
  276.             x = F.pad(x, (0, 0, 0, W % 2, 0, H % 2))
  277.  
  278.         x0 = x[:, 0::2, 0::2, :]  # B H/2 W/2 C
  279.         x1 = x[:, 1::2, 0::2, :]  # B H/2 W/2 C
  280.         x2 = x[:, 0::2, 1::2, :]  # B H/2 W/2 C
  281.         x3 = x[:, 1::2, 1::2, :]  # B H/2 W/2 C
  282.         x = torch.cat([x0, x1, x2, x3], -1)  # B H/2 W/2 4*C
  283.         x = x.view(B, -1, 4 * C)  # B H/2*W/2 4*C
  284.  
  285.         x = self.norm(x)
  286.         x = self.reduction(x)
  287.  
  288.         return x
  289.  
  290.  
  291. class BasicLayer(nn.Module):
  292.     """ A basic Swin Transformer layer for one stage.
  294.     Args:
  295.         dim (int): Number of feature channels
  296.         depth (int): Depths of this stage.
  297.         num_heads (int): Number of attention head.
  298.         window_size (int): Local window size. Default: 7.
  299.         mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4.
  300.         qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True
  301.         qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set.
  302.         drop (float, optional): Dropout rate. Default: 0.0
  303.         attn_drop (float, optional): Attention dropout rate. Default: 0.0
  304.         drop_path (float | tuple[float], optional): Stochastic depth rate. Default: 0.0
  305.         norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm
  306.         downsample (nn.Module | None, optional): Downsample layer at the end of the layer. Default: None
  307.         use_checkpoint (bool): Whether to use checkpointing to save memory. Default: False.
  308.     """
  309.  
  310.     def __init__(self,
  311.                  dim,
  312.                  depth,
  313.                  num_heads,
  314.                  window_size=7,
  315.                  mlp_ratio=4.,
  316.                  qkv_bias=True,
  317.                  qk_scale=None,
  318.                  drop=0.,
  319.                  attn_drop=0.,
  320.                  drop_path=0.,
  321.                  norm_layer=nn.LayerNorm,
  322.                  downsample=None,
  323.                  use_checkpoint=False):
  324.         super().__init__()
  325.         self.window_size = window_size
  326.         self.shift_size = window_size // 2
  327.         self.depth = depth
  328.         self.use_checkpoint = use_checkpoint
  329.  
  330.         # build blocks
  331.         self.blocks = nn.ModuleList([
  332.             SwinTransformerBlock(
  333.                 dim=dim,
  334.                 num_heads=num_heads,
  335.                 window_size=window_size,
  336.                 shift_size=0 if (i % 2 == 0) else window_size // 2,
  337.                 mlp_ratio=mlp_ratio,
  338.                 qkv_bias=qkv_bias,
  339.                 qk_scale=qk_scale,
  340.                 drop=drop,
  341.                 attn_drop=attn_drop,
  342.                 drop_path=drop_path[i] if isinstance(drop_path, list) else drop_path,
  343.                 norm_layer=norm_layer)
  344.             for i in range(depth)])
  345.  
  346.         # patch merging layer
  347.         if downsample is not None:
  348.             self.downsample = downsample(dim=dim, norm_layer=norm_layer)
  349.         else:
  350.             self.downsample = None
  351.  
  352.     def forward(self, x, H, W):
  353.         """ Forward function.
  355.         Args:
  356.             x: Input feature, tensor size (B, H*W, C).
  357.             H, W: Spatial resolution of the input feature.
  358.         """
  359.  
  360.         # calculate attention mask for SW-MSA
  361.         Hp = int(np.ceil(H / self.window_size)) * self.window_size
  362.         Wp = int(np.ceil(W / self.window_size)) * self.window_size
  363.         img_mask = torch.zeros((1, Hp, Wp, 1), device=x.device)  # 1 Hp Wp 1
  364.         h_slices = (slice(0, -self.window_size),
  365.                     slice(-self.window_size, -self.shift_size),
  366.                     slice(-self.shift_size, None))
  367.         w_slices = (slice(0, -self.window_size),
  368.                     slice(-self.window_size, -self.shift_size),
  369.                     slice(-self.shift_size, None))
  370.         cnt = 0
  371.         for h in h_slices:
  372.             for w in w_slices:
  373.                 img_mask[:, h, w, :] = cnt
  374.                 cnt += 1
  375.  
  376.         mask_windows = window_partition(img_mask, self.window_size)  # nW, window_size, window_size, 1
  377.         mask_windows = mask_windows.view(-1, self.window_size * self.window_size)
  378.         attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2)
  379.         attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))
  380.  
  381.         for blk in self.blocks:
  382.             blk.H, blk.W = H, W
  383.             if self.use_checkpoint:
  384.                 x = checkpoint.checkpoint(blk, x, attn_mask)
  385.             else:
  386.                 x = blk(x, attn_mask)
  387.         if self.downsample is not None:
  388.             x_down = self.downsample(x, H, W)
  389.             Wh, Ww = (H + 1) // 2, (W + 1) // 2
  390.             return x, H, W, x_down, Wh, Ww
  391.         else:
  392.             return x, H, W, x, H, W
  393.  
  394.  
  395. class PatchEmbed(nn.Module):
  396.     """ Image to Patch Embedding
  398.     Args:
  399.         patch_size (int): Patch token size. Default: 4.
  400.         in_chans (int): Number of input image channels. Default: 3.
  401.         embed_dim (int): Number of linear projection output channels. Default: 96.
  402.         norm_layer (nn.Module, optional): Normalization layer. Default: None
  403.     """
  404.  
  405.     def __init__(self, patch_size=4, in_chans=3, embed_dim=96, norm_layer=None):
  406.         super().__init__()
  407.         patch_size = to_2tuple(patch_size)
  408.         self.patch_size = patch_size
  409.  
  410.         self.in_chans = in_chans
  411.         self.embed_dim = embed_dim
  412.  
  413.         self.proj = nn.Conv2d(in_chans, embed_dim, kernel_size=patch_size, stride=patch_size)
  414.         if norm_layer is not None:
  415.             self.norm = norm_layer(embed_dim)
  416.         else:
  417.             self.norm = None
  418.  
  419.     def forward(self, x):
  420.         """Forward function."""
  421.         # padding
  422.         _, _, H, W = x.size()
  423.         if W % self.patch_size[1] != 0:
  424.             x = F.pad(x, (0, self.patch_size[1] - W % self.patch_size[1]))
  425.         if H % self.patch_size[0] != 0:
  426.             x = F.pad(x, (0, 0, 0, self.patch_size[0] - H % self.patch_size[0]))
  427.  
  428.         x = self.proj(x)  # B C Wh Ww
  429.         if self.norm is not None:
  430.             Wh, Ww = x.size(2), x.size(3)
  431.             x = x.flatten(2).transpose(1, 2)
  432.             x = self.norm(x)
  433.             x = x.transpose(1, 2).view(-1, self.embed_dim, Wh, Ww)
  434.  
  435.         return x
  436.  
  437. class SwinTransformer(nn.Module):
  438.     """ Swin Transformer backbone.
  439.         A PyTorch impl of : `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows`  -
  440.           https://arxiv.org/pdf/2103.14030
  442.     Args:
  443.         pretrain_img_size (int): Input image size for training the pretrained model,
  444.             used in absolute postion embedding. Default 224.
  445.         patch_size (int | tuple(int)): Patch size. Default: 4.
  446.         in_chans (int): Number of input image channels. Default: 3.
  447.         embed_dim (int): Number of linear projection output channels. Default: 96.
  448.         depths (tuple[int]): Depths of each Swin Transformer stage.
  449.         num_heads (tuple[int]): Number of attention head of each stage.
  450.         window_size (int): Window size. Default: 7.
  451.         mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4.
  452.         qkv_bias (bool): If True, add a learnable bias to query, key, value. Default: True
  453.         qk_scale (float): Override default qk scale of head_dim ** -0.5 if set.
  454.         drop_rate (float): Dropout rate.
  455.         attn_drop_rate (float): Attention dropout rate. Default: 0.
  456.         drop_path_rate (float): Stochastic depth rate. Default: 0.2.
  457.         norm_layer (nn.Module): Normalization layer. Default: nn.LayerNorm.
  458.         ape (bool): If True, add absolute position embedding to the patch embedding. Default: False.
  459.         patch_norm (bool): If True, add normalization after patch embedding. Default: True.
  460.         out_indices (Sequence[int]): Output from which stages.
  461.         frozen_stages (int): Stages to be frozen (stop grad and set eval mode).
  462.             -1 means not freezing any parameters.
  463.         use_checkpoint (bool): Whether to use checkpointing to save memory. Default: False.
  464.     """
  465.  
  466.     def __init__(self,
  467.                  factor=0.5,
  468.                  depth_factor=0.5,
  469.                  pretrain_img_size=224,
  470.                  patch_size=4,
  471.                  in_chans=3,
  472.                  embed_dim=96,
  473.                  depths=[2, 2, 6, 2],
  474.                  num_heads=[3, 6, 12, 24],
  475.                  window_size=7,
  476.                  mlp_ratio=4.,
  477.                  qkv_bias=True,
  478.                  qk_scale=None,
  479.                  drop_rate=0.,
  480.                  attn_drop_rate=0.,
  481.                  drop_path_rate=0.2,
  482.                  norm_layer=nn.LayerNorm,
  483.                  ape=False,
  484.                  patch_norm=True,
  485.                  out_indices=(0, 1, 2, 3),
  486.                  frozen_stages=-1,
  487.                  use_checkpoint=False):
  488.         super().__init__()
  489.         embed_dim = int(embed_dim * factor)
  490.         depths = [max(1, int(dim * depth_factor)) for dim in depths]
  491.         self.pretrain_img_size = pretrain_img_size
  492.         self.num_layers = len(depths)
  493.         self.embed_dim = embed_dim
  494.         self.ape = ape
  495.         self.patch_norm = patch_norm
  496.         self.out_indices = out_indices
  497.         self.frozen_stages = frozen_stages
  498.  
  499.         # split image into non-overlapping patches
  500.         self.patch_embed = PatchEmbed(
  501.             patch_size=patch_size, in_chans=in_chans, embed_dim=embed_dim,
  502.             norm_layer=norm_layer if self.patch_norm else None)
  503.  
  504.         # absolute position embedding
  505.         if self.ape:
  506.             pretrain_img_size = to_2tuple(pretrain_img_size)
  507.             patch_size = to_2tuple(patch_size)
  508.             patches_resolution = [pretrain_img_size[0] // patch_size[0], pretrain_img_size[1] // patch_size[1]]
  509.  
  510.             self.absolute_pos_embed = nn.Parameter(torch.zeros(1, embed_dim, patches_resolution[0], patches_resolution[1]))
  511.             trunc_normal_(self.absolute_pos_embed, std=.02)
  512.  
  513.         self.pos_drop = nn.Dropout(p=drop_rate)
  514.  
  515.         # stochastic depth
  516.         dpr = [x.item() for x in torch.linspace(0, drop_path_rate, sum(depths))]  # stochastic depth decay rule
  517.  
  518.         # build layers
  519.         self.layers = nn.ModuleList()
  520.         for i_layer in range(self.num_layers):
  521.             layer = BasicLayer(
  522.                 dim=int(embed_dim * 2 ** i_layer),
  523.                 depth=depths[i_layer],
  524.                 num_heads=num_heads[i_layer],
  525.                 window_size=window_size,
  526.                 mlp_ratio=mlp_ratio,
  527.                 qkv_bias=qkv_bias,
  528.                 qk_scale=qk_scale,
  529.                 drop=drop_rate,
  530.                 attn_drop=attn_drop_rate,
  531.                 drop_path=dpr[sum(depths[:i_layer]):sum(depths[:i_layer + 1])],
  532.                 norm_layer=norm_layer,
  533.                 downsample=PatchMerging if (i_layer < self.num_layers - 1) else None,
  534.                 use_checkpoint=use_checkpoint)
  535.             self.layers.append(layer)
  536.  
  537.         num_features = [int(embed_dim * 2 ** i) for i in range(self.num_layers)]
  538.         self.num_features = num_features
  539.  
  540.         # add a norm layer for each output
  541.         for i_layer in out_indices:
  542.             layer = norm_layer(num_features[i_layer])
  543.             layer_name = f'norm{i_layer}'
  544.             self.add_module(layer_name, layer)
  545.         self.width_list = [i.size(1) for i in self.forward(torch.randn(1, 3, 640, 640))]
  546.  
  547.     def forward(self, x):
  548.         """Forward function."""
  549.         x = self.patch_embed(x)
  550.  
  551.         Wh, Ww = x.size(2), x.size(3)
  552.         if self.ape:
  553.             # interpolate the position embedding to the corresponding size
  554.             absolute_pos_embed = F.interpolate(self.absolute_pos_embed, size=(Wh, Ww), mode='bicubic')
  555.             x = (x + absolute_pos_embed).flatten(2).transpose(1, 2)  # B Wh*Ww C
  556.         else:
  557.             x = x.flatten(2).transpose(1, 2)
  558.         x = self.pos_drop(x)
  559.  
  560.         outs = []
  561.         for i in range(self.num_layers):
  562.             layer = self.layers[i]
  563.             x_out, H, W, x, Wh, Ww = layer(x, Wh, Ww)
  564.  
  565.             if i in self.out_indices:
  566.                 norm_layer = getattr(self, f'norm{i}')
  567.                 x_out = norm_layer(x_out)
  568.  
  569.                 out = x_out.view(-1, H, W, self.num_features[i]).permute(0, 3, 1, 2).contiguous()
  570.                 outs.append(out)
  571.  
  572.         return outs

​​

4.1 修改一

第一还是建立文件，我们找到如下ultralytics/nn文件夹下建立一个目录名字呢就是'Addmodules'文件夹 ！然后在其内部建立一个新的py文件将核心代码复制粘贴进去即可。

​

4.2 修改二

第二步我们在该目录下创建一个新的py文件名字为'__init__.py'( ，然后在其内部导入我们的检测头如下图所示。

4.3 修改三

第三步我门中到如下文件'ultralytics/nn/tasks.py'进行导入和注册我们的模块( ！

4.4 修改四

添加如下两行代码！！！

​

4.5 修改五

找到七百多行大概把具体看图片，按照图片来修改就行，添加红框内的部分，注意没有()只是 函数 名。

​ 

  1.         elif m in {自行添加对应的模型即可，下面都是一样的}: # 这段代码是自己添加的原代码中没有
  2.             m = m(*args)
  3.             c2 = m.width_list  # 返回通道列表
  4.             backbone = True

4.6 修改六

下面的两个红框内都是需要改动的。

​ 

  1.         if isinstance(c2, list):
  2.             m_ = m
  3.             m_.backbone = True
  4.         else:
  5.             m_ = nn.Sequential(*(m(*args) for _ in range(n))) if n > 1 else m(*args)  # module
  6.             t = str(m)[8:-2].replace('__main__.', '')  # module type
  7.  
  8.  
  9.         m.np = sum(x.numel() for x in m_.parameters())  # number params
  10.         m_.i, m_.f, m_.type = i + 4 if backbone else i, f, t  # attach index, 'from' index, type

4.7 修改七

如下的也需要修改，全部按照我的来。

​

代码如下把原先的代码替换了即可。 

  1.         if verbose:
  2.             LOGGER.info(f'{i:>3}{str(f):>20}{n_:>3}{m.np:10.0f}  {t:<45}{str(args):<30}')  # print
  3.  
  4.         save.extend(x % (i + 4 if backbone else i) for x in ([f] if isinstance(f, int) else f) if x != -1)  # append to savelist
  5.         layers.append(m_)
  6.         if i == 0:
  7.             ch = []
  8.         if isinstance(c2, list):
  9.             ch.extend(c2)
  10.             if len(c2) != 5:
  11.                 ch.insert(0, 0)
  12.         else:
  13.             ch.append(c2)

4.8 修改八

修改七和前面的都不太一样，需要修改前向传播中的一个部分， 已经离开了parse_model方法了。

可以在图片中开代码行数，没有离开task.py文件都是同一个文件。 同时这个部分有好几个前向传播都很相似，大家不要看错了， 是70多行左右的！！！，同时我后面提供了代码，大家直接复制粘贴即可，有时间我针对这里会出一个视频。

​​

代码如下-> 

  1.     def _predict_once(self, x, profile=False, visualize=False, embed=None):
  2.         """
  3.         Perform a forward pass through the network.
  5.         Args:
  6.             x (torch.Tensor): The input tensor to the model.
  7.             profile (bool):  Print the computation time of each layer if True, defaults to False.
  8.             visualize (bool): Save the feature maps of the model if True, defaults to False.
  9.             embed (list, optional): A list of feature vectors/embeddings to return.
  11.         Returns:
  12.             (torch.Tensor): The last output of the model.
  13.         """
  14.         y, dt, embeddings = [], [], []  # outputs
  15.         for m in self.model:
  16.             if m.f != -1:  # if not from previous layer
  17.                 x = y[m.f] if isinstance(m.f, int) else [x if j == -1 else y[j] for j in m.f]  # from earlier layers
  18.             if profile:
  19.                 self._profile_one_layer(m, x, dt)
  20.             if hasattr(m, 'backbone'):
  21.                 x = m(x)
  22.                 if len(x) != 5:  # 0 - 5
  23.                     x.insert(0, None)
  24.                 for index, i in enumerate(x):
  25.                     if index in self.save:
  26.                         y.append(i)
  27.                     else:
  28.                         y.append(None)
  29.                 x = x[-1]  # 最后一个输出传给下一层
  30.             else:
  31.                 x = m(x)  # run
  32.                 y.append(x if m.i in self.save else None)  # save output
  33.             if visualize:
  34.                 feature_visualization(x, m.type, m.i, save_dir=visualize)
  35.             if embed and m.i in embed:
  36.                 embeddings.append(nn.functional.adaptive_avg_pool2d(x, (1, 1)).squeeze(-1).squeeze(-1))  # flatten
  37.                 if m.i == max(embed):
  38.                     return torch.unbind(torch.cat(embeddings, 1), dim=0)
  39.         return x

到这里就完成了修改部分，但是这里面细节很多，大家千万要注意不要替换多余的代码，导致报错，也不要拉下任何一部，都会导致运行失败，而且报错很难排查！！！很难排查！！！

4.9 修改九

我们找到如下文件'ultralytics/utils/torch_utils.py'按照如下的图片进行修改，否则容易打印不出来计算量。

​

五、融合使用教程

训练信息：YOLO11-SwinTransformer-RepGFPN summary: 353 layers, 2,736,687 parameters, 2,736,671 gradients, 6.3 GFLOPs 

  1. # Ultralytics YOLO 🚀, AGPL-3.0 license
  2. # YOLO11 object detection model with P3-P5 outputs. For Usage examples see https://docs.ultralytics.com/tasks/detect
  3.  
  4. # Parameters
  5. nc: 80 # number of classes
  6. scales: # model compound scaling constants, i.e. 'model=yolo11n.yaml' will call yolo11.yaml with scale 'n'
  7.   # [depth, width, max_channels]
  8.   n: [0.50, 0.25, 1024] # summary: 319 layers, 2624080 parameters, 2624064 gradients, 6.6 GFLOPs
  9.   s: [0.50, 0.50, 1024] # summary: 319 layers, 9458752 parameters, 9458736 gradients, 21.7 GFLOPs
  10.   m: [0.50, 1.00, 512] # summary: 409 layers, 20114688 parameters, 20114672 gradients, 68.5 GFLOPs
  11.   l: [1.00, 1.00, 512] # summary: 631 layers, 25372160 parameters, 25372144 gradients, 87.6 GFLOPs
  12.   x: [1.00, 1.50, 512] # summary: 631 layers, 56966176 parameters, 56966160 gradients, 196.0 GFLOPs
  13.  
  14. # 下面 [-1, 1, LSKNet, [0.25，0.5]] 参数位置的0.25是通道放缩的系数, YOLOv11N是0.25 YOLOv11S是0.5 YOLOv11M是1. YOLOv11l是1 YOLOv11是1.5大家根据自己训练的YOLO版本设定即可.
  15.                                 #  0.5对应的是模型的深度系数
  16.  
  17. # YOLO11n backbone
  18. backbone:
  19.   # [from, repeats, module, args]
  20.   - [-1, 1, SwinTransformer, [0.25,0.5]] # 0-4 P1/2
  21.   # 0 1P2 2P3 3P4 4P5
  22.   - [-1, 1, SPPF, [1024, 5]] # 5
  23.   - [-1, 2, C2PSA, [1024]] # 6
  24.  
  25. # YOLO11n head
  26. head:
  27.   - [-1, 1, Conv, [512, 1, 1]] # 7
  28.   - [3, 1, Conv, [512, 3, 2]]
  29.   - [[-1, -2], 1, Concat, [1]]
  30.   - [-1, 2, C3k2, [512, False]] # 10
  31.  
  32.   - [-1, 1, nn.Upsample, [None, 2, 'nearest']] #11
  33.   - [2, 1, Conv, [256, 3, 2]] # 12
  34.   - [[-2, -1, 3], 1, Concat, [1]]
  35.   - [-1, 2, C3k2, [512, False]] # 14
  36.  
  37.   - [-1, 1, nn.Upsample, [None, 2, 'nearest']]
  38.   - [[-1, 2], 1, Concat, [1]]
  39.   - [-1, 2, C3k2, [256, False]] # 17
  40.  
  41.   - [-1, 1, Conv, [256, 3, 2]]
  42.   - [[-1, 14], 1, Concat, [1]]
  43.   - [-1, 2, C3k2, [512, False]] # 20
  44.  
  45.   - [14, 1, Conv, [256, 3, 2]] # 21
  46.   - [20, 1, Conv, [256, 3, 2]] # 22
  47.   - [[10, 21, -1], 1, Concat, [1]]
  48.   - [-1, 2, C3k2, [1024, True]] # 24
  49.  
  50.   - [[17, 20, 24], 1, Detect, [nc]]  # Detect(P3, P4, P5)

六、成功运行截图

七、本文总结

到此本文的正式分享内容就结束了，在这里给大家推荐我的YOLOv11改进有效涨点专栏，本专栏目前为新开的平均质量分97分，后期我会根据各种最新的前沿顶会进行论文复现，也会对一些老的改进机制进行补充，如果大家觉得本文帮助到你了，订阅本专栏，关注后续更多的更新~